Wesley really is in the worst shape of any of them. Angel had problems with the way he mishandled Faith and took him in only as a charity case, so now Angelus doesn't have a lot of emotional connection to him. It means Angelus is a little meaner with him as the natural personality comes out.