Xander is surrounded by a whole lot of powerful women, so I can see why he doesn't worry too much about them. He's a little more worried about getting out of this conversation he started.