The loss of freedom is hard... the idea of breeding all those women though, that's what's both driving him insane and giving him the strength to fight back. You're right that it would break him to know he had abandoned children to the slavers. That's just not something that a Xander *can* do as loyal and loving as he is.