Unfortunately Xauth and IKE-Mode-Config are broken on Linux using ipsec-tools <= 0.6. As soon as Xauth is working I will put up some hints on using these.
Using ESP in tunnel mode without compression increases the size of the packets transferred. This even happens sometimes when compression is activated. This can cause problems in tunnel mode when the client does not know that the packet is to be encapsulated. If the clients sends a packet of 1500 bytes the additional encapsulation will increase the size of the packet. For TCP-packets you can solve this problem setting the MSS on both sides of the tunnel using the iptables command:
iptables -t mangle -A PREROUTING -p esp -j MARK --set-mark 1 iptables -A FORWARD -m mark --mark 1 -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --set-mss 1400