IPsec is a well-known and widely-used VPN solution. It seems that it's not widely known that Niels Ferguson and Bruce Schneier performed a detailed security analysis of IPsec and that the results were not very positive.
We strongly discourage the use of IPsec in its current form for protection of any kind of valuable information, and hope that future iterations of the design will be improved.
I conveniently left out the second part:
However, we even more strongly discourage any current alterantives, and recommend IPsec when the alternative is an insecure network. Such are the realities of the world.
To put this in context: keep in mind that this paper was released in 2003 and the actual research may even be older (1999!). OpenVPN, an open-source SSL-based VPN solution was born in 2001 and was still maturing in 2003. So there actually was no real alternative back then.
It worries me that this research done by Ferguson and Schneier is more than a decade old. I've been looking for more recent articles on the current security status of IPsec, but I couldn't find much. There have been some new RFCs been published about IPsec but I'm not familiar enough with the material to understand the implications. They make a lot of recommendations in the paper to improve IPsec security, but are they actually implemented?
I did find a presentation from 2013 by Peter Gutmann (University of Auckland). Based on his Wikipedia page, he seems to 'have some knowledge' about cryptography. The paper adresses the Snowden leaks about the NSA and also touches on IPsec. He basically relies on the paper written by Ferguson and Schneier.
But let's think about this: Ferguson and Schneier criticises the design of IPsec. It is flawed by design. That's one of the worst criticisms any thing related to cryptography can get. That design has probably not changed much, from what I understand. So if their critique on IPsec is still mostly valid, all the more reason not to use IPsec.
So this is part of the conclusion and it doesn't beat around the bush:
We have found serious security weaknesses in all major components of IPsec.
As always in security, there is no prize for getting 90% right; you have to get
everything right. IPsec falls well short of that target, and will require some major
changes before it can possibly provide a good level of security.
What worries us more than the weaknesses we have identified is the complexity
of the system. In our opinion, current evaluation methods cannot handle
systems of such a high complexity, and current implementation methods are not
capable of creating a secure implementation of a system as complex as this.
So if not IPsec, what should you use? I would opt to use an SSL/TLS-based VPN solution like OpenVPN.
I can't vouch for the security for OpenVPN, but a well-known Dutch security firm Fox-IT has released a stripped-down version of the OpenVPN software (removed features) that they consider fit for (Dutch) governmental use. Not to say that you should use that particular OpenVPN version: the point is that OpenVPN is deemed secure enough to be used for governmental usage. For whatever that's worth.
At least, SSL-based VPN solutions have the benefit that they use SSL/TLS, which may have it's own problems, but is at least not as complex as IPsec.