There are two main problems with wearing lenses when swimming.
The first is that they can get easily lost as the water can float them off the surface of your eyes (obviously closing your eyes whilst under water helps reduce this risk). The other, more serious problem is that the swimming pool water can enter into the contact lens and increase the risk of getting an eye infection, in particular the Acanthamoeba organism.
This is especially dangerous and can survive in the space between the contact lens and the eye. It is more commonly found in lakes and rivers and if the eye becomes infected with this organism it can be very severe and progress very quickly to cause vision loss.
Using daily disposable lenses can be very convenient whilst on holiday and are a safer option as long as they are discarded after use in the pool (or sea). Wearing water-tight swimming goggles are a better choice and can also be ordered as prescription lenses. For divers and snorkelers, prescription (and non-prescription) diving masks are also available.
Swimming in the sea can cause additional problems for contact lens wearers. Salt water will dry out your lens and salt water also contains lots of micro-organisms that could be harmful for your lenses and your eyes.