My son is 12 years old and a very good swimmer. Whilst I would be ok to let him swim in the pool by himself, I don't feel comfortable letting him go in the surf unsupervised, even when he is swimming between the flags.
I'm really funny about water - I'd not be ok with anyone swimming in the pool if I wasn't home - that said my 8 year old can swim fine and I wouldn't have to be right there all the time for him (though as I don't have a pool, I usually am). And ditto with surf - Supervise that forever!
Depends on the swimming ability of the child and the depth of the pool etc. Accidents happen around a swimming pool not through actual drowning but cocky horseplay that can mean a child falls too close to the edge, hits their head and can become unconcious. If it was my home pool and my kids had friends over, then I'd still be keeping an eye on them whatever their age. Same for the beach as well - I consider myself a fairly strong swimmer, but I've been caught in the rip a number of times and it is tricky getting out of it when that happens. I'm probably being over cautious, but perhaps that is better than underestimating the dangers of the water.
Let me tell you about what happened to mine. A visitor's kid thought holding my daughter's head under water was fun until his brother told us. All could swim but there are dangers always so best to watch regardless of ages.