I've been a bit spoiled, having been to a lot of secluded, beautiful beaches. I definately like the peace and quiet that goes with that, but the flipside is that those beaches aren't patrolled by life guards. Though ironically the closest I have ever come to drowning was on a patrolled beach and it was my Dad, not a life guard, who saved me.
If you swim at a secluded beach you need to bear in mind some basic safety stuff, like don't go alone, take at least one friend with you in case you get a cramp or caught in a rip or something, and take note of the tide. There are times of day that are safter for swimming than others.
Another concern is that beaches that aren't set aside for swimmers are often classed as roads and you have to watch for cars which can be going quite fast. That can be a problem if you have young children who can be impulsive.