There is a certain set of people called icebergs who swim all year round, particularly in winter. Some countries have events to celebrate winter swimming. It is purported to have special health benefits.
I love swimming in the ocean during winter when I'm visiting the tropics. The beaches around Broome, Cairns and the islands of the Great Barrier Reef are gorgeous at this time of the year. Although it's not as hot as summer, the ocean is always pleasantly warm in the middle of the day...just perfect for a dip. Anywhere further south...probably not!