As harsh as it sounds, I tend to finish up and end the call. My line of work necessitates many hours on the phone negotiating with cast, crew, agents, locations and suppliers in general. When one call ends another starts, so I try and wind up a call once business has been done. Similarly once I've finished chatting with family and friends, I would rather end the call sooner and catch up in person instead (where possible).
It depends on who the caller is. I would never be the first to hang up on my mother but I frequently initiate the end of busines coversations. In the workplace it's a time management and efficiency issue.