I have never had children but if I had I would never have gone on holidays without them. A family is a whole unit not just segments to be disassembled and reassembled at will. Philosophy aside, children make everything so much more enjoyable. They are more aprreciative of things and their joy can be infectious.
That's a personal choice. My children are young and I wouldn't want to go on holiday without them because I want to share that time together as a family and I don't want them to miss out. I suppose something might come up where I would need to travel without them for some reason though. It's different with teenaged or adult children who might not want to go on holiday with their parents.
I don't think you should or you shouldn't, I think families should work out what they want to do. However, I personally take my kids everywhere with me. I had them so I could raise a family, and holidays as a family are a part of that!