I actually tried this, well a mock version of this as I do not wear mascara -at first my mouth was open and when I really concentrated, I could shut it but it is a natural thing for me to have my mouth slightly open.
As to why, I do not really understand this -it must just be a reaction?
You are right!! So many women DO open their mouths when putting on mascara!!! I do not wear the stuff so I do not have any idea whether I would open my mouth or not!! Some how I don't feel that I would though,as when I have to use Eye Drops,I do not open my mouth!! Or maybe I just don't want to get the drops in my mouth! I remember as a kid I would open my mouth or poke my tongue out of one corner a little bit when doing certain tasks. Dad used to joke that what ever the task was,the result was due to the way we held our mouth or tongue!!!
I heard this being said on some show on the TV and since then I actually make a conscious effort to keep my mouth closed. For some reason I wanted to prove that person wrong - really don't know why! So I don't open my mouth when putting on mascara!