It probably depends on what material the plugs are made from. The orange soft rubber ones I wear can be worn for hours at a time with little discomfort. The urethane expanding type (you squish'em before you put'em in) are very effective but indeed tough on the ear. Of course, I'd prefer nothing, but for hours-long wears, good plugs are much better than muffs. Esp w/ eyewear.
But, as I said initially, if the situation is one where the protection is intermittent and frequent, on/off (muffs) is indeed better than in/out (plugs).
I have both around in my own shop or at work. As it turns out, I wear plugs at work almost exclusively, and muffs in my shop almost exclusively. But, at home, I now wear plugs to mow the lawn. :)
Also has to do with how dirty your hands are when removing equipment. Dirty greasy hands tend to favor muffs.
All different situations. ymmv.
Also, you can buy many pairs of plugs, and have'em all over. Ditto w/ my 99c muffs. Bought almost a dozen. :)