I think surprisingly, most people who say that phrase are just joking, not actually feeling guilty. It's a way of admitting something's kind of silly while still enjoying it. Banning it seems like it would just take the fun out of stuff.

The phrase is actually a pressure valve, not a real judgment. You make a fair point that it's just about silly stuff, but that's why it works. Banning it would take away a harmless way to laugh at yourself and others. I just don't see the harm.