itinerary poopers

I work in an office building with five floors, and a few years ago someone in my office told me that they go to a different floor to poop.  I think the rationale was that when you stink up a bathroom, people won't recognize that it was their coworker who did it, but I'm adding a little to get to this explanation because it wasn't fully articulated.

We have several offices on our floor, and a few such as a bank and an office of lawyers and accountants have customers who visit throughout the day, so there are a lot of people who use the floor bathroom who I don't recognize. This being said, in the past few days I've witnessed two different men walking from a different floor into our bathroom to poop.  So, while I thought before that travelling to poop was a quirky activity that only one guy did, I'm noticing a small pattern.

I'm not bothered by the fact that some people might decide it's better to go to a different floor than the one they work on to poop.  Everyone has their thing.  I'm perplexed though, because I'd be more embarrassed for someone to notice I'm going from a different floor to poop in their bathroom than I would be for a coworker to notice I stunk up the bathroom.  Am I alone on this?