I suspect that people tend to stick with the same brands, so may not even realize the extent to which Android is monkeyed around with by the manufacturers. Or even if they do, they might like it if they like their phones.
But also I don't think Android One is well-known. I was in a shop yesterday saying I wanted Android One, and the clerk kept looking at me confused and trying to correct me pointing at Android 11 on the phone box.
After I left, he probably regaled his colleagues with the story of that idiot that was asking for "Android 1" and doesn't even realize Android is on version 11 now!