To me, A&F is not related to "whiteness" per se. I am more likely to think of preppy jocks who bully others. Their former CEO kinda reinforces this view of the company and company culture.
This is a reason why i don't buy their clothes, rather than the racist line of clothing (which doesn't really bother me).
As for people who buy it, they probably don't know all the scandals behind the brand. And their marketing of hiring good looking people and putting good looking people in their clothes works at making people want to buy their clothes.
I've seen more people who proudly list they work at A&F rather than at Bossini, H&M and M&S... even they are all just sales assistants.