Yup, if they were asking locals if living next to 'oi gok yan' was ok, they were probably thinking gweilos, and not a chas and haak gwais.
Speaking Cantonese and being brown can either help or hurt...some people treat me better, and others automatically think I'm a public housing estate guy on welfare or something, especially in lower-income areas. If I try to explain I speak several Asian and European languages, have British citizenship and went to college in the US, they don't really know what to make of me. lol. Some think I'm trying to show off. Many still think you're lower class, no matter how much education or money you have, simply because of your ethnicity.
Speaking perfect English while brown tends to be a big surprise for a lot of people...conversely speaking Cantonese in certain areas where people don't see many brownies gets me lots of WAAAAAs. It entirely depends on the person I'm speaking with and their attitudes. People are becoming more tolerant, however, whether the 'conservatives' (lol) like it or not.
Locals can be the most lovely people if they like you, but they can also be absolutely horrendous, and many don't see their behavior or beliefs as racist at all. Brown skin means you're lower class, and that's just the way it is many people's eyes. These attitudes occur in all strata and education levels.
Gary is right about the South Asian crime wave thing, actually. The new immigrants tend to be very poor, and lack opportunities to advance themselves because of their language skills and because many people simply won't hire them. Many HK-born, fluent Cantonese-speaking Indians and Pakistanis are stuck working low-income jobs. The Nepalese youth (many of whom don't speak Cantonese at all) have the same issue. When you're stuck in a dead-end job, crime and drugs look a lot more attractive.
Poverty and no hope of bucking the odds breed crime. Go into one of the lower income areas in HK at night and you'll see plenty of locals committing crimes (many of which are only reported on in the Cantonese papers). It's the same story all over the world.
I know most people on this planet haven't been fortunate enough to grow up with experiences and a lifestyle as exotic as mine, so I usually let stupidity slide. More recently, though, I've been confronting it head on, which the idiots never expect. It's pretty liberating.