A significant factor is that there is very little chance to learn Cantonese. For example, even here at the _Chinese_ U of HK, there are only two short Level 1 Canto courses for foreigners pursuing degrees. I can take Japanese up to Level 4 and Putonghua up to Level 6, but if I want to go beyond Level one Canto it means enrolling in full time programs or going to TST for evening classes. Repeated requests to the language center to offer higher level courses, even if only three or four students would enroll, have been answered with "Why don't you learn Putonghua, it is much more useful anyway".
While the local language in many other places - think France or Japan - is considered an essential part of the countries culture, here in HK I get the impression that people would actually prefer if they all had learnt Putonghua as their native language. They are positively surprised when they find out that I do speak a little, but I've huge troubles understanding them - they just speak way too fast. Hence, if it is someone who has a basic command of English, they'd just speak English to me then. I've yet to meet someone - other than my language exchange partners - who actually makes the effort to speak slow enough for me to understand.