I stayed 3 months in Belgium, and learnt quite a bit of Dutch.
I have stayed 3 years in HK, with very limited Canto (just the numbres and a few other words).
The reasons for me are simple:
It is difficult to learn because -
a. Most other languages we learn are European, and we have some exposure to them in one way or the other, the letters are familiar, there is similarity with English words since they share the same roots
b.Canto is tone based, something I have never encountered in other languages (I came across Canto and Putonghua parallely) which makes it a very difficult concept to grasp. In my language, or other languages I am familiar with (English, Hindi, Dutch etc.) tones are used only to express emotions.
c.If I try to speak, my tones go awry (would happen for a newbie), I find local people refuse to even try to understand...like mentally you can put the sentence together, and figure out that this particular word is not making sense, but if it were a slightly different tone, it would make sense. This is called "trying to understand" when a foreigner makes an effort to speak your language.
I have seen people do that when we tried speaking Dutch (which sounded very funny), I have seen people doing that when foreigners come to my country and speak a few phrases of Hindi or Bengali...but somehow, people in China (HK AND the mainland) cannot do that. Maybe it is something in their language, or maybe they just don't bother...I do not know...either way, it leaves a beginner frustrated.