"Too inconvenient" is the typical answer…and I don't really understand it.
People see it as an Island, but really since it's connected to a bridge it's as much of an Island than HK Island, Tsing Yi or Lantau Island.
Also, what other places in HK allow you to be 30 minutes away from the Airport, and 20 min away from pretty much anywhere (Central, TST, New Territories, Kowloon with direct shuttle bus to Kwai Fong, Tsing Yi, Tsuen Wan)
It's 16 minutes away from Central by taxi. I do it sometimes from the HSBC Building or even Pacific Place. It does set you back about $220 but it takes you less than 18 min each time! Also it has the convenience of very regular and scheduled buses or ferries. I used to live in Kennedy Town and it took me anywhere between 8 minutes to 45 mins to get to work in Central depending on the weather and how crowded the buses are. From Park Island it takes me 35 min door to door. No more, no less. can't be more consistent.
So I think 'inconvenience' is quite inaccurate in fact.