4 main parts of HK-
Hong Kong Island;
Kowloon (includes TST);
New Territories, except for the islands;
the islands, which i'm going to leave out, since I don't know them (though someone will hopefully chime in about Lantau, at least).
so, of the other three (HK, K, NT), Kowloon is by far the worst for cycling. Cramped, flat, just nothing going on.
HK Island seems to have some good riding, it's certainly got hills to climb.
NT has a bunch of cycling options. Eventually you'll run out of Hong Kong to ride in, but there's, relatively speaking, a lot of options.
Housing is much cheaper in NT than HK Island, though it wouldn't surprise me if there are more studios to be found on HK Island. NT seems to be the world of the 3-bedroom apartment. I'm sure that other posters will point out exceptions.
If you're on the West Rail Line, you can get to East TST station without changing, and you can walk to TST station in the underground air conditioned station if you wish.
From the East Rail Line, same thing just with one change. Those would be the options I'd suggest.