Despite HK's image as an impossibly dense urban center, if you want to live "not-in-the-city," you actually have a choice between lots of different modes of living, like villages, buildings in older residential areas, New Towns, outlying islands, etc. And of course, the south side of HK island. Anyway, my advice is to take advantage of your few weeks before you have to find a place, and have a look around everywhere you can go to, before you lock your mind into a decision, cuz what you can read on the internet about "recommended places to live in Hong Kong" is just a tiny fraction of what Hong Kong has to offer.
Useless New Territories trivia: There is actually a Causeway Bay in Shatin, near Tai Wai station --- Causeway Bay village. Though the area is not so much of a rural village anymore, increasingly urbanized (or New Town-ized) these days, since it's in a good location. (We also have a Causeway Bay in Shenzhen, around Huaqiang North Road ... that one's a shopping mall.)