During the weekends and evenings all the public pools will be crowded during the summer months.
If you have daytimes free, then mid morning is usually a good time slot to get some lane time without sharing it with 20 others. (you will still have to share a lane, but it maybe 2-4 people instead of 20)
Universities have decent, uncrowded pools and are relatively inexpensive to join, even as a regular resident. ($300 per month)
You can of course swim for free in the ocean, which many people still do on a daily basis (morning and night). Near the Science Park in University/Tai Po there is always a few men and women swimming off the pier. Same goes for Ma on Shan/Wu Kai Sha piers.