Yes, there are many street stalls, small bookshops etc. selling inexpensive books printed on cheap paper, but the books will mainly be in Chinese.
There are a number of second-hand bookshops that sell English books (see the many previous threads on this topic) but given that they cater to the expat market they don't tend to give significant enough savings over buying new to justify traveling to them unless you happen to live nearby, in my experience.
Another way is to go to the Central Library second-hand book sale thing they have every year where books can be purchased very cheaply and stock up on books there but it's very popular so be prepared to spend a couple of hours queuing up to get in. I went once but wouldn't go again as I can't be arsed to queue so long.
If you're buying new you don't have to go to places like Bookazine, Dymocks etc. Depending on your interests, large branches of local bookshops like Cosmos, Joint Publishing/Commercial Press often have much better ranges of English books (certainly in terms of history, philosophy, literature etc) than the airport novel stuff that Dymocks and Bookazine tend to specialise in.
You can also look online at places like the sister shopping site here or Paddyfields.