We've done that. What it results in is having to queue up at the manned immigration counter to use a passport instead of using the automatic card reading machines, i.e. it takes about 5 minutes instead of 2 minutes to get through.
We'd even asked about the procedure on the way out of HK as we'd realised that an ID card had been left at home. Outgoing immigration said it was completely standard to use passports and not to sweat it.