Lamma is a bit of a dump to be honest. It's a nice place to visit but I wouldn't want to live there. The accommodation standard looks poor and there are not too many things to do.
It's packed by tourists on the weekend and you can't move along any of the streets. There are plenty of places in Hong Kong where you can eat nice seafood too. The commute from Lamma to Kowloon would be horrific - well over an hour every day even if you time the boat ride correctly. You have a deeply unpleasant 10 minute walk in the heat from Ferry Pier 4 in Central to the station too.
Assuming you're not going to go to beaches in the mornings or evenings, I would say it is better to head there on the weekend to enjoy the beach and take the journey when you need to - rather than every day. Also, you can eat good seafood throughout Hong Kong - you don't need to live on Lamma island.
Come and have a look by all means, but I wouldn't set your heart on living there before you see it with your own eyes.