Koh Phangan is amazing - all the locals are friendly, it's cheap as hell, both for food and accomodation, there's tonnes of things to do (elephant rides, snorkelling, jungle tours) and the beaches are gorgeous. if you're going with kids then the 'downtime' after the full moon party would be a good slot to go, not overly crowded as all the party animals leave ASAP.
it's very safe and a relatively small island - most of the things you would sign up to do have a pick up and drop off service, so there would be no need to get taxis or anything. if you stay in Haadrin everything is in walking distance - restaurants, the beaches, convinient stores, internet cafes, spas... it's got enough to keep you entertained but it's not too overwhelming for a first-time traveling family.