In my office it is cakes (was donuts from that place that has now closed down or egg/custard tart/pie). You take the box around and offer and say your farewells.
As a side, while I like this idea I would like to do the same thing on my first day and am wondering how this would be received? Too rude or presumptuous?