A personalised music box or cross with the name and date on? Or a keepsake box? I think a cross bracelet or necklace could also work, but I would then go for an adjustable one that can withstand the test of time. There’s also customised night lights (with or without a music box) on Etsy. Get some clay and have an imprint of the baby’s hands and/or feet to hang up in a frame. I feel like there’s loads of options!
If you don’t want to go the traditional route you can just get any (baby) gift and add clothes, diapers or baby wipes if you want to make a little gift basket.