Go on the trip. 100% go on the trip.
Create new memories and reclaim those places you have memories with him. To be blunt if your ex stopped you from seeing your friends in the past then honestly you’ve probably let them down before, time to start being a better friend again and put them first, go spend time with them.
He held you back in life for long enough, if you continue to let him hold you back, it’s no longer him holding you back - you are now holding yourself back from living your own life, you are now choosing to hold yourself back. You would be doing yourself a major disservice to do that. You need to put yourself first for the first time in 4 years and yes it’s really hard but you are fully capable. I don’t think you will regret going on this trip but if you don’t go, I think you will regret it, regret letting his opinion dictate your life so much.
You need to find yourself and your independence again. You are a full whole person without him, you don’t need him.