As you said yourself he’s a selfish individual. That’s who he is and he’s not going to change. It’s definitely time to think about if that’s the kind of person you want to have in your life. I’ve been in a similar situation where I was aware of that but until I saw certain things, I didn’t truly realise how ugly of a personality trait selfishness is. This is definitely something you should not just let go and ignore, sounds like it’s already taking a toll on you. Relationships shouldn’t drain you.