Semen can throw off your ph down there allowing for infections to occur. Otherwise he might be also passing it to you, yeast infections in men are usually asymptomatic. He should get tested and treat it if that's the case.
Otherwise, neither of you should use scented soaps to clean your privates, ideally only water or very mild and unscented soap.