User blog comment:B214/Talk About What You Want! 3/@comment-24098614-20150501094248/@comment-26209278-20150501163313

I believe he does know, considering the stuff he has done with her, and his thoughts on her (can't remember specifics, i'm sure its on the wikia).

As much as he loves Rias, i doubt he'd drop the harem. He (finally) realises the women love him, and it would be unfair on them if he turnt his back on them. But he'd always get Oppai from Rias, he just wouldn't get a variety like he currently does...