User blog comment:SpookyBoogie27/Light Novel Spoilers/@comment-24098614-20131223074221/@comment-9339850-20131224051227

There are basically only 4 girls in the harem that Rias informally allows at the moment; And they are Rias herself, Asia (because she can feel the love Asia has for Issei), Akeno (cant provide an example but earliest inclinations would be since the date Akeno and Issei went on) and Koneko (from the time Koneko had started to change from going into heat, then proposing to Issei for the most part).

And i dont think Rias acutally thinks of the other girls (Xenovia, Irina, Ravel, probably eventually Rossweisse) as "harem members" yet, not until they have had that personal connection with Issei that the first 4 girls have had.