User blog comment:B214/Talk About What You Want! 2/@comment-74.77.52.225-20140712011028/@comment-1359322-20140712014708

Considering how many Devils have Harem, i doubt that they can use that as an excuse. Plus we don't know if Sirzechs and Rias are the only child of the family, remember much like Venelana when she was still in the Bael, the Gremory might have a main family and a branch family. The main family are like Sirzechs and Rias who are Pure-blooded and are the children of the first wife. Then there's the branch family who aren't fully Pure-blooded or are Pure-blooded but are not from the first wife.

Also the low birth rate may also contribute to that. Plus if they want to use that as an excuse, they would face lots of issues from not only the Gremory but other factions too. Like Akeno, the daughter of the current Vice Governor of the Fallen Angels, it won't be so easy for them to target the Gremory without issues.