In the grimiore it says they are female I know but they are called Wizards in game and the grimiore cards still classy them as Wizards so call them what they are they are Wizards not witches
-
For the most part, they're female. It refers to them being brood mothers to ogres and thralls, and all the important one's you kill are female. Maybe it's just an oversight, like the phogoth fight when he roared after your ghost says " That doesn't sound god." Or when Variks says " They're blocking communications" then like thirty seconds later communicates with you?