Like any great empire, once it is defeated, it does not simply cease to exist. It fragments and fractures. What you see in Halo 4, is simply the largest fragment of the Covenant, lead by Jul 'Mdama.