Guardian's do not intimidate Fallen Leaders, they do scare Vandals and Dregs however. If guardians become powerful enough or can prove how benevolent they are Vandals and Dregs would fight against their own kind almost inevitably. As for other loyals, they probably are just like how I was, hiding in a House waiting for rescue.