Do you think it's possible that the Speaker isn't even a Guardian? He kind of reminds me of a Pope, in that, he comes in and claims he can speak with/commune with our God (the Traveller) and everyone just accepts it because, after all, no one has ever heard the Traveller speak, anyway.