Well, the Didact sort of Composed most of humanity so only a few with geas were left.

So the Librarian placed these genetic seeds within us, bound for great things, but the Didact screwed us over of course with the Composer. John just happens to be from a line of men with those genetic gifts?

Yeah. There was supposed to be 10,000,000 of us for the reseeding effort armed with existing knowledge on steamboats and minor engineering along with other advancements included in the Librarian's geas. However, the Didact composed the vast majority of that, forcing the Librarian to scramble back to Earth and grab the few hundred remaining humans left on Earth. Those of which left on Earth were old, and had a genetic makeup that didn't suit the librarian's plans. This why humanity is at a low technological level compared to the Covenant. Once again, the Covenant misinterpret scenarios. Truth does this in Halo 3. "Your forefathers wisely set aside their compassion, steeled themselves for what needed to be done! I see now why they left you behind. You were weak, and gods, must be strong!" As you can see, he uses this fact in an attempt to justify why the Covenant is more worthy for the Forerunner legacy.