The letter Y is a common misconception for the Old English letter, called [i]thorne[/i](þ)
which makes a [u]th[/u] sound. Bad handwriting of scholars and such make "þ"look like a "y".
However, "Olde" is correct. There was in fact an E on the end of most words from Old English to early modern English, i.e. Jamestowne.
-
If it looks like a Y than its the same thing ye olde is pronounced the old