There's a limit of how many settlers you can have. I think it's 10 and every point into charisma increases the limit by 1. You might be at the limit