Charisma affect how fast you rank up, which again affect which recruits you get in your roster. A min CHA recruit can be denied of highest rank even at last battle~
Charisma also affect barter, but since you can hire recruit specifically for barter skill, perk, this is not an important loss.
CHA buff items are not enough, and not usesable enough to help a min CHA recruit, not if you know where to look for that lack.
Generally, a normal recruit can ignore CHA, as that provide no help. But the main character, is who I would invest in CHA. I dont want the irritation at last battle where my main char is not the highest rank officer in the field.