I would be interested in learning more of the details about this. But I believe that, even though ID was not required to vote, there were other means of accomplishing the same thing that ID does. Voting was typically limited to white, land owning, males (based on state or local laws, which I understand varied, so this generalization may not have held everywhere), and election officials had access to tax and land ownership records, and would typically know the voters personally. People whose status was not known to the election officials could swear oaths affirming they met the requirements with legal penalties if it turned out they didn't. And there were provisions made for people to vouch for others.
I don't support restricting voting to whites (although restricting it to land owners doesn't sound so bad, restricting it to men strikes me as a great idea, and I can think of some other restrictions that make a lot of sense). But I have trouble seeing what's controversial about requiring ID to vote.