The reason they put restrictions on black immigration (and on the expansion of slavery to the territories) was because the labor of blacks was an "insult" to white workers, who they believed were superior workers and therefore should be the ones who worked, and that blacks shouldn't have jobs. That is also the reason why many free Northern blacks went South in the period before the war, because in the South there was always someone there to hire them and white people didn't care that they worked alongside blacks, while Northerners refused to work with blacks, believing them to be inferior. Also read de Toqueville On Democracy in America about his observations that racism was more noticeable in the North than in the South, and the GOP was originally the Northern sectional party.