Honestly I think the biggest reason for unemployment is simply uncertainty. There's a good deal of hype over taxes of all kinds - death taxes, payroll taxes, cap and trade, more or less ad infinitum. It's essentially manipulation of market indicators - in this scenario, it is hard for companies and consumers alike to evaluate risk. Should Company X hire 20 new workers? How should they know: cap and trade taxes might be instated and they may thus need the wage money for a "rainy day" instead.
Or consider Consumer Z. He has a little extra money coming in, but what if there's an unexpected change in money supply or income taxes. Obviously, that makes him more hesitant to spend and save money where the markets would best dictate it to go based on risk and reward. Screwing around with economic indicators always brings uncertainty, which makes people more hesitant to do anything, especially taking the risk of hiring some new workers.
Lastly, I think that free trade and the exporting of blue collar jobs does not have any major, negative, long term impact on the average middle-class person. For one, it means that lower income families are able to get goods at cheaper prices. Logically, if foreign countries begin specializing in blue-collar jobs, businesses in the United States can focus on other things, which is why we have seen such an exponential increase in white collar jobs in the last five or so decades.