I don't know if the clip was taken out of context, but if it wasn't, then Colbert is right. You can't kick someone out of the country for listening to words. You have to show that they have taken action towards harming someone.
Rogan is anti-taxes and anti-war and pro-Ron Paul, though he's probably not a pure libertarian quite yet because he watches all these documentaries about the collapse and how all the financial firms heisted all this money from taxpayers in the bailouts which is true... but then he seems to come out of it thinking we need more 'regulation' on the banking industry like some of those films advocate and I don't think he has an in depth understanding of free market monetary policy and how that would regulate the banking industry. I also get the impression that he thinks we need at least "some" regulation on food and medicine, although he's very much against the war on drugs and knows big pharma pretty much controls the FDA.
I just found Doug Stanhope a few days ago and I'm hooked. I like him far more than Bill Hicks or George Carlin, who I would put in the same category of social commentary comedians.