That could be expensive. Say you had $10,000 in one. Ten percent penalty knocks you down to $9000 and then it counts as ordinary income for you. If you are in a 25% tax bracket, you could lose another $2500 (not sure if the tax applies to amounts before or after the penalty is applied) or $2,025 which would drop your $1000 down to about $6500. They keep 20% as withholdings against the taxes. You just lost $3500. Now if you wanted to put that into a different investment, it would have to go up by 46% just to break even (ignoring costs of the new investment) with what you had in the 401k (to get the $6500 back up to $10,000). That is a big return you need. And that assumes that the 401k didn't grow at all in that time- otherwise you need even more growth to simply break even with keeping the account.
If you are seriously worried (and I am not), I would not close the account but perhaps put any new money into something else and leave the money you already have in the 401k.