He's against the DoE on constitutional grounds. There is no justification for the federal government usurping state power for controlling education.
I believe his opposition to the DoHS is that it is insanely inefficient. A big problem leading to 9/11 was the large bureaucracy in our military intelligence. To solve it they figured they would add another layer to the bureaucracy. They're still working out the details for it, *years* later.
So, I believe his primary opposition to the DoE is principled, and the DoHS is practical problems.