A TL;DR rundown of the template locations.
Where to Find Armor Templates
The information below tells you the locations of the templates hidden underground. If you want to solve the clues for yourself, do not read further.
Some of the templates took multiple runs to encounter, but they did eventually show up. Good luck!
Along the wall, you will encounter a tiny blue mushroom, which reveals a shrinky-knight tunnel.
Use the floating puff ball enemies to shrink and make your way through.
The template is inside this area, past a zig zag of platforms and a few beetle enemies.
When you clear one of the floors, you may encounter a ruined and boarded up tunnel. The template is buried in the shallow dirt.
Use the roulette mole NPC in one of the side rooms. This might take multiple attempts, but you can slap the barrel with your shovel to stop it early to try and stop it on the template.
The Grub Pit
In areas where bees circle around you and attack occasionally, look for an area where the bees start circling around a tile of dirt instead of you.
The template is hidden the dirt tile being circled by the bees.
This took me multiple attempts, so it may not be guaranteed.
There are thin metal land mines hanging on the wall. Just slap them off the wall with your shovel.
The template is hiding behind one of these.
Drill Knight Castle
You will need a way to maneuver yourself over the insta-kill buzz saw that shows up then you linger on a screen for too long.
You must bounce on top of the brown metal cover with the spiral pattern.
The template will appear after the cover breaks.
I intentionally damaged myself with an enemy to get just enough invulnerability frames to jump through the saw and start bouncing on top.