John,
You should be looking to have your tools fit into the hardy hole loosely by hand. What I mean is that they should be secure and not fall out or flop around when you use them, but, they should be easy to remove with your hand. If they fit too tightly into the hardy hole they may cause more problems if they are worked too hard. For example, if you really wail on a hot or cold cut that is tight in the hole, this could cause the heel of the anvil to crack through the hardy hole. Not something you want. If any of your current tools fit tightly, consider filing/grinding them so that they drop easily into the hole.
Hope this helps, Paul