HF bandsaw as a scroll saw?

Reducing or eliminating the set from one side of the blade isn't a good idea. That will cause the blade to cut towards the side with set and make it difficult to control the line of cut.
Having the teeth riding on the pulley shouldn't be a problem. This is the normal position of the blade on a standard vertical bandsaw. The blade guides will be your problem. Perhaps they could be replace with a different set that won't hit the teeth of the blade?

Jeez, this doesn't sound like a good idea to me. I was just at the Goodwill store and saw an old Craftsman scroll saw sitting there, $19.95USD. Looked fine to me, didn't bother to plug it in. Buying two blades for your 4x6 saw in those sizes (very nonstandard) will cost you more than a decent used scroll saw. Also, the whole point of a scroll saw is to be able to do piercing cuts and you can't do this on a bandsaw without cutting/welding the blade in situ which is a royal pain.
However, if you want to do it, also consider modifying the wheels, adding crowned rubber tires. Then you should have zero problems. Don't forget to adjust your blade guides to the thinner blades, don't know if they go that far.
Please post your result. I'm pursuing a similar goal from a different angle: I'm working on getting an old die filer's overarm saw attachment working. I'll post my results, you post yours.
GWE

THis will give you a blade that will cut a curve in one direction only, and you will never get a straight cut. For a saw to give a proper cut, left, right or straight, the set must be equal on both sides.

are there rear blade guides on this saw? If so, the adjust them so that the teeth are proud of the side blade guides, adjust the tracking with the wheels and carry on. If not, consider making some. A couple of pieces of 1/4 or 3/8 drill rod with clean ends attached above the upper guide and below the lower guide, adjustable to press on the back of the blade.
Mark Rand (hey grandma, do you know how to suck eggs? :-) RTFM

[ ... ]
Agreed -- and will also limit the ability to cut a tight radius, which is what you want it to be able to do.
Except that the standard vertical bandsaw has rubber tires, so the teeth simply embed in the tires, instead of being crushed to reduce the set.
Perhaps you can manage to fit the H/V bandsaw with some crowned tires of some sort, so you can get around the problem. This might also allow you to run the blade a more centered on the wheels, so you can adjust the guides to do what you need.
I think that all you need is a way to offset the back guide. Does the HF version use ball bearing races, like those on my MSC-branded H/V of the same general sort? For that, the solution would be to set the rear rollers higher than the two side ones, so it could be offset forward to force the teeth to clear the front edge of the side rollers.
Another possibility might be to turn a groove in the side rollers to accommodate the teeth for the narrower blade, and to turn a similar groove in the big wheels.
I've not needed to do any of these, because I got in addition a three-wheel bandsaw by EMCO-Maier (from eBay -- no longer made) which has crowned rubber tires on all three wheels, and three possible belt positions (two on the back wheel for high speeds,and one on the bottom wheel for really slow speeds. This makes it good for both aluminum/wood and for steel. The three-wheel bandsaw has a lot deeper throat, which you really need for scroll work on anything but the tiniest workpieces.
Good Luck, DoN.