Reducing the shaft diameter would worsen the problem, Chuck, IMO. Even if
the bearings aren't positioned perfectly parallel to each other, a small
degree of error wouldn't matter a lot at the slow speed of the blade wheel.
I can't recall from memory just how the bearings are situated, but if there
is a spacer between them in a single bore, I'd check that the ends of the
spacer are parallel.
If the ball bearings aren't perfectly centered axially, this would likely be
more of a problem, IMO, but since the hole(s) is/was probably cut in one
operation, I would expect them to be positioned relatively accurately.
My 4x6 is not so old, but I haven't had any problems with the blade walking
off the wheels, and the saw has seen considerable use.
Before I used the saw, I practically disassembled the entire machine, and
checked the alignment of almost everything as I reassembled it.
I noted that shimming the free wheel was required to bring it to the same
plane as the drive wheel.
I placed a tap handle (the long type not the T-handle type) on the input
pulley shaft and wound the blade around by hand while watching the tensioned
blade go around.
Looking back now, a variable speed drill motor would've probably been
easier, but it allowed me to see that the blade was tracking properly in
slow motion (although not cutting any metal).
Some other details of the 4x6 bandsaw are here: