If you want to avoid the bit walking all over the place, cut the bit in half so it is now not so long. Cut it as short as you can and have it useful. The drill will be stiffer and do less walking.
Another trick is to put a tube sleeve over most of the bit exposing only a short length at the tip. This again is a way of stiffening the bit to reduce it walking .
al