Well, the most common cause is the front end of the tool is not parallel to the spindle axis. But, if the side of the tool is not at an exact right angle to the lathe axis, then the side will rub on the cut face as the tool goes in, and it will force the tool to one side or the other. You don't get this with a tool that is tapered behind the cutting edge, but the long parting-off tools have no side relief whatsoever, and will rub and be deflected. I generally accept that this will happen, and accomodate it.
Jon