Mebbe a 5/16 drill, followed by a 3/8 em. You might also find that with the straight 3/8 drill, yer hole is not perfectly round, either.
The other thing you can do, if possible, is drill stacks, mebbe stacks of 8 or more. This way, only the bottom pc is an issue. And, you'll have a rounder hole with 3/8 drill, and much greater efficiency.
I think your speeds and feeds could be ramped up, as well. Mebbe to 2038 rpm and 16.27 ipm. :) :) Seriously, I think you could easily double both.
Some alum is just crappy and gummy.