The trick is that you use a center punch to mark the hole position and guide the drill. This works great in aluminum.
If you want a drill press, don't get the "Dremel tool drill press"; it's total crap. Get a small bench-top drill press. Take a look at Grizzly G7942,
The next step up is an X-Y table for a drill press. Then you can put holes in the right place consistently. Grizzly Industrial used to have a low-precision one for about $100, but they seem to have discontinued it. We had one of those at Team Overbot, on a low-end drill press. Very convenient. We had to make many mounting brackets, and used 1/8" x 1" steel bar stock, a bender, and a drill press for most of them.
Beyond that, you're into machining for the sake of machining. I know three people who have full-sized milling machines at home, but they don't use them enough to justify owning the beasts.
There's also "emachineshop.com". Download their program, design the part, pick the material, get an estimate on line, put in a credit card number, select number of copies desired, and start the job. Their free CAD program is quite impressive, and it understands their manufacturing capabilities. It won't let you submit a design they can't make, and it knows when you're doing something dumb, like specifying a square inside corner on a milled part.
John Nagle