There are always several ways to do any job. When notching pipe to weld together, a really pro shop will have several different techniques, depending on material, size, wall thickness, and end use. A tubing notcher, like the manual ones sold by williams lowbuck tools, will notch tubing- they wont work very well on schedule 40 1 1/2" pipe, as they are made for much thinner walls. Rigid conduit is thinner than pipe, might work with a tubing notcher. But when you need a very precise notch, you wouldnt use a mechanical notcher anyway, either hand powered or one that fits in a punch press or ironworker. These mechanical notchers are not too precise, and are used in applications like roll bars, where a mig or tig weld will not even be sanded out. For real precise fits, a hole saw will work- I use annualar cutters, like Hougen makes for mag drills, instead of ordinary hole saws, in an "ol joint jigger" set up in my bridgeport, and it will give very accurate notches, at any angle, in most pipes. Other techniques include abrasive pipe notchers- basically belt sanders with the end roller interchangeable, so the size of its radius matches the pipe size you want notched. Or mechanical mini milling machines that use end mills. Or, in extreme cases, laser or plasma cnc machines with 4 axis of movement.
How big a hammer you need depends on the nail.
You ought to be able to to some notching with your drill press. You will find on larger stuff that the drill press is just not rigid enough, and probably a woodworking tool, and wont go slow enough, for some metalworking. The morse taper slip fit of the chuck in not intended for heavy loads, and past a certain size of hole or thickness of material, you will find the chuck comes loose and spins free- This happened to me with a much bigger drill press than yours, when drilling hundreds of 1 1/2" holes in 1/2" stainless bar.
These guys sell a range of device to do this job- the purple hole saw guide would be the logical next step for you, as it would allow you to do angles as well-
formatting link