Tons of people have done it. I have the Shooting Star 2 axis outfit on my old vertical mill and love it. I know what you are talking about, worrying that you won't do any permanent damage. I suggest taking the time to make simple jigs to assure that holes get drilled in the right place. If you have to tap some of them, get some brand new high quality taps. Maybe you need to spot face/file/grind some of the locations where holes need to be drilled.
Don't work on it when you are tired, or after too many beers. Don't "Expect" that you will make mistakes. That may be just a self-fullfiling prophecy. Let your gut tell you to stop, think, get more info or skill, as necessary.