How to size the capacitor on the dc power supply for stepper motor?

You should be able to find info on power supply design on the web. Offhand, I'd say you're in trouble on this, and you might be better off springing $100 or so to buy a regulated supply off the shelf.

Roughly speaking, you want to keep the ripple down to 1%, which is probably still high, and you'll need roughly 50,000 uF.

At 68v, your 48v stepper will pull something like (68/48) * 3A = 4.2 Amp. If you use full-wave rectification, then the half period is about

0.008 sec. For a 1% drop, the dv will be about linear, so you can estimate the capacitance from I = C dv/dt, or C = I * dt/dv, giving

C = 4.2A * 0.008 sec / (0.01 * 68V) = 49,000 uF

First guess.

Thanks. In this case, I would probably to spare $100 to buy a regularated power supply and at the same time I would build one power supply to try out.

You could probably forget the transformer all together and rectify the AC to

200+ volts into a smallish capacitor, and then switch down about 150 watts into an L/C filter. That's what most power supplies do these days,

Take a look at CNCZONE.COM, if you search you can find lots of info about stepper motors on that site.

As far as the power supply capacitor, the general "Rule Of Thumb" appears to be something in the range of 1000 microFarid PER Load Volt.

While you CAN run steppers at many times there RATED Voltage ( some sources say up to 20x ) it is not advisable th exceed the PER COIL ( normally what is marked on the motor, or in the spec sheet ) Current Rating (your case 3 Amps) or you will burn the coils out in a hurry. Voltage = Speed : Current = Torque