I model in N scale, and it's been a long time since I last ballasted any track. However, I did basically what most people here suggested. I forget what ballast material I used, but probably Woodland Scenics.
One thing I would suggest is to avoid white glue and wood glue, as they tend to dry stiff or brittle or hard, with the result that cars rolling along the track make an out-of-scale amount of noise. (See previous discussions of roadbed and subroadbed.) Instead, try Aleene's Tacky Glue, which is kinda like white glue but dries less stiff. Better yet, try "matte medium", which dries to a fairly spongy consistency *and* has a matte finish so your ballast won't look all shiny. (These two are the ones with which I have some experience -- others have suggested some other non-white-glue materials that may be even better and/or cheaper.) Again, mix 1:1 or 1:2 with water and add a drop of detergent or equivalent.
Also, my brush of choice (again, for N scale) was a 1"-wide flat paint brush, of the sort used for interior house finish painting or staining.
Oh -- if you use natural materials from around the yard, beach, or gravel pit, as some have suggested, you might want to go through it (after sifting it down to size) with a magnet to remove any magnetic metal particles that could work loose and cause horrible engine failures later.
HTH