It depends on if you want (or need) the benefits that the primer affords. The primer will fill in a few basic seams (nothing major). It also will give you a solid color so you can tell if you've got any particular seam sanded/puttied out or if you need to give it a bit more attention.
The other main thing, is with paint adhesion. Now, if you're using enamels then this likely will not be a problem for you. If using acrylics though, you might run into some problems. Certain paints (like Testor's Acryl or the craft type paints at Wal-Mart or Michaels) will not stick well to smooth plastic, and will peel right off if you place masking tape on them. If you notice this happen, then you will need to prime the model before painting it. Now, personally, I tend to use quite a bit of the craft paints (they're pretty quick and cheap, and when thinned right produce a good finish). For that reason I prime every model I build. Not necessarily every part, but any major assembly. After the primer coat the paint will not pull off so easily and I can paint as needed.
Light or dark colored primers also will have a very small effect on the warmth of the paint job after finishing, but it's not something I have noticed to any great degree (I use the standard neutral gray primer though which I doubt would have any effect).