It depends on the child. At 4, my son had his own 2x4 foot N scale layout (a double track circle with crossovers and a couple of internal spurs) with Atlas block controls and a MRC twin controller. He could operate two trains at a time by himself (under supervision of course).
The important thing was that he understood the difference between his toy trains (the diecast Thomas toys) and his model trains. If the child can understand that difference, he's capable of dealing with the models.