Can't speak much to MTH since I use TMCC and bypass MTH engines, but, I think folks should review O gauge catalogs for the last several years. Three rail manufacturers usually make clear distinctions. (The exception, for me, was K-Line.) Granted that tinplate might be considered to be more toy like, but much non-tinplate, 1:48 (O gauge/quarter scale) isn't and is big enough for someone to see the amount of detail. My only real problem with Atlas O is that some of their offerings are so detailed that they're a bit fragile. I also tend to remove truck chains from engines since they've been known to come loose and get caught in gears. You also need to be concerned that not all three rail is 1:48. Car length is frequently given. As an example 1:48 intermodal is substantially larger. I believe the smaller items are frequently 1:64. Lionel's JLC steam should warm the heart of any rivet counter---and mine have been excellent runners.
I also can't speak to the virtues of HO since I've no experience with it. Mine has been with N scale and O gauge.
The O gauge manufacturers I with which I have experience are Lionel and Atlas O for both engines and rolling stock, and MTH, Weaver and Williams for rolling stock. I do have some Weaver motive, but that's a bit limited. My only Williams and MTH motive has been modified for TMCC. I also have several 3rd Rail pieces for CB&Q steam. Golden Gate Madison cars are extremely detailed--and a few do take up a lot of track. Fortunately, I've been able to convert an 18'x30' space to house the layout. It's got about 450' of track. So this does demonstrate that in order to run O gauge, you need a bit of room.