Worried about my O.S. .40 FX

The OS 40 FX is my favorite engine. I'm very much afraid you have ruined yours. Let's go through your symptoms and their causes.
The engine is running very hot, because it isn't being cooled properly. Cooling comes from two sources: the lubricant in the fuel and the air flow over the engine. Let's discuss the lubricant first. If the engine is run at a very lean setting, then it will get very hot, since it isn't getting enough lubricant. The lubricant helps carry away heat plus it reduces friction. The cure for the second problem is obvious.
I suspect the fuel and/or the needle valve setting is your problem. The first thing I would do is get some fresh fuel. Make sure that it's at least 18% lubricant. In general, some castor oil content is a good idea to protect your engine against lean runs, but you don't need 20%. BTW, the black exhaust is the Castor Oil getting cooked!
Next take out the needle valve and make sure there isn't any blockage. Also make sure nothing is blocking the fuel lines. Any fuel restriction will cause leaness. Next replace the NV and back it out 1.5 turns from a seated position and start the engine. Let it idle and examine the exhaust. Run the engine up a few times and note the exhaust. It should have a thick cloudy mist which is a sign of being overly rich.
Now turn the NV in two clicks and run the engine up. Each time detect the change in RPM. We aren't striving for the optimum NV setting but just a point where the engine runs well enough you can fly. Now fly the plane and see how it performs. If the engine is still too hot, you're going to need a rebuild. The black exhaust should have disappeared with a lower Castor content.
If everything is OK, you can tune the engine. Continue leaning it out until it just stumbles and then back the NV out five clicks.
Regarding a rebuild, OS parts are fairly expensive, and I believe that you'll be better off getting a new engine. With the new one, thoroughly read the manual and follow the break in instructions - no flying out of the box!
Ciao,
Mr Akimoto

"Black" exhaust is rarely cooked castor, also occurs with straight synthetic fuels. = aluminium alloy rubbing on aluminium alloy e.g. slightly movement in vibrating muffler, use of starter motor pushing crank pin back against backplate. Cooked castor is usually dark brown. No 1 cure is to make a gasket from cardboard. Press a piece of grey card from a cereal packet or similar against exhaust port and use hole punch to cut the bolt holes and a row of holes across the exhaust slot - cut a diagonal corner to corner across the slot. Bolt muffler to motor then use sharp knife to cut excess card around the exhaust header = easier than trying to cut a thin edged gasket before fitting. When motor starts, the excess card in slot will blow out into header, swell with oil and prevent the "Blacks". Another point of rubbing will be the prop drive washer when prop overtightened, especially if the drive washer is not a firm fit to shaft. For confirmation, refer to many of the links on my web page under motors, tuning etc. regards Alan T. Alan's Hobby, Model & RC Web Links

AT may have a hobby shop, but in general, he doesn't know what he is talking about. Yes Castor oil is normally dark brown, but you're running your engine in an extremely lean state with a fuel with extraordinarily high castor content.
To prevent the leakage at the muffler-engine flange, just use some silicone sealant.
I might also add that a carb teardown to look for dirt might be a good idea. OS carbs are pretty easy to disassemble, but the have some tiny parts which will be impossible to find in your rug.
Ciao,
Mr Akimoto

Akimoto said: "Yes Castor oil is normally dark brown ... ."
The Yahoo Encyclopedia says: "castor oil, yellowish oil obtained from the seed of the castor bean... ." But what would they know?
Tom

Yellowish in it's unburnt state. After it has been through a model engine at the high heats they cause, it comes out a varying colors of brown depending on how hot your running your engine. Almost black is way too hot. Way too hot is excessively lean.

Ed -
Running straight castor lube is a good clue. The engine is running hot because the cylinder wall is coated with a varnish composed of cooked castor oil. The varnish effectively insulates the piston from the cylinder wall and cooling fins, so the whole heat transfer mechanism designed into the engine gets screwed up. It's a self-sustaining cycle, as the heat causes more varnish buildup, and so on. Pull the cylinder and give it a good scrubbing with a plastic scotch-brite pad. If that doesn't remove the brown residue and leave a bright chrome surface, use some fine steel wool and be sure you abrade the cylinder wall circumferentially, not up and down. Do the same with piston, which will probably be browned on the skirt, and not near the crown. You don't want to remove any metal near the top of the piston, so try not to abrade it. Reassemble (after washing with hot soapy water), and use a fuel that has a lube blend of castor and synthetic, with emphasis on the latter. Castor is a good thing in ABC engines to prevent damage if you have a lean run, but just a few % is enough.
Abel

AT is completely correct. In order to turn the excess castor to black, you would have to be running the engine lean to the point of melting the aluminum. Castor can easily survive sustained temps in excess of 800 deg F. Your engine can't.