I've seen 10:1 or 15:1 worm gears backdrive pretty easily. Cone Drive claims that ratios over 40:1 usually will not in the absence of vibration, but under certain conditions even a 100:1 gearset may overhaul.
If I'm not mistaken the physics comes down to a couple of inclined planes in which the tangent of the minimum angle would be equal to coefficient of friction between the two materials. My "Marks' Standard Handbook for Mechanical Engineers" lists the coefficient of friction for steel on steel from 0.013 to infinity depending on coatings, and atmosphere.
It goes on to list coefficients of static friction of hard steel on hard steel as 0.78 for dry and:
0.0052 stearic acid 0.0075 palmitic acid 0.11 oleic acid or lard oil 0.23 light mineral oil 0.15 castor oil Assuming your not using the first two, we can pick the 0.15 as middle ground and that gives us an angle of 8.5 degrees.
If I'm way off base here, I'm sure somebody will chime in.
This depends on the material(s) involved with the two parts and their static coefficient of friction. You can look these coefficients up in a table, Machinery's Handbook has one. The coefficient is merely the tangent of the maximum angle that you can put the two pieces together and have them stick. One of the classic high-school physics experiments is to take a wooden block and a plank and figure out the coefficient of friction by tilting the assembly until the block slides and then measuring the angle. So, with a little button-twiddling on the calculator for some trig functions, you can figure the angle of the worm needed and then figure the worm's pitch. This also gets complicated by the lubrication needed, friction values vary a lot with different lubricants. So, to start with, you need to have some idea what you want to make your parts from and what you want to lubricate them with. A basic second-year engineering mechanics problem. Or you can see if you can find a suitable small winch(which would be my approach, let somebody else do the engineering). Some gear engineering books also would have some calculation techniques.
.... which is some 1500 km away from where I am, alas. The materials would likely be a bronze alloy for the wheel and s/steel for the worm, most likely 304 because I hate machining 316. Especially threading it.
I'd do that if I could find something close to reverse-engineer. There's plenty of time on this project tho, so I can keep looking. Indeed if I get really bored I can make some bits up and see for myself. One of the aims was to avoid having to use pawls to stop reverse travel. If I had to, I could compromise on this but you know how it is - the chances of part failure increase as an exponential factor of the part count......
As I remember Boston Gear mentions this in their catalog. They do have a web site with a series of articles on gearology as they call it. I don't have a fast internet connection so left the reading to you. As I remember a two or more lead worm will let the wheel drive the worm at a higher ratio than a one lead worm.
Handbook guidance based on experience would be best, but you could do a simple experiment. Make two blocks of the materials you'd use, e.g, bronze and stainless, with surface finish comparable to that you'll achieve on your parts. Lubricate them with whatever you'll lube the worm and gear. Place one block on top of the other, and tilt it up until the top block starts sliding. Measure the angle from horizontal. This would be the helix angle of a gear and worm that could backdrive. You're measuring coefficient of static friction, a ratio of sliding force to normal force (weight in this case). Since it's a ratio, the actual force doesn't matter: more normal force will produce correspondingly more friction.
The tangent of the helix angle would be the reciprocal of the number of teeth in a gear driven by a single-lead worm. (I think) Example: if the angle is 5 degrees, tan(5 deg) = .0874, number of teeth (and ratio) is 11.4 :1 so the next integral ratio (12:1) would be self-locking with given materials, finish and lubrication.
If vibration will be present then you should vibrate or "nudge" your experiment becuase running friction is always less than static friction and vibration can "break things loose". Then the angle would be the angle at which the block stops sliding after being given a nudge to get it moving.
I think I made an error in previous post. I think the pitch angle is just a function of the diameter and pitch of the worm. The ratio would depend on the diameter of the gear of same pitch, which could be anything from infinity down to some lower practical limit. Therefore, neglecting other friction as in bearings, it now seems to me that the factors determining self-locking or not would be coefficent of friction and pitch angle, but not necessarily ratio.
The pitch angle is arctan(1/(pi * wormdiameter*pitch), regardless of the diameter of or number of teeth in the gear.
If the pitch angle is the same as or less than the angle of non-slip in the friction experiment, I think the system would not t backdrive.