I was busy scribing off a piece of sheet metal for cutting and bending when it occurred to me I was WAAAAAAAAY over thinking the problem. Sure the little pin vises with a decent knob of handle will still be useful, but I don't need near as many as I was thinking. There in my hand was the solution to all the sizes I wanted to just leave chucked up.
Some time ago I made a couple carbide scribes by pressing broken 1/8" shank end mills into a hole in the end of a piece of aluminum rod. Then I ground a conical point on the piece of carbide,and knurled the rod. I use them all the time. For the 1/8" drill bit, and the 1/16" drill bit I use all the time on the tackle making bench I can just make some more aluminum rods like the ones I used for the scribes and add a set screw to make the bits replaceable.
Yeah a couple pin vises with tiny chucks would still be handy, but not as many as I was thinking. Cost... well I do sometimes find myself standing around watching machine run. I have lots of aluminum scrap.