I'm guessing that you end up freezing the xref insertion layer, and
therefore make the entire thing disappear.
I have a "point and shoot" lisp routine to freeze layers in a floating
viewport, which has the options related to subentity layers etc. It's called
Click to see the full signature.