How do you keep an agent's room notes valid when the map shifts under it?

You're not overthinking it, raw coordinates are just the wrong storage format for long-lived notes. I hit the same wall with a little wheeled bot I run at home, and the fix was to stop treating the map as the memory. Now the notes live in thier own store and each one is anchored to somthing stable: a room name, plus a landmark and a rough offset from it, like 'left of the fridge, against the wall' or 'second shelf of the hall closet'. At query time the agent resolves that anchor against whatever the current map says the fridge or closet is, and only then gets coordinates to navigate to. When the map rebuilds and shifts everything, the notes don't notice beacuse they were never tied to the global frame in the first place.

The trick is picking anchors that don't move often. Furniture is decent, walls and doorways are better, room names are best of all. A note saying 'in the kitchen, near the doorway' survives a remap with zero effort. A note saying 'at 2.34, -0.87' dies the moment the map shifts. The other half is accepting that some notes go stale anyway and building the re-look behavior you mentioned. I give each note a last-verified timestamp, and when the agent fetches something and finds nothing there, it doesn't just fail, it updates the note to 'missing, search nearby' and does a little sweep of the room before giving up. Anything it finds during the sweep gets a fresh anchor and a fresh timestamp. That way the memory self-heals instead of quietly rotting. Storing nightly scans really is wasteful, and honestly unnecessary. If your anchors are semantic, a remap is almost free: worst case a note says 'near the couch' and the couch moved three feet, and the verification pass catches it on the next failed fetch. The combination of relative anchoring plus lazy re-verification has held up for months on mine without me re-telling it anything.

One thing I'd add is that a remap also quietly invalidates anything else you saved in the old frame, not just the notes, so any saved dock approach angles or favorite parking spots need the same treatment or they'll drift along with everything else. I also found it helps to log which map version a note was written under, because when a fetch fails that tells you wheather the object moved or the map moved, which are very different problems. Question for the relative-anchoring approach though: what do you do with objects that are mobile by nature, like a laptop or a mug? Those seem to want a 'usually here' style note rather than a fixed anchor.

Been running something similar on my garage bot for about six months, maybe 45 notes now, and the one thing that saved me was attaching a cropped camera snapshot to each note when it gets written. When the agent goes to fetch something, it compares what it sees against the stored image instead of trusting coordinates blindly. Caught a case last month where the bot was standing exactly where the map said the toolbox should be, and it just wasn't, because my kid moved it. Snapshot matching plus a room-level search radius handled the recovery on its own. Does your setup have a camera in the loop, or is it lidar only? That changes a lot about what anchors you can actually verify.

Interesting reading this as someone who went a slightly diffrent route with my rover. I skipped the fancy stuff and just stuck a few visual markers around the house, like a colored sticker on the pantry doorframe, and anchored my notes to those. Crude but it survived every remap so far. My worry with the semantic anchor idea is the agent misidentifying the landmark in the first place, like confusing the hall closet with the coat closet since they look alike. Has anyone had issues with the anchor itself resolving to the wrong spot? Maybe that's just a camera-only problem, I don't know yet.