Two kinds of search, one on borrowed time
I keep a study desk: reading notes, course material and a few thousand saved passages, with a search index over all of it. The index stores two things for every passage. One is the text, which powers ordinary keyword search. The other is a numerical fingerprint of the passage's meaning, which lets a search find the right note even when it shares no words with the question.
The fingerprints had been computed by a paid outside service, and that service had announced the date it would shut down. Keyword search does not depend on it, so everything appeared to work. It would have gone on appearing to work, with half of the index dead underneath. This kind of failure does not crash. It rots.
The plan was straightforward: switch to a model running on our own machine that produces fingerprints of the same size, so that the database did not need to change, discard the old ones, and recompute them all.
The connection dropped halfway
The migration script had safety checks at the start and took backups before changing anything. Partway through the first run, my wifi dropped and took the session with it. When I ran the script again, it refused to start: the settings were no longer in the state it expected.
My first guess was that the dropped connection had somehow confused the check. That was wrong; the check only reads a local file. The first run had got as far as changing the settings before it died, so the second run's own safety check was correctly refusing to begin again from the top. The refusal was the script working. The proof of exactly how far the first run had got was sitting on disk: the backup it had written before it was cut off.
A result that looked perfect and meant nothing
We resumed from the right step, and the log reported completion. A test search returned excellent results. One line in the same log disagreed.
Passages with a usable fingerprint
The results looked perfect. The count said zero. I believed the count.
A lock that nobody was holding
The log also explained why nothing had been computed: the tool believed another run was already in progress and had politely stepped aside. No other run existed. The first, interrupted run had left its "in progress" marker behind.
We looked before touching anything. The marker was not a file, which is where I would have searched. It was a row in a database table, with an expiry time. No process was running that could own it, which made it stale by definition and safe to remove. By the time we went to remove it, it had expired by itself.
The clean finish
The final run was started in a way that left it independent of my connection, with its log written to a file, so that a second dropped signal could not interrupt it. It recomputed 832 notes, 2,000 passages, in 2 minutes and 18 seconds on our own hardware, for nothing.
This time it was verified by the count and not by the look of the results: 2,000 passages, 2,000 fingerprints. There was one last scare, when the script appeared to hang on a table of results. It was only a display tool waiting for me to press a key.
What I took from it
- Verify by count, not by appearance. A fallback can hide an empty store perfectly.
- A refusal from a safety check is information. Find out what it knows before working around it.
- Before removing a lock, prove that nobody holds it.
- Work that a dropped connection can kill should not be attached to the connection.
"I admit that twice two makes four is an excellent thing, but if we are to give everything its due, twice two makes five is sometimes a very charming thing too."
Previous: The dryer is doneNext: The rules the AI cannot argue with