The one-line diff that mattered
An early version of the comparison code that scores an uploaded file against everyone else's files looked like this:
scores = [compare_files(upload, base) for base in group_files]Nothing about that line is wrong. It reads as the obvious implementation, and it produces the correct similarity scores. Against a group of a thousand files it also took 44 seconds, because compare_files did something the line hides completely: it built a brand new TfidfVectorizer, fit it on exactly two documents, computed cosine similarity, and threw the whole thing away - a thousand times, once per base file, to answer one question.
The fix wasn't a faster algorithm. It was recognizing that a fitted vectorizer is a fact about a corpus, and a corpus doesn't change on every request.
What "fitting" actually does
TfidfVectorizer.fit_transform does two things that have nothing to do with any single document: it builds a vocabulary across everything it's given, and it computes document frequencies - how many documents each term appears in - to derive the inverse-document-frequency weight for that term. Both of those numbers are properties of the whole collection. Fitting on two files doesn't just waste the work of fitting on a thousand; it produces a different, degenerate answer, because every term in a two-document corpus appears in either one document or both. There's no middle ground for IDF to discriminate on. A word that happens to appear in both notes gets the same near-zero weight as "the" would in a real corpus, whether or not it's actually distinctive.
So the naive loop had a correctness problem sitting inside its performance problem, and fixing the second one accidentally fixed the first.
Two things that look like the same fix, and aren't
The obvious next step is to fit once instead of per-pair:
vectorizer = TfidfVectorizer(ngram_range=(1, 2), sublinear_tf=True)
matrix = vectorizer.fit_transform(all_texts) # upload + every base file, once
scores = cosine_similarity(matrix[0:1], matrix[1:])[0]That's a real fix, and on its own it's roughly a 9x improvement over the pairwise loop at a thousand files - from 44 seconds down to about 5. It's also still doing the wrong amount of work for what the request actually needs, because it refits the vocabulary from scratch on every single upload, even though the group of existing files hasn't changed since the last one.
The distinction that matters is between the corpus and the query. The base files a new upload gets compared against are the corpus - they define the vocabulary and the IDF weights. The upload itself is a query against that corpus. Nothing about scoring a query should require rebuilding the thing it's being scored against.
Building the index once
FileIndex in filemetric-engine is immutable on purpose. It's built once per group of files, holds the fitted vectorizer and the resulting document-term matrix, and answers queries without touching either again:
class FileIndex:
"""Immutable TF-IDF index. Build once, query many times.
Each query is a single sparse matrix-vector multiply."""
@classmethod
def build(cls, base_files, cache=None, ...):
texts = _read_all(base_files, cache)
vectorizer = TfidfVectorizer(ngram_range=(1, 2), sublinear_tf=True)
matrix = vectorizer.fit_transform(texts)
return cls(vectorizer, matrix, [f.path for f in base_files])
def query(self, cleaned_text, top_n=10, threshold=0.0):
query_vec = self._vectorizer.transform([cleaned_text])
scores = cosine_similarity(query_vec, self._matrix)[0]
...The whole trick is in one word: transform, not fit_transform. The vectorizer already has its vocabulary and its IDF weights locked in from the build step, so scoring a new upload just projects it into that existing vector space and runs a single sparse matrix-vector product against the prebuilt matrix. There's no vocabulary construction, no document-frequency pass, no allocation of a fresh sparse matrix - the expensive O(n · L) work already happened once, at build time, and every query after that is just linear algebra against a fixed basis.
Measured on synthetic study notes against a thousand base files: the naive per-pair loop took 44.2 seconds, hoisting the fit out of the loop brought that to 4.9 seconds, and querying a prebuilt index took 59 milliseconds. That's a 744x gap between the first version and the last, and the difference between the middle version and the index isn't a smarter formula - it's the same math, minus the part where the corpus gets rebuilt for no reason.
The part that isn't free
Building an index once instead of per-request only works because the underlying assumption holds: the corpus doesn't change on every request. In a document-comparison feature, it does change - just not on every request. Someone uploads a new file, and now the corpus the index was built from is stale.
Refitting the whole index on every single upload puts the exact cost back into the request path that the index was built to avoid. filemetric-engine handles this by treating new uploads as a small pending buffer that gets merged into the main index every m additions rather than every one, so an upload is an O(1) append and the O(n · L) rebuild is amortized across a batch instead of paid in full on each request. Queries in between run against both the built index and the pending buffer and merge the results, so nothing in the buffer goes unsearched while it waits for the next merge.
That's a separate design decision from the one this post is about, but it exists for the same reason: the index is only fast because it defers work that doesn't need to happen synchronously. The buffer is what keeps that assumption true when the corpus is allowed to grow.
What actually generalizes
The specific numbers here are TF-IDF and cosine similarity, but the shape of the mistake isn't specific to text search. Any time a request handler recomputes something from a fixed dataset - refits a model, rebuilds a lookup structure, re-derives statistics - on every call instead of once when the dataset changes, the fix looks the same: separate "things that depend on the whole corpus" from "things that depend on one query," build the first once, and make the second cheap. The naive loop wasn't slow because TF-IDF is expensive. It was slow because it kept answering a question about the corpus that had already been answered a thousand times before, and just never wrote the answer down.