The index solved reads and broke writes
A prebuilt TF-IDF index makes querying near-instant: fit the vectorizer once over the whole group of files, then score a new document by projecting it into that fixed vector space and running a single sparse matrix-vector product. That is what gets a file-comparison query from 44 seconds down to 59 milliseconds against a thousand files.
It also creates a problem the naive per-pair loop never had. Adding one file to the group is not a cheap update to that index - it is a reason to rebuild it. Inverse document frequency is a property of the whole corpus: a term's weight depends on how many documents in the group contain it, and a new document can shift that count for every term it introduces. There is no incremental way to patch document frequencies for one addition without touching the whole matrix.
Refit on every upload and the O(n · L) cost that indexing was built to remove from the read path lands squarely back in the write path instead. Every upload becomes as slow as a cold index build.
Two tiers, not one rebuild policy
DynamicFileIndex splits reads and writes into separate structures. A built FileIndex answers queries the fast way. New uploads land in a plain list - the pending buffer - and stay there until enough of them have accumulated to justify a rebuild.
def add_files(self, paths, force_merge=False):
self._pending.extend(str(p) for p in paths)
merged = False
if force_merge or len(self._pending) >= self.merge_threshold:
self._merge()
merged = True
return mergedAn upload is a Python list append: O(1), no vectorizer touched. The expensive part - refitting across the whole group - only runs when self._pending reaches merge_threshold, which defaults to 50. force_merge exists for the one caller that cannot wait: a query that needs the buffer folded in before it runs, rather than merged and queried separately.
What "amortized" costs in milliseconds
At a thousand files, a cold index build measured at 6.57 seconds. Spread that rebuild across the 50 uploads that triggered it and the amortized cost per upload is about 131 milliseconds. Compare that to the unamortized alternative - refit on every single upload - which pays the full 6.57 seconds fifty times over for the same fifty files.
The batched version does not make the rebuild cheaper. It makes it rare. Forty- nine uploads in every fifty pay nothing but an append; the fiftieth pays the full bill. Average latency per upload drops by roughly two orders of magnitude, at the cost of one upload in fifty being visibly slower than the rest.
That is the trade a merge threshold is actually making: not a reduction in total work, but a redistribution of it, away from a synchronous cost users feel on every request and toward an occasional cost that shows up on a schedule you control.
Queries can't wait for the next merge
A buffer that only gets searched after it merges would make new uploads invisible to comparisons for up to 49 uploads at a time, which defeats the point of a near-duplicate detector. So a query has to check both tiers, every time, regardless of merge state:
if self._index is not None:
result = self._index.query(main_file, cache=self.cache,
threshold=threshold, sort=False)
all_matches.extend(result.compare)
if self._pending:
buf_result = compare_one_to_many(main_file, self._pending,
cache=self.cache,
threshold=threshold, sort=False)
all_matches.extend(buf_result.compare)
if sort:
all_matches.sort(key=lambda m: m.percentage, reverse=True)Both branches pass sort=False and the combined list sorts once at the end. Sorting each tier first would be wasted work on lists that are about to be concatenated, and it hides a sharper bug: applying top_n inside either branch before the merge could drop a strong match that happened to sit in the smaller tier, just because it lost a local ranking that never should have been computed.
The buffer is not scored the same way
The main index queries with a single sparse matrix-vector product against a fixed, already-fitted vocabulary. The pending buffer has no such thing yet, so it falls back to compare_one_to_many - a fresh vectorizer fit over just the buffer's contents for that one query.
That is a deliberate asymmetry, and it is worth stating plainly rather than burying it: a file sitting in the pending buffer is scored against a vocabulary built from whatever else happens to be in the buffer at that moment, not against the full group. Two files with identical content could score slightly differently depending on whether one has already been merged into the main index and the other is still waiting. The scores converge again after the next merge refits everything on the same basis.
Lower the merge threshold and that inconsistency window shrinks, at the cost of triggering the expensive rebuild more often. Raise it and uploads stay cheaper for longer, at the cost of buffer-tier scores drifting further from what the same file would score once merged. The threshold is the one dial that trades between those two costs directly.
What generalizes past TF-IDF
The pattern here is not specific to text indexes. Any data structure that is fast to query because it was expensive to build - a sorted index, a compiled lookup table, a trained model - faces the same choice on every write: rebuild immediately and pay the full cost on the critical path, or buffer and rebuild on a schedule. Buffering only works when three things hold: writes can be served from a cheap fallback structure while they wait, reads check both the built structure and the buffer so nothing goes temporarily invisible, and there is an explicit escape hatch - here, force_merge - for the caller that cannot tolerate the staleness window at all.
Get any one of those three wrong and the buffer either loses data, serves stale reads silently, or turns back into the thing it was built to avoid. Get them right and the expensive operation stops being something every write pays for, and becomes something a batch of writes pays for once.