The score that looked plausible
Early filemetric-engine occasionally returned near-duplicate scores that looked reasonable in isolation and wrong side by side - two uploads that shared nothing but boilerplate would sometimes rank close to two uploads that were genuine near-duplicates. The cause was not a formula bug. It was that the formula was being handed a corpus of two documents, and a corpus of two documents cannot produce the number inverse document frequency is supposed to produce.
What IDF has to work with
scikit-learn's default smoothed IDF is:
idf(t) = ln((1 + n) / (1 + df(t))) + 1where n is the number of documents in the corpus and df(t) is how many of them contain term t. The formula exists to spread weight across the possible values of df(t): a term that appears in three documents out of a thousand should score far higher than one that appears in five hundred out of a thousand. That spread is the entire mechanism by which "distinctive" gets separated from "common."
A corpus of two has no range
Compare one uploaded file against exactly one base file, and fit a vectorizer on just those two documents. Every term in the vocabulary now has exactly two possible values for df(t): 1, if it appears in only one of the documents, or 2, if it appears in both. Plug both into the formula with n = 2:
| Term appears in | df(t) | idf(t) |
|---|---|---|
| both documents | 2 | ln(3/3) + 1 = 1.000 |
| one document | 1 | ln(3/2) + 1 = 1.405 |
That is the whole range. No matter how common or rare a term genuinely is across the group's real corpus - whether it's "the," appearing in every file, or a specific phrase that appears nowhere else - inside a two-document fit it can only land on one of two values, and the more discriminating of the two is never more than 1.4x the weight of the less. A word that is a genuine duplicate signal and a word that is filler compete on close to even terms. The formula is not wrong. It is answering a question that the input contains no information to answer.
Where that shows up as a false positive
Near-duplicate detection lives or dies on that spread. Two documents that share course boilerplate - headers, a disclaimer, the professor's name, a citation format - but differ in actual content should score low, because the boilerplate terms are common and the content terms are rare, and it is the rare terms that should dominate cosine similarity. Fit over two documents and that ordering collapses: boilerplate terms and content terms that happen to appear in both files get nearly identical weight, since both have df = 2, while a term unique to one file is barely ahead. The similarity score stops tracking shared content and starts tracking something closer to shared length.
This is easy to miss in testing, because it does not crash and it does not misfire on every pair. Two files with almost nothing in common still score low, since there is little for the degenerate weighting to inflate. It shows up specifically on pairs that share moderate boilerplate against a background of real, different content - which is exactly the shape of two students' notes on the same lecture.
The fix was already the performance fix
filemetric-engine fits TfidfVectorizer once, across every file in the group, not once per pair:
vectorizer = TfidfVectorizer(ngram_range=(1, 2), sublinear_tf=True)
matrix = vectorizer.fit_transform(all_texts) # every file in the group, one fit
scores = cosine_similarity(matrix[0:1], matrix[1:])[0]At n = 1,000, df(t) can land anywhere from 1 to 1,000, and idf(t) spans roughly 1.7 at the most common end to 6.9 at the rarest. That is the range the formula needs to do its job: a term that is genuinely distinctive across the group gets a weight several times that of a term everyone uses. The same change that turns the naive pairwise loop from seconds into milliseconds - fitting once over the group instead of once per pair, covered in the indexing post - is also the change that makes the near-duplicate scores mean something. That was not a lucky side effect. A fitted vectorizer only produces a meaningful IDF when it is fit on the population the question is actually about, and the group is that population. The pair being compared never was.
The corpus you fit on is a claim
The general version of this: any weighting scheme defined relative to a population is only meaningful if it is actually fit on that population, and "the two things I'm currently comparing" is a population of two whether or not the code calls it a corpus. It is tempting to treat the base file and the candidate as the whole world for the duration of one comparison, because it is the only data available at that call site. But IDF over two documents is not a smaller, cheaper version of IDF over a thousand - it is a different, less informative number wearing the same variable name. Fitting late, on the pair, instead of once, on the group, does not just cost more CPU. It throws away the information the group was providing in the first place.
Where the floor still has to come from
Fitting over the full group fixes the degeneracy, but it reintroduces a smaller version of the same question in filemetric-engine's pending buffer: new uploads are scored against a vectorizer fit on the buffer alone until the next merge, and a buffer of a handful of files sits closer to the two-document case than the thousand-document one. filemetric-engine's answer there is a similarity threshold rather than a raw ranking - a match still has to clear a bar, not just outrank whatever else happens to be in a small, possibly degenerate set of alternatives. It is a narrower version of the same lesson: the smaller the corpus a score is fit on, the less that score is measuring what it looks like it is measuring.