Search-methods appendix · 1820–1963

What does a search
let us see?

Different searches bring different stories into view. Follow a question, compare the results, and read the evidence.

46,449 OCR passages24,346 saved page identifiersSelected by identity words · not all reportingHow this was made ↗
Follow a question

Loading the computed map. The source below is ready to read.

A landscape of language

Nearby passages use related language.

One dot = one OCR passageLanguage similarity, not geography

Map labels open example searches; they are not topic boundaries. The source list below provides another way to explore.

Places to begin

Choose a search above to compare actual rankings.

    Three ways to look closer

    The search is part of the story.

    A school can be far from Mississippi.

    The top result for our schooling question mentions California. A related subject does not establish a local connection.

    Repetition can look like abundance.

    A named farmer brings us back to versions of one account. Open the inspected printings and compare their words.

    What the editorial did, and what these results mean

    A real, saved model run

    MiniLM encoded all 46,449 passages into 384-dimensional vectors. The map projects those vectors into two dimensions with UMAP. Search rankings use the original vectors, not distances on this screen.

    Five fixed questions were run through the model and through word TF-IDF. Each method returns its first 100 distinct saved page keys. Switching methods displays those actual rankings; it does not ask a chatbot to invent results.

    The same saved run was asked for the nearest passages to each document in the places exhibit. Those candidates are kept here, in the appendix, rather than beneath a document’s transcription: similar wording is a lead to follow, not a related record. A passage from a page the exhibit already publishes for that document is suppressed, so an advertisement cannot fill its own list with its own reprints.

    Download the run and checks · Nearest passages per placed document · Model documentation

    A tool for reading, not a historical census

    The inputs are short, unreviewed OCR windows from an identity-word search, including historical terms with other meanings. They can mix adjacent items, contain recognition errors, and repeat the same account. A dot is not a verified article or person.

    We tested 16 model groups with two starting seeds. Their assignments differed substantially (adjusted Rand index 0.384), so we do not present fixed topic territories. The map is one saved exploratory projection.

    Neither ranking is human-validated here. Results elsewhere in the world can match a Mississippi question. This run does not measure public opinion, historical prevalence, or recovery beyond the original identity filter.

    Two complete views of the same newspapers do not agree

    A separate, completed pass read the official bulk OCR for all 39 Mississippi batches, 1850–1963: 306,555 pages, of which 23,320 carry one of ten identity words. Against the same batches, a partial run of the Chronicling America search index returned 3,314 pages, and only 1,370 of those appear in both. 1,944 pages came back from the index that a literal word match over the distributed OCR never flagged.

    The index run covered only part of its universe, so this does not show that a search misses most pages. It shows something narrower and harder to ignore: which pages you can find depends on which copy of the text you search. 1924, 1925, 1926 contain no matching page at all in these batches, the years of the Rosedale case. That is a fact about what has been digitised, not about what was printed. Newspaper titles in this pool are still provisional: 23 identifiers on matched pages are absent from the inventory's own title lists, and the registry that would settle them refused the request.

    Read from the corpus worktree on 2026-09-13; the artefacts and their hashes are listed in data/coverage-inventory.json. A fair test of these search methods would need reading every page, by people, in a blinded queue. That has not been done, so no comparison of search methods is reported here.

    Original source