3.2 Text Data
In 1787 and 1788, eighty-five essays appeared in New York newspapers arguing for the ratification of the American constitution. They were signed Publius. Three men were behind the pseudonym: Alexander Hamilton, James Madison, and John Jay.
Jay's five essays were never in doubt. The rest were, and for a peculiar reason: both Hamilton and Madison later left lists claiming authorship, and the lists disagreed. Twelve essays were claimed by both men.27 For a century and a half, historians argued about them with the tools historians had — style, politics, biography, handwriting — and reached no settlement.
In 1963 two statisticians settled it, and the way they did it is the origin of everything in this chapter.
Frederick Mosteller and David Wallace reasoned that an author's fingerprint would not be in the words that carry the argument. Words like war, executive and legislature rise and fall with the topic, and the disputed essays were all about the same topics. The fingerprint would be in the words nobody chooses deliberately.
Their own term for these was filler words: articles, prepositions, conjunctions — words whose rate of use stays nearly constant when the subject changes. And the single best discriminator they found was the word upon. Hamilton used it about 3 times per thousand words. Madison used it about one-sixth of a time per thousand — roughly an eighteen-fold difference in a word neither man ever thought about.
From 165 candidate words they kept 30. Then they computed. All twelve disputed essays came out Madison. The weakest case, Federalist No. 55, still gave odds of 80 to 1 after the authors deliberately deflated their own estimate.
Hold on to the punchline, because this chapter is about to contradict it. The words that identified the author are exactly the words that every text-mining tutorial, including the one below, deletes in the second step.
3.2.1 Is Text Just a Long String in a Cell?
This book has argued from Chapter 2.1 onwards that rectangular data — one row per observation, one column per variable — is the workhorse structure of empirical research. Text seems to fit fine. A review is a value; put it in a cell.
So the honest question, before any technique: when is that enough, and when is it not?
The rest of this chapter is a ladder with four rungs. At each one, something becomes possible that was not possible before, and something is given up. The point is not that the top rung is best. It is that each rung earns its complexity by unlocking a specific kind of analysis — and if you do not need that analysis, you do not need that rung.
Our corpus is real, and it is small enough to see all the way through.
Where this corpus comes from. On 2 June 2022 a seminar travelled to the Humboldt Forum in Berlin to see the exhibition Berlin Global — an interdisciplinary excursion, economics students together with history students.28 Afterwards each economics student wrote a review and posted it publicly on TripAdvisor, addressing two questions: how globality is represented, and how the colonial past is depicted.
At the time the exhibition had almost no other reviews. The 46 reviews analysed here are therefore, in effect, one seminar's collective response — written for a public audience rather than for a grade sheet.
Every one of those facts is part of the data. Who wrote the texts, why, for whom, and under what prompt determines what the word counts below can and cannot mean.
3.2.2 Rung 0: Text in a Cell
Start with the simplest thing that could possibly work.
library(tidyverse)
library(tidytext)
trip <- read.csv("data/tripadvisor_berlin_global.csv")
dim(trip)
#> [1] 46 4
summary(nchar(trip$Text))
#> Min. 1st Qu. Median Mean 3rd Qu. Max.
#> 428.0 901.8 1416.5 1480.0 1920.2 2885.0Forty-six rows, four columns. Each review sits in one cell, between 428 and 2,885 characters. This is an ordinary data frame, and everything you already know applies to it.
A surprising amount of analysis is available at this rung, as long as the question can be answered by searching rather than counting:
sum(str_detect(tolower(trip$Text), "colonial"))
#> [1] 38
sum(str_detect(tolower(trip$Text), "global"))
#> [1] 42Thirty-eight of forty-six reviews mention the colonial past; forty-two mention globality. That is a real finding about whether an assignment was followed, and it required no text mining at all — just a regular expression against a column.
Definition
Text data is data whose values are sequences of characters carrying meaning in a natural language.
It is called unstructured not because it has no structure — language is enormously structured — but because its structure is not the structure of a table.
Why this is not enough. Ask a slightly different question: how often does each review mention the colonial past, and which words does it use to do so? Now you need the parts, and a cell has no parts. You cannot count words, compare two reviews, or match a review against a list of sentiment terms while the text remains a single string.
The obstacle is not the table. It is that the unit of observation you need — the word — is not the unit the table stores.
3.2.3 Rung 1: One Token per Row
The move is to change the unit of observation. One row per token.
Definition
A token is a meaningful unit of text — most often a word, sometimes a sentence, a character, or a pair of adjacent words (a bigram).
Tokenization is the process of splitting text into tokens.
tokens <- trip %>%
select(Author, Text) %>%
unnest_tokens(word, Text)
nrow(tokens)
#> [1] 11314
n_distinct(tokens$word)
#> [1] 2095Forty-six reviews have become 11,378 rows, drawn from a vocabulary of 2,053 distinct words. The table got much longer and much narrower, and — this is the whole point — it is still a table. Every verb from Chapter ?? works unchanged: count(), group_by(), filter(), anti_join(), and ggplot() on the result.29
That is why this format belongs in this book. It buys word-level analysis without asking you to learn a new data structure.
tokens %>% count(word, sort = TRUE) %>% head(5)
#> word n
#> 1 the 869
#> 2 and 400
#> 3 of 398
#> 4 to 275
#> 5 in 248And there is the problem Mosteller and Wallace built a career on, staring back at us. The five most frequent words are the, and, of, to, in. They tell us nothing about Berlin.
3.2.3.1 Stopwords, and a warning
The standard remedy is to remove them.
data(stop_words)
clean <- tokens %>% anti_join(stop_words, by = "word")
round(100 * (1 - nrow(clean) / nrow(tokens)), 1) # per cent removed
#> [1] 61.5
clean %>% count(word, sort = TRUE) %>% head(8)
#> word n
#> 1 berlin 133
#> 2 exhibition 107
#> 3 world 94
#> 4 global 75
#> 5 people 70
#> 6 globality 60
#> 7 past 58
#> 8 colonial 53Sixty-two per cent of every token in the corpus is a stopword. What remains is immediately interpretable: berlin, exhibition, world, global, people, globality, past, colonial. The class answered the assignment.
Now recall the first page of this chapter. Mosteller and Wallace identified the author of twelve contested political essays using upon, an, of and whilst — and we have just deleted 62 % of our data on the grounds that such words carry no information.
Both things are true, because they are answers to different questions.
Stopwords are not information-free. They are topic-free.
That is precisely what makes them useless for finding out what a text is about, and precisely what makes them ideal for finding out who wrote it. Function-word rates stay stable when the subject changes; content-word rates do not.
This generalises into the most under-appreciated fact about text analysis: preprocessing is not cleaning, it is modelling. Denny and Spirling identify seven common binary preprocessing steps — punctuation, numbers, lowercasing, stemming, stopwords, infrequent terms, n-grams — which combine into 64 or 128 distinct specifications, and show that the choice materially changes the results of real analyses on real data.30 For topic models specifically, removing stopwords after fitting works about as well as removing them before, and is more transparent.31
There is a historical footnote here that ought to be uncomfortable. Mosteller and Wallace kept their preprocessing decisions — how to treat quotations, numerals, hyphenation, foreign phrases — in a private notebook. It has been lost. The founding study of quantitative text analysis is, in the strict sense, not reproducible.32
Your Turn
The word globality appears in the cleaned top eight. It is not an everyday English word — it comes from the assignment prompt.
Does its frequency tell you something about the exhibition, or something about the instructions?
Now remove it, along with berlin and exhibition, and look at the next ten words. Does the picture change?
3.2.3.2 A second corpus, from the same students
The course questionnaire from Chapter 4.2 had one column we ignored there. Alongside age, degree and semesters, every student wrote a free-text answer to a single question: what do you expect from this course?
course <- read.csv(paste0("https://raw.githubusercontent.com/MarcoKuehne/",
"marcokuehne.github.io/main/data/Course/GF_AllTime.csv"),
sep = ";")
course <- course[trimws(course$Expectations) != "", ]
expect <- course %>%
select(Term, Expectations) %>%
unnest_tokens(word, Expectations) %>%
anti_join(stop_words, by = "word")
nrow(course)
#> [1] 233
expect %>% count(word, sort = TRUE) %>% head(10)
#> word n
#> 1 data 160
#> 2 learn 78
#> 3 analysis 48
#> 4 programming 43
#> 5 knowledge 41
#> 6 expect 27
#> 7 skills 27
#> 8 academic 26
#> 9 improve 24
#> 10 statistical 20Two hundred and thirty-three answers, and the ranking is almost poignant in its consistency: data, learn, analysis, programming, knowledge, expect, skills, academic, improve, statistical.
Note what this is. The same file gave us a rectangular data set in Chapter 4.2 and a corpus here. Text is often not a separate kind of study — it is the column everybody drops.
3.2.4 Rung 2: The Document-Term Matrix
Rung 1 answers questions about words. It does not answer questions about documents.
Which two reviews are most alike? Do the reviews fall into groups? Can a model be trained to distinguish them? Every one of these needs each document represented as a whole object that can be compared with another — and a long, thin table of tokens does not provide that.
The structure that does is a matrix with one row per document and one column per term.
Definition
A document-term matrix (DTM) has one row per document, one column per term in the vocabulary, and in each cell the count of that term in that document.
Because it discards word order entirely, this representation is known as a bag of words.
library(Matrix)
dtm <- clean %>%
count(Author, word) %>%
cast_sparse(Author, word, n)
dim(dtm)
#> [1] 46 1683
round(100 * (1 - length(dtm@x) / prod(dim(dtm))), 1) # per cent zeros
#> [1] 95.4Here is where the rectangular assumption of this book finally strains. Forty-six documents and 1,637 terms make 75,302 cells, of which 95.3 % are zero. A vocabulary is long; any single document uses a tiny slice of it.
Storing the zeros costs four times the memory of not storing them, and this is a toy corpus. At realistic scale — tens of thousands of documents, a vocabulary in the hundreds of thousands — a dense matrix is simply impossible. This is the moment where text stops fitting in a data frame and requires a sparse matrix, which records only the non-zero entries and their coordinates.
This is the same kind of moment as in Chapter 3.3, where coordinates, projections and shapefiles arrived because points on a curved earth do not fit in two ordinary numeric columns.
A new data structure is not a complication for its own sake. It is what happens when the question outgrows the table.
What the DTM buys is comparison. With documents as vectors, the angle between them measures similarity:
norm <- dtm / sqrt(rowSums(dtm^2))
sim <- as.matrix(norm %*% t(norm))
diag(sim) <- NA
round(c(median = median(sim, na.rm = TRUE),
max = max(sim, na.rm = TRUE)), 3)
#> median max
#> 0.243 0.587Typical pairs of reviews share about a quarter of their direction; the most similar pair reaches 0.59. From here the methods of Chapter 8.1 and Chapter 8.2 apply directly — a DTM is just a very wide, very sparse data matrix, and principal components and cluster analysis do not care that its columns are words.
What it costs is word order.
a <- "the exhibition was not good at all"
b <- "the exhibition was good not at all"
identical(sort(strsplit(a, " ")[[1]]),
sort(strsplit(b, " ")[[1]]))
#> [1] TRUETo a bag of words these are the same document. One of them is a complaint.
3.2.5 Rung 3: Embeddings
The DTM has a second limitation, subtler and more consequential than word order.
Every term is its own column, and columns are independent. In that geometry museum and exhibition are exactly as unrelated as museum and bicycle — orthogonal dimensions, sharing nothing. Two reviews saying the same thing in different words register as dissimilar. The matrix knows how words are spelled, not what they mean.
An embedding replaces the vocabulary-length sparse vector with a short dense one — typically a few hundred numbers — positioned so that words used in similar contexts land near one another. Museum and exhibition end up close; bicycle does not.
Notice where this leaves us. An embedding matrix is documents × dimensions: rectangular, dense, and perfectly ordinary to work with.
But its columns have no names and no interpretation. Dimension 47 does not mean anything.
That should sound familiar. It is Chapter 8.3 again — observed indicators explained by a smaller set of unobserved dimensions — with one difference that matters: in factor analysis the researcher names the factors from the content of their strongest indicators. In an embedding, nobody does. The dimensions are estimated, useful, and mute.
Because embeddings are learned from how people actually write, they absorb what people actually assume. Caliskan and colleagues showed this with unusual precision: the gender association a standard embedding assigns to an occupation term predicts the actual percentage of women in that occupation at r = 0.90, across 50 occupations, against official labour statistics.33
Read that twice. It is simultaneously a demonstration that embeddings encode real social structure — genuinely useful for measurement — and that they encode stereotype, because the model has no way to distinguish the two. The same number supports both readings.
3.2.6 Sentiment: A Method Worth Distrusting
With the ladder in place, one technique deserves close attention, because it is the most used and the most abused.
Definition
Sentiment analysis assigns a document a position on an evaluative scale — typically positive to negative — from its content.
The dictionary approach does this by counting words from prepared lists.
Rather than call a package, we build the smallest possible version by hand, so that nothing is hidden:
positive <- c("good", "great", "interesting", "impressive", "beautiful",
"modern", "fun", "informative", "recommend", "amazing")
negative <- c("boring", "bad", "confusing", "disappointing", "expensive",
"crowded", "difficult", "poor", "worse", "lacking")
scores <- tokens %>%
group_by(Author) %>%
summarise(pos = sum(word %in% positive),
neg = sum(word %in% negative),
n = n(), .groups = "drop") %>%
mutate(score = (pos - neg) / n * 100)
sum(scores$score > 0) # reviews scored positive
#> [1] 28
sum(scores$pos == 0 & scores$neg == 0) # reviews with no dictionary word at all
#> [1] 17Twenty-eight of forty-six reviews come out positive, which matches the impression of anyone who reads them. Then look at the second number.
Seventeen reviews — more than a third of the corpus — contain not one word from either list. For those students the method has no opinion whatsoever, and yet a score of zero was computed and will happily flow into a mean.
Enlarging the dictionary reduces this problem without solving it, and introduces a new one, which is that longer lists contain more words that mean something different in your domain. The classic demonstration is financial: of the words a general-purpose negative dictionary flags in corporate annual reports, 73.8 % are not negative in a financial context — tax, cost, capital, liability, depreciation.34
3.2.6.1 How well does it actually work?
This has been measured properly. Van Atteveldt and colleagues compared dictionaries, machine learning and humans against a gold standard of hand-coded news headlines.35
| Method | Krippendorff's \(\alpha\) |
|---|---|
| Human coder, single | 0.82 |
| Human coders, majority of three | 0.90 |
| Crowd coder, single | 0.75 |
| Deep learning (CNN) | 0.50 |
| Support vector machine | 0.41 |
| Best dictionary | 0.34 |
| Other dictionaries | 0.07 – 0.32 |
The gap is not marginal. Dictionaries do not perform slightly worse than humans; they perform closer to noise than to humans. A separate study found off-the-shelf tools agreeing with each other at only \(\alpha\) = 0.09 to 0.34 — they do not merely disagree with people, they disagree among themselves.36
Two specific failure modes explain much of this. Negation: handling it properly is worth roughly 15 F1 points on the sentences where it occurs, and a plain word count handles it not at all.37 Irony: after a shared task with 43 competing systems, on tweets deliberately collected to be ironic, the best system reached F1 = 0.71 — while a positive-word counter gets every ironic sentence exactly backwards.38
Dictionary sentiment is not a measurement of what people feel. It is a count of words from a list somebody else wrote, for a domain that may not be yours.
It can be adequate for aggregate trends over many documents, where errors partly cancel — correlations with human coding rise substantially when scores are averaged to the weekly level. It is not adequate for the sentiment of any individual document.
3.2.7 What About Language Models?
The obvious modern move is to hand the reviews to a language model and ask it to code them. This works better than dictionaries, and it fails in a way that is easy to miss.
It works: Gilardi and colleagues found ChatGPT exceeded crowd workers' annotation accuracy by about 25 percentage points across four datasets of tweets and news, at under $0.003 per annotation — roughly thirty times cheaper than the crowd platform.39
It fails: analysing 2,407 interviews with Rohingya refugees and Bangladeshi hosts, LLM coding reached F1 = 0.414 against 0.542 for a purpose-trained supervised model. The damaging finding is not the accuracy gap. It is that the errors correlated with respondents' household characteristics in 10 of 19 codes — the model was not noisy, it was biased with respect to exactly the variables the study was about.40
And there is a statistical result that should govern all such work. If you feed model-produced labels into a downstream regression, then even at 90 % annotation accuracy you can expect roughly 18 % standardised bias and about 58 % coverage from a nominal 95 % confidence interval. Raising accuracy to 95 % makes it worse, because higher accuracy invites larger samples without removing the non-random component of the error. The fix is not a better model; it is a human-coded probability sample — in one application, expert-coding about 8 % of the documents restored valid inference.41
An annotation that is 90 % accurate is not "good enough." Without a human-coded validation sample, the confidence intervals around anything computed from those labels are simply wrong — and no increase in accuracy repairs them.
A third option sidesteps some of this: use a language model to generate training text, then train an ordinary transparent classifier on it. A multilingual populism classifier trained entirely on synthetic text reached 0.87 accuracy against hand-annotated manifesto statements.42 The model is used as a source of data, not as a source of judgements — which is the same distinction Chapter 4.2 drew between synthesis and simulation.
3.2.8 Four Principles
Grimmer and Stewart set out four principles for text as data. More than a decade on, they have aged better than most of the methods.43
- All quantitative models of language are wrong—but some are useful.
- Quantitative methods for text amplify resources and augment humans.
- There is no globally best method for automated text analysis.
- Validate, Validate, Validate.
The fourth is repeated three times for a reason, and the evidence in this chapter is what it looks like when it is ignored: dictionaries at \(\alpha\) = 0.34 reported as measurements, language-model labels fed into regressions without a coded sample, preprocessing choices made by default and never reported.
Text is not one data structure but a ladder of them, and each rung is a trade.
A cell holds the text and supports searching. One token per row buys word-level analysis and keeps the tidy table. A document-term matrix buys comparison between documents and costs word order and dense storage. An embedding buys meaning and costs interpretability.
Choose the lowest rung that answers your question. And whatever you compute, validate it against people reading the text — because every method in this chapter is an approximation of something a human does effortlessly and a computer does not do at all.
3.2.9 What the Numbers Do Not Say
A closing observation, and it is the reason the last section of this chapter is not a technique.
We now know that 38 of 46 students wrote about the colonial past, that globality was among their most frequent words, and that 28 reviews score positive. Not one of those numbers tells us what any student actually thought about a museum built in a reconstructed Prussian palace, displaying objects from places Germany once colonised.
Counting is not reading. It scales in a way reading cannot, and it makes visible patterns no reader could hold in their head at once. But every technique in this chapter converts language into numbers by discarding what makes language language — order, context, irony, hesitation, the sentence that means the opposite of its words.
That discarded remainder is often the finding.
Mosteller, F., and Wallace, D. L., Inference in an Authorship Problem: A Comparative Study of Discrimination Methods Applied to the Authorship of the Disputed Federalist Papers, Journal of the American Statistical Association 58(302), 1963, 275--309, https://doi.org/10.1080/01621459.1963.10500849. Extended in Inference and Disputed Authorship: The Federalist, Addison-Wesley, 1964; second edition as Applied Bayesian and Classical Inference: The Case of The Federalist Papers, Springer, 1984. The observation that while and whilst discriminate the two authors is credited in the paper to the historian Douglass Adair.↩︎
Kühne, M., Data is everywhere -- Exkursionen als Motivatoren in einer praxisorientierten Lehre, viaLehre -- Newsletter Lehre der Europa-Universität Viadrina, Ausgabe 14, https://www.europa-uni.de/de/universitaet/einrichtungen/serviceeinrichtungen/zentrum-lehre-lernen/newsletter-lehre/_bilder-und-pdfs/Newsletter-Lehre_Ausgabe-14.pdf. The article describes the excursions behind this corpus, including a companion visit to the Futurium in which students fact-checked the exhibition's own statistics against their original sources.↩︎
Silge, J., and Robinson, D., tidytext: Text Mining and Analysis Using Tidy Data Principles in R, Journal of Open Source Software 1(3), 2016, 37, https://doi.org/10.21105/joss.00037. The book-length treatment, freely available, is Text Mining with R: A Tidy Approach, O'Reilly, 2017, https://www.tidytextmining.com/. For the matrix-first alternative see Benoit, K., et al., quanteda: An R Package for the Quantitative Analysis of Textual Data, JOSS 3(30), 2018, 774.↩︎
Denny, M. J., and Spirling, A., Text Preprocessing For Unsupervised Learning: Why It Matters, When It Misleads, And What To Do About It, Political Analysis 26(2), 2018, 168--189, https://doi.org/10.1017/pan.2017.44.↩︎
Schofield, A., Magnusson, M., and Mimno, D., Pulling Out the Stops: Rethinking Stopword Removal for Topic Models, Proceedings of EACL 2017, 432--436, https://aclanthology.org/E17-2069/.↩︎
Rudman, J., The Non-Traditional Case for the Authorship of the Twelve Disputed Federalist Papers: A Monument Built on Sand?, ACH/ALLC 2005. The attribution to Madison has been reproduced repeatedly by other methods; it is the reproducibility of the original pipeline, not the conclusion, that is in question.↩︎
Caliskan, A., Bryson, J. J., and Narayanan, A., Semantics derived automatically from language corpora contain human-like biases, Science 356(6334), 2017, 183--186, https://doi.org/10.1126/science.aal4230.↩︎
Loughran, T., and McDonald, B., When Is a Liability Not a Liability? Textual Analysis, Dictionaries, and 10-Ks, The Journal of Finance 66(1), 2011, 35--65, https://doi.org/10.1111/j.1540-6261.2010.01625.x.↩︎
van Atteveldt, W., van der Velden, M. A. C. G., and Boukes, M., The Validity of Sentiment Analysis: Comparing Manual Annotation, Crowd-Coding, Dictionary Approaches, and Machine Learning Algorithms, Communication Methods and Measures 15(2), 2021, 121--140, https://doi.org/10.1080/19312458.2020.1869198.↩︎
Boukes, M., van de Velde, B., Araujo, T., and Vliegenthart, R., What's the Tone? Easy Doesn't Do It: Analyzing Performance and Agreement Between Off-the-Shelf Sentiment Analysis Tools, Communication Methods and Measures 14(2), 2020, 83--104, https://doi.org/10.1080/19312458.2019.1671966. Their aggregate-level result is the constructive one: weekly averages correlate with human coding up to r = 0.63.↩︎
Councill, I. G., McDonald, R., and Velikovich, L., What's Great and What's Not: Learning to Classify the Scope of Negation for Improved Sentiment Analysis, Proceedings of NeSp-NLP 2010, 51--59, https://aclanthology.org/W10-3110/.↩︎
Van Hee, C., Lefever, E., and Hoste, V., SemEval-2018 Task 3: Irony Detection in English Tweets, Proceedings of SemEval-2018, 39--50, https://doi.org/10.18653/v1/S18-1005.↩︎
Gilardi, F., Alizadeh, M., and Kubli, M., ChatGPT outperforms crowd workers for text-annotation tasks, PNAS 120(30), 2023, e2305016120, https://doi.org/10.1073/pnas.2305016120. Note the benchmark: agreement with trained human annotators, using GPT-3.5 in early 2023.↩︎
Ashwin, J., Chhabra, A., and Rao, V., Using Large Language Models for Qualitative Analysis Can Introduce Serious Bias, Sociological Methods & Research 55(3), 2026, https://doi.org/10.1177/00491241251338246.↩︎
Egami, N., Hinck, M., Stewart, B. M., and Wei, H., Using Imperfect Surrogates for Downstream Inference: Design-Based Supervised Learning for Social Science Applications of Large Language Models, Advances in Neural Information Processing Systems 36, 2023, 68589--68601. The
dslR package implements the correction: https://naokiegami.com/dsl/.↩︎Halterman, A., Synthetically Generated Text for Supervised Text Analysis, Political Analysis 33(3), 2025, 181--194, https://doi.org/10.1017/pan.2024.31.↩︎
Grimmer, J., and Stewart, B. M., Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts, Political Analysis 21(3), 2013, 267--297, https://doi.org/10.1093/pan/mps028. The principles are quoted from their Table 1. Book-length treatment: Grimmer, J., Roberts, M. E., and Stewart, B. M., Text as Data: A New Framework for Machine Learning and the Social Sciences, Princeton University Press, 2022.↩︎