Count the words people use
The words people use for a problem are worth collecting on their own: they’re the words to put in an ad, a landing page or a search query. The market research guide explains why. This chapter counts them in Python, sees where in the thread they turn up, and ends with what a count can’t tell you.
import re
import pandas as pd
from sklearn.feature_extraction.text import ENGLISH_STOP_WORDS
from threadtools import check_tree, load_export
pd.set_option("display.width", 100)
pd.set_option("display.max_colwidth", 60)
records, meta = load_export("data/sample_thread.json")
post, comments = check_tree(records)
# Deleted and removed comments have no text to count
texts = comments[comments.comment_state == "normal"].copy()comments: 95, orphans: 0, depth mismatches: 0
comment_state
normal 92
deleted 2
removed 1
hidden or missing scores: 2
Count comments, not occurrences¶
Someone who writes “price” five times is still one person. So the unit here is the comment: for each comment, the set of words and two-word phrases in it, each counted once.
# Filler with no topic in a casual thread, added to scikit-learn's English list
EXTRA_STOP_WORDS = {"just", "did", "didn", "don", "got", "like", "really", "pretty",
"honestly", "lol", "yes", "ones", "thing", "way", "kind", "good",
"people", "use", "exactly", "agreed", "fair"}
STOP_WORDS = ENGLISH_STOP_WORDS | EXTRA_STOP_WORDS
def comment_terms(text):
"""The words and two-word phrases in one comment.
A phrase is two neighbouring words, neither of them a stop word.
"""
tokens = re.findall(r"[a-z]+(?:'[a-z]+)?", text.lower())
kept = [None if token in STOP_WORDS or "'" in token else token for token in tokens]
words = {token for token in kept if token and len(token) > 2}
phrases = {f"{a} {b}" for a, b in zip(kept, kept[1:]) if a and b}
return words | phrases
comment_terms("Price creep plus shrinking portions. Same price, less food."){'creep',
'creep plus',
'food',
'plus',
'plus shrinking',
'portions',
'price',
'price creep',
'shrinking',
'shrinking portions'}Phrases are built before stop words are dropped, so “cheaper than groceries” doesn’t turn into the phrase “cheaper groceries”, which nobody wrote. scikit-learn’s CountVectorizer with ngram_range=(1, 2) removes stop words first and does produce phrases like that.
“Recipe” and “recipes” should count as one word. A full lemmatiser is more than this thread needs, so the rule below only drops a final “s” when the shorter word also appears in the thread. It merges “recipes” into “recipe” and leaves “packs” alone, because nobody wrote “pack”.
texts["terms"] = texts.body.map(comment_terms)
vocabulary = set().union(*texts.terms)
def singular(term):
return term[:-1] if term.endswith("s") and term[:-1] in vocabulary else term
texts["terms"] = texts.terms.map(lambda terms: {singular(term) for term in terms})Now one row per comment and term, and a count of comments for each term, split into top-level comments and replies:
rows = texts[["comment_id", "depth", "terms"]].explode("terms").dropna()
counts = rows.groupby("terms").agg(
comments=("comment_id", "size"),
top_level=("depth", lambda depth: int((depth == 0).sum())),
)
counts["replies"] = counts.comments - counts.top_level
counts = (counts.rename(columns={"top_level": "top-level"}).rename_axis("term")
.reset_index().sort_values(["comments", "term"], ascending=[False, True])
.set_index("term"))
is_phrase = counts.index.str.contains(" ")
counts[~is_phrase].head(12)“Recipe” leads with 13 comments, then “month” with 9. “Price” is in 6. On a thread of 92 comments with text, the counts are small, and a difference of one or two means little. What the table is good for is the vocabulary, and phrases show it better than single words:
from charts import phrases_by_depth
phrases = counts[is_phrase & (counts.comments >= 2)]
phrases_by_depth(phrases, "Two-word phrases used in at least two comments");
These are the thread’s own words: “ice packs”, “pause instead”, “intro discount”, “recipe cards”, “price creep”, “half bottles”. An ad written from this list would sound like the people in the thread, which a list of single words doesn’t give you.
Some words live in replies¶
The split by depth matters as much as the total. Words that mostly turn up in replies get missed if you only read top-level comments:
common = counts[~is_phrase & (counts.comments >= 4)]
common.assign(reply_share=(common.replies / common.comments).round(2)).sort_values(
"reply_share", ascending=False).head(8)“Depend” is in four comments, all of them replies. Printing them shows why:
def comments_with(term, texts, width=78):
for row in texts[texts.terms.map(lambda terms: term in terms)].itertuples():
text = " ".join(row.body.split())[:width]
print(f"[{row.comment_id}] d{row.depth} s{row.score}: {text}")
comments_with("depend", texts)[ni47wqa] d5 s22: Fair, it depends entirely on how stocked your pantry already is.
[n7v3di8] d2 s27: Depends on your city. Ours takes the cardboard but not the liner, and the gel
[necxvm1] d3 s12: Ours were honored for one last box after we paused. Might depend on whether yo
[nz14skw] d1 s22: Depends a lot on the recipe. The pasta ones were huge, the fish ones tiny.
Each one qualifies the comment above it: recycling depends on your city, portions on the recipe, the cost gap on your pantry, the credits on whether you paused or cancelled. Read only the top-level comments and you get the claims without the conditions.
A word count isn’t a complaint count¶
Counting words is a fast way to find comments about a topic. It isn’t a way to count them. Here’s a rule that flags price comments by their words, checked against my own reading of each comment it touches:
PRICE_WORDS = r"\$|price|pricing|cost|cheap|expensive|discount|surcharge|receipt"
flagged = texts[texts.body.str.contains(PRICE_WORDS, case=False, regex=True)]
labels = pd.read_csv("data/price_labels.csv", dtype=str, keep_default_na=False)
labels = labels.set_index("comment_id")
flagged.comment_id.map(labels.about_price).value_counts()comment_id
yes 16
no 2
Name: count, dtype: int64data/price_labels.csv holds a yes or no for every comment in this section, with a short note on why. The rule flags 18 comments, and 16 of them are about price. The other two use the words for something else:
wrong = flagged[flagged.comment_id.map(labels.about_price) == "no"]
for row in wrong.itertuples():
print(f"[{row.comment_id}] {' '.join(row.body.split())[:70]}")
print(f" {labels.note[row.comment_id]}")[nywlecf] Exactly. I had $40 in credits when I cancelled and they just disappear
lost $40 in credits when cancelling
[nn9g8te] The cancel flow. Four screens of 'are you sure', a discount offer, a s
a discount offer inside the cancel flow
The rule also misses comments, because a reply doesn’t have to repeat its parent’s words. Reading the direct replies to the 18 flagged comments that the rule didn’t flag itself:
is_flagged = texts.comment_id.isin(flagged.comment_id)
missed = texts[texts.parent_id.isin(flagged.comment_id) & ~is_flagged]
about_price = missed[missed.comment_id.map(labels.about_price) == "yes"]
print(f"{len(missed)} replies read, {len(about_price)} about price:")
for row in about_price.itertuples():
print(f" [{row.comment_id}] d{row.depth}: {' '.join(row.body.split())[:66]}")12 replies read, 5 about price:
[nz8iq9y] d1: Did you ever compare it to just buying the same recipes at the gro
[nkttnjf] d4: This is the most useful comment in the thread. Nobody counts the l
[ni47wqa] d5: Fair, it depends entirely on how stocked your pantry already is.
[na35owi] d4: Is that documented anywhere or just vibes?
[n0t4wy5] d2: Agreed.
So at least 21 comments are about price, not the 18 the keyword rule found, and two of those 18 shouldn’t count. “Agreed.” under a comment about dishonest pricing is a vote for that point, with no price word in it. And 21 is still a floor: “How big was the gap for you?” is about price too, but it sits two levels below a flagged comment, so it wasn’t in the 30 comments I read.
None of this needs a bigger model or a better word list. A count tells you where to look, and the number you report has to come from reading. The next chapter gives an LLM the reading, and checks its quotes in code.