Filter by depth, score and OP
In the sample thread, 11 of the 15 highest-scoring comments are top-level, so sorting by score mostly shows you those. That’s a fine place to start reading, but the export has three more columns to filter on: how deep a comment sits, what it replies to and whether the original poster (OP) wrote it. This chapter uses all three.
import pandas as pd
from threadtools import check_tree, load_export
pd.set_option("display.width", 100)
pd.set_option("display.max_colwidth", 60)
records, meta = load_export("data/sample_thread.json")
post, comments = check_tree(records)comments: 95, orphans: 0, depth mismatches: 0
comment_state
normal 92
deleted 2
removed 1
hidden or missing scores: 2
Score by depth¶
scored = comments[(comments.comment_state == "normal") & comments.score.notna()]
level = scored.depth.clip(upper=3) # 3 means "3 or deeper"
scored.score.groupby(level).agg(comments="size", median="median")top_15 = scored.nlargest(15, "score")
top_15.depth.value_counts().sort_index()depth
0 11
1 3
3 1
Name: count, dtype: Int64The median score drops from 88 for top-level comments to between 15.5 and 26 for replies, and the top 15 are mostly top-level. The sample’s scores are invented, so they can’t tell you why this happens on real threads. On Reddit, deep replies often sit behind “load more comments” and “continue this thread” links, which may mean fewer people see them, but this sample can’t measure that. On this thread, a fixed score cut-off would keep mostly top-level comments. Check the same table on your own export before you choose one.
Replies that beat their parent¶
A fairer comparison for a reply is the comment it answers. Comparing the two scores finds pairs worth reading:
def beat_their_parent(comments, min_score=10):
"""Replies that outscored the comment they answer. Under ~10 points it's noise."""
parent_score = comments.set_index("comment_id").score
normal = comments.comment_state == "normal"
replies = comments[(comments.depth > 0) & normal].copy()
replies["parent_score"] = replies.parent_id.map(parent_score)
# A hidden score on either side makes the comparison unknown; count that as no.
higher = replies.score > replies.parent_score
beat = (higher & (replies.score >= min_score)).fillna(False)
return replies[beat].sort_values("score", ascending=False)
winners = beat_their_parent(comments)
columns = ["comment_id", "depth", "score", "parent_score", "body"]
winners.assign(body=winners.body.str[:50])[columns]Eleven replies pass. The first is a cost comparison three levels down with 187 points, under a question that got 23. The second, ns6rvyp, answers an “unpopular opinion” comment with 57 points by turning it around: people don’t expect a meal kit to be cheaper than groceries, they expect the price they signed up at to stay. Read only top-level comments and you get the unpopular opinion without the reply that outscored it.
Treat this list as a reason to read a pair of comments, not as a result. The export records each comment’s score. It can’t tell you whether the same people voted on both.
from charts import score_by_depth
score_by_depth(scored, winners);
Read a comment with everything above it¶
The 187-point comparison looks like the thread’s verdict on price. Read in place, it’s one step in a longer exchange. show_in_context prints a comment under all of its ancestors, and the deepest reply in the thread happens to sit under it:
def chain_to_top(comment_id, by_id):
"""The comment and its ancestors, top-level first.
Stops early if a parent wasn't fetched.
"""
chain = [comment_id]
while (by_id.loc[chain[-1], "depth"] > 0
and by_id.loc[chain[-1], "parent_id"] in by_id.index):
chain.append(by_id.loc[chain[-1], "parent_id"])
return chain[::-1]
def show_in_context(comment_id, comments, width=100):
"""Print a comment under every comment above it, to read a quote in place."""
by_id = comments.set_index("comment_id")
for cid in chain_to_top(comment_id, by_id):
row = by_id.loc[cid]
who = "OP" if row.is_op else "--"
text = " ".join(row.body.split()) or f"[{row.comment_state}]"
print(f"{' ' * row.depth}[{cid}] {who} s{row.score}: {text[:width]}")
print(f" {by_id.loc[comment_id, 'permalink']}")
from threadtools import show_in_context
deepest = comments.loc[comments.depth.idxmax(), "comment_id"]
show_in_context(deepest, comments, width=72)[nk9p34y] -- s412: Price. It started at $8 a serving with the intro discount and crept up t
[nz8iq9y] -- s37: Did you ever compare it to just buying the same recipes at the grocery s
[njzb6cn] -- s23: How big was the gap for you?
[nz43dvy] -- s187: I priced 9 recipes against my local store. Kit was $11.90 a serving, gro
[nf5jxvl] -- s29: Your $9 assumes you throw out the leftovers. I cook a lot so the fish sa
[ni47wqa] -- s22: Fair, it depends entirely on how stocked your pantry already is.
[n9vq24z] -- s31: Which is exactly the person these kits aren't for. Beginners have empty
https://www.reddit.com/r/ExampleMealKits/comments/1nq7x2k/cancelled_our_meal_kit_after_14_months_what/n9vq24z/
The comparison answers a direct question under a top comment about price creep, and the reply right below it, nf5jxvl, says its 9, quote that objection with it. In a real export, the permalink on the last line opens that comment on Reddit, so whoever reads your quote can check it in place. This sample thread is invented, so its permalinks lead nowhere.
Where the OP joined in¶
op = comments[comments.is_op]
op.assign(body=op.body.str[:60])[["comment_id", "depth", "score", "body"]]The OP wrote five comments, all of them replies to other people, and none scored above 27. Sorting by score buries them. Filtering on is_op finds them in one line, and they’re often where the OP adds detail after posting, like “We definitely paid for two months we didn’t use before I noticed.”
The export has no usernames, so “who replies to whom” can only mean the post, the OP or someone else. That’s still enough for a small table:
is_op = comments.set_index("comment_id").is_op
def party(parent_id):
if parent_id == post.thread_id:
return "post"
if parent_id not in is_op.index: # parent not in the file: could be the OP
return "unknown"
return "OP" if is_op[parent_id] else "others"
pd.crosstab(comments.parent_id.map(party).rename("replying to"),
comments.is_op.map({True: "OP", False: "others"}).rename("reply by"))All five OP comments answer someone else, and nobody replied to the OP. Replies whose parent is missing from the file would be counted in a separate “unknown” row.
The shape of each branch¶
Each top-level comment starts a branch. Its reply count and its deepest level tell broad branches from long chains. Whether people agree or argue in them, you find out by reading:
def branch_shapes(comments):
children = comments.groupby("parent_id").comment_id.apply(list).to_dict()
by_id = comments.set_index("comment_id")
rows = []
for root in comments[comments.depth == 0].itertuples():
members, stack = [], [root.comment_id]
while stack: # no recursion: real reply chains can be very deep
current = stack.pop()
members.append(current)
stack.extend(children.get(current, []))
branch = by_id.loc[members]
rows.append({
"comment_id": root.comment_id,
"score": root.score,
"replies": len(members) - 1,
"deepest": int(branch.depth.max()),
"by_op": int(branch.is_op.sum()),
"starts_with": " ".join(root.body.split())[:38] or "[deleted]",
})
return pd.DataFrame(rows).sort_values("replies", ascending=False)
branch_shapes(comments).head(6)The two branches with six replies have very different shapes. nk8w3tz, about repetitive recipes, is wide and shallow: three separate replies, each answered once, nothing below depth 2. n1wzcwu is one long chain down to depth 6. In it, someone asks whether shrinking portions are “documented anywhere or just vibes” and gets an answer about receipts and salmon fillets. Nobody shows the receipts, but you come away with a specific claim to check, which a row of “same here” replies under a popular comment doesn’t give you.
On Hacker News¶
Hacker News exports have no comment scores, so beat_their_parent returns nothing and the score table is empty. Depth, parent_id and is_op work the same, and so do show_in_context and branch_shapes.
The next chapter turns to the text itself and counts the words people use.