Skip to article frontmatterSkip to article content
Site not loading correctly?

This may be due to an incorrect BASE_URL configuration. See the MyST Documentation for reference.

Load an export and check it

Comment Scraper

Every chapter starts from the same file, a Reddit thread exported as “JSON, one row per comment”. Before any analysis, this chapter checks that the reply tree in it holds together and goes through the details that quietly produce wrong numbers later: hidden scores, deleted comments, the CSV version of the same export and Hacker News threads.

Load the file

An export holds a list of threads. A single export has one, a batch export has several. Each thread has a meta block and its rows, and load_export puts the rows of all of them in one table. The nested JSON format stores replies inside their parents instead, so the loader first flattens it back into rows, in the same order: the post, then each comment followed by its replies.

threadtools.py
def flatten_nested(post, comments):
    """Turn a "JSON, replies nested" thread back into rows: post, then depth-first."""
    rows = [post]
    stack = list(reversed(comments))
    while stack:
        node = stack.pop()
        rows.append({key: value for key, value in node.items() if key != "replies"})
        stack.extend(reversed(node["replies"]))
    return rows


def load_export(path):
    """Read a Comment Scraper JSON export, flat or nested: rows and metadata."""
    export = json.loads(Path(path).read_text(encoding="utf-8"))
    metas, frames = [], []
    for thread in export["threads"]:
        metas.append(thread["meta"])
        if "records" in thread:
            rows = thread["records"]
        else:
            rows = flatten_nested(thread["post"], thread["comments"])
        frames.append(pd.DataFrame(rows))
    records = pd.concat(frames, ignore_index=True)
    # A hidden score is a missing value, not zero. Hacker News comments never have one.
    records["score"] = records["score"].astype("Int64")
    records["depth"] = records["depth"].astype("Int64")
    records["created_utc"] = pd.to_datetime(records["created_utc"])
    return records, pd.DataFrame(metas)


def one_thread(records):
    """A batch export holds several threads; the functions below take one at a time."""
    thread_ids = records.thread_id.unique()
    if len(thread_ids) != 1:
        raise ValueError(f"{len(thread_ids)} threads in these rows; pick one first, "
                         "e.g. records[records.thread_id == records.thread_id.iloc[0]]")

Loading...

These are the three numbers the extension reports for every export, explained on the help page. The file holds 95 comments and the source reported 98. The status says the extension found no collapsed reply links left to open, and that alone doesn’t explain a gap. The help page lists the possible causes: deleted comments, comments you aren’t allowed to see, and comments the export missed. The file can’t tell you which. In this sample I set the source count to 98 and wrote 95 comments. On real threads, the help page notes, the two often don’t match.

When the status is partial, treat every count you compute as a lower bound, and say in your write-up that medians and shares describe only the comments you have.

The nested export of the same thread loads to exactly the same table:

True

Check the tree

threadtools.py
def one_thread(records):
    """A batch export holds several threads; the functions below take one at a time."""
    thread_ids = records.thread_id.unique()
    if len(thread_ids) != 1:
        raise ValueError(f"{len(thread_ids)} threads in these rows; pick one first, "
                         "e.g. records[records.thread_id == records.thread_id.iloc[0]]")


def check_tree(records):
    """Split one thread into post and comments; check replies point one level up."""
    one_thread(records)
    post = records[records.record_type == "post"].iloc[0]
    comments = records[records.record_type == "comment"].copy()

    known = set(comments.comment_id) | {post.thread_id}
    has_parent = comments.parent_id.isin(known)
    orphans = comments[~has_parent]
    depth_of = dict(zip(comments.comment_id, comments.depth))
    depth_of[post.thread_id] = -1
    parent_depth = comments.parent_id.map(depth_of)
    wrong_depth = comments[has_parent & (parent_depth + 1 != comments.depth)]

    print(f"comments: {len(comments)}, orphans: {len(orphans)}, "
          f"depth mismatches: {len(wrong_depth)}")
    print(comments.comment_state.value_counts().to_string())
    print(f"hidden or missing scores: {comments.score.isna().sum()}")
    return post, comments

comments: 95, orphans: 0, depth mismatches: 0
comment_state
normal     92
deleted     2
removed     1
hidden or missing scores: 2

check_tree stops if the rows hold more than one thread, so pick one thread of a batch export before calling it.

No orphans and no depth mismatches: every reply points at a comment in the file, exactly one level up. If a fetched reply’s parent is missing, which can happen on a partial export, the extension marks the row with parent_missing, and the check above counts it as an orphan.

Deleted and removed comments stay in the file with an empty body. Their text is gone, but their place in the tree isn’t:

Loading...

The deleted top-level comment n128n6o still has a reply under it, and that reply has one of its own. Drop the deleted row and both lose their way back to the top of the thread. So filter deleted and removed comments out when you analyse text, and keep them when you analyse structure.

Hidden scores are missing, not zero

A subreddit can hide new comments’ scores for a while. When Reddit hides a score, the export leaves score empty and sets score_hidden to true. load_export reads the column as pandas’ nullable integer type, so those two comments stay missing instead of turning into zeros or forcing the whole column to floats. It matters:

Loading...

Filling the two hidden scores with zero pulls the top-level median from 88 to 84.5 and the depth-1 median from 26 to 25, and drops both comments to the bottom of any score ranking. Zero is a guess like any other number: the export doesn’t say what either score is.

The CSV version

The CSV export is meant for spreadsheets, and two things change on the way. The file starts with a byte order mark so that Excel reads it as UTF-8, and any text cell starting with =, +, - or @ gets a leading apostrophe so a spreadsheet won’t run it as a formula. A comment written as a Markdown list starts with a hyphen:

[nf7ag3b]  csv: "'- Price went up\n- Delivery "
           json: '- Price went up\n- Delivery m'

That’s the only comment that differs between the two files. For analysis, read the JSON. If you have to read the CSV, pass the arguments above:

  • encoding="utf-8-sig" drops the byte order mark. pandas 3.1, used here, drops it even without that argument, but Python’s own csv module doesn’t: open the file as plain UTF-8 and the first column comes back as "\ufeffschema_version".

  • keep_default_na=False, na_values=[""] treats only empty cells as missing. By default pandas also reads a cell that says NA, null or None as missing, and on a forum that’s sometimes the whole comment.

  • dtype keeps the IDs as text. Reddit IDs are letters and digits, so they survive anyway. Hacker News IDs are plain numbers, and the next section shows what pandas does to them.

Hacker News threads

The same loader reads a Hacker News export:

comments: 11, orphans: 0, depth mismatches: 0
comment_state
normal     10
deleted     1
hidden or missing scores: 11
Loading...

Three differences from Reddit:

  • Hacker News doesn’t show comment points, so score is empty for every comment. Anything in the next chapter that filters or sorts by score has nothing to work with on a Hacker News thread. Depth and OP still work.

  • There’s no subreddit, and the source’s comment count is unknown, so reported_by_source is empty.

  • IDs are numbers. In the JSON they’re strings, which is how they should stay. Read the CSV without dtype and they don’t:

Loading...

Both columns became floats, like 45120424.0, because the post row has no comment ID and no parent, and pandas fills those gaps with NaN. Neither column matches the text IDs in the JSON or in a permalink any more: merge on them raises a type error, and a lookup with isin or an index join quietly finds none of the comments. With dtype=ids both stay text.

The next chapter starts filtering. If you only want the comments in Python and you use LangChain, the LangChain loader reads the same JSON files into one Document per comment.