Skip to content

Parser Improvements and newly labeled speeches - #90

Open
jutton1 wants to merge 8 commits into
unitedstates:mainfrom
jutton1:fix-text-mislabeling
Open

Parser Improvements and newly labeled speeches#90
jutton1 wants to merge 8 commits into
unitedstates:mainfrom
jutton1:fix-text-mislabeling

Conversation

@jutton1

@jutton1 jutton1 commented Aug 8, 2026

Copy link
Copy Markdown

Sorry about the delay with the PR... was busier than intended.

I also spent around 30 min a day hand labeling starts of speeches within sections of the record.

Both the old and new parses were run over the whole corpus and compared over 858,732 documents

count share
documents unchanged 854,079 99.46%
documents with ≥1 changed speech item 4,653 0.54%
speech items unchanged 2,682,217 99.665%
speech items newly attributed 5,096 0.189%
speech items reattributed 1,986 0.074%
speech items, boundary moved 1,703 0.063%
speech items no longer attributed 220 0.008%
words unchanged 576,175,058 99.755%
words newly attributed 567,175 0.098%
words reattributed 834,621 0.144%
words no longer attributed 15,897 0.003%

29.7% of speech items (799,542) get their speaker from the fallback rather than from
mods.xml, but for proof of concept, I reparsed without as mods.xml was hiding some of the failings of the old parser.

with mods.xml without ratio
documents changed 4,653 63,392 13.6×
speech items newly attributed 5,096 59,030 11.6×
words newly attributed 567,175 15,719,712 27.7×
words no longer attributed 15,897 16,671 1.05×

The hand auditing was 250 windows of 40 lines each. 150 were centred on an item the
change moves, 100 from documents that parse identically. I did do this blind, so i had no clue which was which.

Metadata shows I spent around 182 minutes on hand labeling. I have placed the hand labeled in tests, I'll wire in something to utilize it later.

exact-match 95% CI precision speaker IoU
old 70.4% [66.8%, 73.8%] 98.7% 69.9%
this PR 99.7% [98.9%, 99.9%] 99.5% 99.4%

Exact match means the rate at which the old and new parser matched what I hand labeled.
Precision is the rate at which i matched what the parser stated.
Speaker IoU is exact match over union of parser + human

closes #36 , closes #77, closes #79 , closes #80 , closes #81 , closes #82 , closes #83 , closes #84 , closes #85 , closes #86 , closes #87 , closes #88 , closes #89

@jutton1 jutton1 mentioned this pull request Aug 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment