RedditHarbor is a Python library for collecting Reddit data into a database you control. It reads from the official Reddit Data API through PRAW, writes to a Postgres database hosted on Supabase, and stores only the columns you declare. It is built for social-science researchers who need a defensible data pipeline more than another scraping script.
Deprecation notice. RedditHarbor 0.3 and earlier are no longer supported. They collect every column by default, which is incompatible with GDPR data minimisation, and they depend on Supabase's legacy
service_rolekeys, which Supabase retires at the end of 2026. Upgrade withpip install --upgrade redditharbor(Python 3.10 or later) and follow the migration guide. Details are in the changelog.
When Pushshift lost API access in 2023, Reddit research lost the infrastructure it had quietly relied on. What remained was a choice between bulk archives of uncertain legal standing and a free API that leaves you to build everything yourself. RedditHarbor is the third option: collection through the official API, storage in your own database, and a schema you settle before the first row is written.
Collect only what you declare. You pass a columns dictionary naming the fields your study needs.
Anything not listed is neither requested from Reddit nor stored, and the extra API calls behind fields
such as moderator status or trophies are never made unless asked for.
Keep the data yours. Rows land in Postgres tables you own, de-duplicated by Reddit id, with free text optionally anonymised at collection time. Export to CSV, JSON, text or images when you are ready to analyse.
Track change over time. Scores, upvote ratios and comment counts are stored as timestamped histories and refreshed on a schedule that respects Reddit's rate limit, which a static snapshot cannot do.
Python 3.10 or later.
pip install redditharbor # core
pip install "redditharbor[pii]" # adds presidio and spaCy for mask_pii=TrueThree inputs: credentials, table names, and the columns you will collect.
import redditharbor.login as login
from redditharbor import schema
from redditharbor.dock.pipeline import collect
DB_CONFIG = {"user": "test_redditor", "submission": "test_submission", "comment": "test_comment"}
COLUMNS = {
"user": ["redditor_id", "created_at", "karma"],
"submission": ["submission_id", "redditor_id", "created_at", "title", "text", "subreddit", "score"],
"comment": ["comment_id", "link_id", "parent_id", "redditor_id", "created_at", "body"],
}
# 1. Create the tables: paste the printed SQL into the Supabase SQL editor.
print(schema.create_table_sql(DB_CONFIG, COLUMNS))
# 2. Connect. The Supabase key is the secret key (sb_secret_...) under Settings > API Keys.
reddit_client = login.reddit(public_key="...", secret_key="...", user_agent="Institution:Project (u/username)")
supabase_client = login.supabase(url="https://<project-id>.supabase.co", private_key="sb_secret_...")
# 3. Collect.
harbor = collect(reddit_client, supabase_client, db_config=DB_CONFIG, columns=COLUMNS)
harbor.subreddit_submission(["python", "learnpython"], ["hot", "top"], limit=5, mask_pii=True)That last call stores the five hottest and five top submissions from each subreddit with exactly the
seven submission columns above, plus a user row per author with the three user columns. Leave "user"
out of COLUMNS and no user profiles are collected at all. Credentials are saved to a local .env
on first use, so later scripts can call login.reddit() and login.supabase() with no arguments.
Each table has a primary key that is always stored; everything else is opt-in.
| Table | Always | Optional |
|---|---|---|
user |
redditor_id |
name, created_at, karma, is_gold, is_mod, trophy, removed |
submission |
submission_id |
redditor_id, created_at, title, text, subreddit, permalink, attachment, flair, awards, score, upvote_ratio, num_comments, edited, archived, removed, poll |
comment |
comment_id |
link_id, subreddit, parent_id, redditor_id, created_at, body, score, edited, removed |
name is the only direct identifier. title, text, body and flair are free text that may
contain personal data; mask_pii=True anonymises text and body with
presidio before they are stored. The
Choosing Columns page lists
types, personal-data flags, and what changes when a column is left out.
| Method | Collects | Typical use |
|---|---|---|
subreddit_submission(subreddits, sort_types, limit) |
submissions, and their authors | community studies |
subreddit_comment(subreddits, sort_types, limit, level) |
comments under listed submissions, to a reply depth | discourse analysis |
subreddit_submission_and_comment(...) |
both of the above in one pass | full community snapshots |
submission_by_keyword(subreddits, query, limit) |
submissions matching a boolean query | topic studies |
comment_from_submission(submission_ids, level) |
comment threads of submissions already in your database | expanding a corpus |
submission_from_user(user_names, ...), comment_from_user(user_names, ...) |
the activity of known users | posting trajectories |
Keyword queries accept AND, OR, NOT and parentheses, so
"renewable energy NOT (fossil OR coal)" does what it says. sort_types takes any of hot, new,
rising, top and controversial; limit=None fetches as many as the API allows.
Scheduled updates append fresh values to the metric histories:
from redditharbor.dock.pipeline import update
update(reddit_client, supabase_client, DB_CONFIG, COLUMNS).schedule_task(task="submission", duration="1d")The interval is chosen from the table size, from every 10 minutes for up to 1,000 rows to daily above
72,000, so a run stays inside Reddit's 100 queries per minute. Only the metrics you collect (score,
upvote_ratio, num_comments) are refreshed, and archived submissions are skipped when archived is
collected.
from redditharbor.utils import download
submissions = download.submission(supabase_client, DB_CONFIG["submission"])
submissions.to_csv(columns="all", file_path="exports") # every column present in the table
submissions.to_json(columns=["submission_id", "title", "score"], file_path="exports")
submissions.to_img(file_path="exports") # attached images; needs `attachment`to_csv, to_json, to_txt and to_pkl exist for download.submission, download.comment and
download.user, paginating through tables of any size.
redditharbor/
login.py Reddit and Supabase clients, .env credential storage, key validation
schema.py column definitions, validation, CREATE TABLE generation
dock/pipeline.py collect (collection modes) and update (scheduled metric refresh)
utils/download.py export tables to csv / json / txt / pickle / images
utils/fetch.py read submission ids and user names back out of the database
utils/updates.py PyPI update notice (set REDDITHARBOR_NO_UPDATE_CHECK=1 to disable)
utils/subreddit_collections.py curated subreddit lists by social-science domain
docs/ Jupyter Book source for socius-org.github.io/RedditHarbor
app/ Next.js dashboard, work in progress
The documentation covers prerequisites (Reddit and Supabase credentials), table setup, each collection mode, exports, scheduled updates, and the migration from 0.3.
@software{oh2026redditharbor,
author = {Oh, Nick},
title = {RedditHarbor: collect Reddit data into a database you control},
year = {2026},
version = {0.4.0},
url = {https://github.com/socius-org/RedditHarbor}
}Released under the MIT License. Data collected with RedditHarbor remains subject to Reddit's Data API terms and to the ethics and data-protection requirements of your institution.