This project tests the overreaction hypothesis that abnormal single-day price jumps are at least partly temporary, and therefore predictably reverse. Using five decades of daily prices across 200 tickers and FinBERT-scored headlines from the FNSPID corpus, the pipeline detects jumps via a dual z-score / rolling-percentile filter, measures reversal within a 60-day window, and asks whether direction, news sentiment, or market regime predicts that reversal.
Python · pandas · FinBERT · yfinance · statsmodels · matplotlib · seaborn
Five decades of daily prices across 200 tickers. Around the dot-com crash, Lehman collapse, COVID pandemic, and Liberation Day clustered price jumps are visible, and appear to fade out.
Zooming in on a single event: the price drops sharply at day 0, then recovers. The horizontal line marks the event-day close. A reversal is recorded when this crossing happens within 60 days (median observed recovery duration).
Measuring this across all detected events:
- Down jumps reverse significantly more often than up jumps (~51% vs ~40%; ANOVA F = 25.6, p < 0.001). Markets appear to overreact more strongly to negative shocks.
- FinBERT detected news sentiment does not predict reversals. Whether headline sentiment is aligned, contrary, neutral, or absent has no significant effect on reversal rate (p = 0.139).
- Market regime (bull/bear) has a modest main effect (p = 0.040), but does not interact with jump direction (p = 0.975). The direction asymmetry holds in both regimes. This could simply be explained by the tendency for stocks to go up eventually.
→ Full methodology and figures: RESULTS.md
| Source | Description | Coverage |
|---|---|---|
| FNSPID | Financial news articles with ticker labels | 2009–2020 |
| yfinance | Daily OHLCV prices, split- and dividend-adjusted | Full available history per ticker |
The ticker universe spans large-cap equities, sector ETFs, and ADRs selected by news coverage density. See tickers.yaml for the full list and data/fnspid_metadata.csv for per-ticker article counts and date ranges.
git clone https://github.com/mqcapelle/market-overreaction.git
cd market-overreaction
conda env create -f environment.yaml
conda activate market-overreactionDownload the FNSPID dataset from Hugging Face and place the file at:
data/raw/fnspid/FNSPID_full.csv
Each script is independently runnable and writes its output before the next script reads it.
python pipeline/01_select_tickers.py
python pipeline/02_download_prices.py
python pipeline/03_build_price_df.py
python pipeline/04_build_event_df.py
python pipeline/05_build_news_df.py # slowest step — runs FinBERT on all articles
python pipeline/06_merge_event_sentiment.py
python pipeline/07_analysis.pyFigures are written to figures/.
market-overreaction/
├── pipeline/
│ ├── 01_select_tickers.py # rank and filter tickers by news coverage
│ ├── 02_download_prices.py # fetch OHLCV from yfinance
│ ├── 03_build_price_df.py # log returns, rolling vol, z-scores, regime labels
│ ├── 04_build_event_df.py # detect price-jump events, measure reversals
│ ├── 05_build_news_df.py # run FinBERT sentiment on FNSPID articles
│ ├── 06_merge_event_sentiment.py # join events and sentiment on [ticker, date]
│ └── 07_analysis.py # aggregate statistics and figures
├── utils/
│ ├── plot_style.py # shared matplotlib style
│ └── utils.py # shared helpers (load_tickers, schema checks)
├── data/
│ ├── raw/ # prices and FNSPID splits (not tracked in git)
│ ├── processed/ # parquet outputs (not tracked in git)
│ ├── fnspid_metadata.csv # coverage index: ticker, date range, article count
│ └── tickers.yaml # selected ticker universe (editable)
├── figures/ # output plots (subfolders not tracked in git)
├── config.py # all parameters — edit here
├── dataset_spec.md # full schema for every pipeline dataset
└── environment.yaml
- The 60-day reversal window is set to the median observed reversal duration in the data — events that recover more slowly are not counted as reversals, which means the reversal fractions reported here are a lower bound on eventual mean-reversion.
- The bull/bear regime label is derived from a 200-day moving average of SPY, which is a lagging indicator. Brief dislocations (e.g. the COVID crash) may be mis-labelled for several weeks.
- News coverage in FNSPID runs from 2009 to 2020. All sentiment analysis is limited to this window; the price-only analysis uses the full five-decade history.
FNSPID dataset
Zihan Chen et al. FNSPID: A Comprehensive Financial News Dataset. Hugging Face, 2024. https://huggingface.co/datasets/Zihan1004/FNSPID
FinBERT
Dogu Araci. FinBERT: Financial Sentiment Analysis with Pre-trained Language Models. arXiv:1908.10063, 2019. https://arxiv.org/abs/1908.10063
MIT — see LICENSE.