Advanced NLP on Reddit Data

A comprehensive Natural Language Processing pipeline that collects Reddit data and performs advanced analysis including sentiment analysis, topic detection, embeddings generation, and LLM-based insight generation.

📋 Overview

This project demonstrates a complete NLP workflow for analyzing Reddit discussions. It combines traditional NLP techniques with modern deep learning approaches, including vector databases and large language models for generating actionable insights from unstructured text data.

🎯 Project Objectives

• Collect and explore Reddit data at scale
• Perform comprehensive text preprocessing and cleaning
• Analyze sentiment patterns in discussions
• Detect underlying topics using advanced techniques
• Generate embeddings for semantic search capabilities
• Leverage LLMs to extract meaningful insights
• Evaluate model performance and results

📊 Project Structure

Adncanved-NLP-Reddit/
├── 01-Data Collection and Exploratory Data Analysis.ipynb
├── 02-Text Preprocessing.ipynb
├── 03-Sentiment Analysis.ipynb
├── 04-Topic Detection.ipynb
├── 05-Embeddings-FAISS-RAG.ipynb
├── 06-LLM-Insight-Generation.ipynb
├── 07-Evaluation.ipynb
├── data/                          # Raw and processed data
├── outputs/                       # Analysis results and visualizations
├── vectorstore/                   # FAISS vector store for embeddings
├── dashboard/                     # Dashboard application files
└── www.reddit.com_cookies.txt    # Reddit authentication credentials

📚 Notebooks Description

1. Data Collection and Exploratory Data Analysis

• Collects Reddit posts and comments using the Reddit API
• Performs initial data exploration and statistical analysis
• Visualizes data distribution and key patterns
• Key outputs: Raw dataset overview and basic statistics

2. Text Preprocessing

• Cleans and normalizes text data
• Removes noise, special characters, and stopwords
• Tokenization and lemmatization
• Prepares data for downstream NLP tasks
• Key outputs: Cleaned and preprocessed text corpus

3. Sentiment Analysis

• Analyzes sentiment polarity of Reddit discussions
• Classifies posts/comments as positive, negative, or neutral
• Generates sentiment distribution visualizations
• Key outputs: Sentiment labels and confidence scores

4. Topic Detection

• Discovers latent topics in the Reddit data
• Uses advanced topic modeling techniques
• Identifies key themes and discussion clusters
• Key outputs: Topic distributions and interpretable topics

5. Embeddings & FAISS-RAG

• Generates semantic embeddings for all text
• Creates FAISS vector store for efficient similarity search
• Implements Retrieval-Augmented Generation (RAG) capabilities
• Enables semantic search and retrieval
• Key outputs: Embedding vectors and indexed vector store

6. LLM Insight Generation

• Leverages Large Language Models to generate insights
• Summarizes discussions and extracts key takeaways
• Answers complex questions about the data
• Generates contextual narratives from RAG results
• Key outputs: AI-generated insights and summaries

7. Evaluation

• Evaluates model performance across all components
• Validates sentiment analysis accuracy
• Assesses topic quality and coherence
• Measures retrieval effectiveness
• Key outputs: Performance metrics and evaluation reports

🛠️ Technologies & Libraries

• Data Collection: PRAW (Reddit API wrapper)
• NLP & Text Processing: NLTK, spaCy, TextBlob
• Topic Modeling: scikit-learn, Gensim
• Embeddings: Sentence Transformers, OpenAI Embeddings
• Vector Search: FAISS (Facebook AI Similarity Search)
• LLM Integration: LangChain, OpenAI API
• Data Analysis: pandas, NumPy
• Visualization: Matplotlib, Seaborn, Plotly
• Notebooks: Jupyter Notebook

🚀 Getting Started

Prerequisites

• Python 3.8+
• Jupyter Notebook
• Reddit API credentials
• OpenAI API key (for LLM features)

Installation

1. Clone the repository:

git clone https://github.com/shauravkhadka/Adncanved-NLP-Reddit.git
cd Adncanved-NLP-Reddit

2. Create a virtual environment:

python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

3. Install dependencies:

pip install -r requirements.txt

4. Set up credentials:
   • Add your Reddit API credentials to www.reddit.com_cookies.txt
   • Set your OpenAI API key as an environment variable

Running the Pipeline

Execute the notebooks in sequential order:

1. Start with 01-Data Collection and Exploratory Data Analysis.ipynb to gather Reddit data
2. Follow with 02-Text Preprocessing.ipynb to clean the data
3. Run sentiment and topic analysis notebooks (03-04)
4. Generate embeddings and vector store (05)
5. Extract insights using LLMs (06)
6. Evaluate results (07)

📁 Output Directories

• data/ - Raw Reddit data and processed datasets
• outputs/ - Generated analysis results, visualizations, and reports
• vectorstore/ - FAISS vector index and embedding stores
• dashboard/ - Interactive dashboard components and configuration

📈 Key Results

This pipeline produces:

• Comprehensive sentiment analysis reports
• Topic hierarchies and interpretations
• Semantic search capabilities via vector embeddings
• AI-generated summaries and insights
• Performance evaluation metrics

🔐 Privacy & Compliance

• Respects Reddit's Terms of Service and API usage guidelines
• Uses proper authentication via official APIs
• Handles user data responsibly
• Suitable for research and educational purposes

🤝 Contributing

Contributions are welcome! Feel free to:

• Submit issues and suggestions
• Fork and create pull requests
• Improve documentation
• Extend the analysis pipeline

📝 License

This project is available for educational and research purposes.

✨ Future Enhancements

• Add multi-language support
• Implement real-time data streaming
• Develop interactive web dashboard
• Integrate additional LLM models
• Add clustering and community detection
• Enhance visualization capabilities

📧 Contact

For questions or collaboration inquiries, please reach out via GitHub issues.

---

Language Composition: 99.1% Jupyter Notebook, 0.9% Python