Skip to content

README.md

Latest commit

 

History

History
251 lines (193 loc) · 7.41 KB

README.md

File metadata and controls

251 lines (193 loc) · 7.41 KB
Raw

__# GF-TADs Data Analysis and Visualization

This project provides a comprehensive solution for extracting, analyzing, and visualizing data from GF-TADs (Global Framework for the Progressive Control of Transboundary Animal Diseases) documents.

🎯 Features

Data Extraction

  • PDF Processing: Extract text from PDF documents using multiple libraries (pdfplumber, PyPDF2)
  • Entity Extraction: Identify What, When, Who, Where, Impact, and Objectives from unstructured text
  • Natural Language Processing: Use spaCy and NLTK for advanced text analysis
  • Confidence Scoring: Assign confidence scores to extracted information

Data Analysis

  • Temporal Analysis: Track activities and recommendations over time
  • Organization Analysis: Identify key stakeholders and their involvement
  • Objective Categorization: Classify activities by their strategic objectives
  • Impact Assessment: Analyze the expected impact of different activities

Visualization

  • Interactive Dashboards: Comprehensive overview of all extracted data
  • Timeline Visualizations: Track activities across meetings and years
  • Word Clouds: Visual representation of key terms and concepts
  • Network Analysis: Relationships between organizations and activities
  • Statistical Analysis: Confidence scores and data quality metrics

Web Interface

  • Streamlit Dashboard: User-friendly web interface for data analysis
  • Real-time Processing: Extract and analyze data in real-time
  • Export Capabilities: Download results in Excel, CSV, or JSON format

📋 Requirements

Python Dependencies

  • PyPDF2==3.0.1
  • pdfplumber==0.10.0
  • pandas==2.1.4
  • numpy==1.25.2
  • matplotlib==3.8.2
  • seaborn==0.13.0
  • plotly==5.17.0
  • wordcloud==1.9.2
  • spacy==3.7.2
  • nltk==3.8.1
  • scikit-learn==1.3.2
  • textblob==0.17.1
  • streamlit==1.29.0

Additional Setup

# Install spaCy English model
python -m spacy download en_core_web_sm

# NLTK data will be downloaded automatically when first run

🚀 Quick Start

1. Installation

# Clone or download the project files
# Navigate to the project directory
cd path/to/gftad/project

# (Recommended) Create and activate a virtual environment (Windows):
python -m venv .venv
.venv\Scripts\activate

# Install dependencies
pip install --upgrade pip setuptools wheel
pip install -r requirements.txt

# Install spaCy model (if using spaCy features)
python -m spacy download en_core_web_sm

2. Prepare Your Data

Organize your GF-TADs documents in the following structure:

your_project_folder/
├── GSC_Recommendations/
│   ├── document1.pdf
│   ├── document2.pdf
│   └── ...
├── GSC_Reports/
│   ├── report1.pdf
│   ├── report2.pdf
│   └── ...

3. Run the Analysis

Option A: Web Interface (Recommended)

streamlit run streamlit_app.py

Then open your browser to http://localhost:8501

Option B: Command Line

from data_extractor import GFTADsDataExtractor
from visualizer import GFTADsVisualizer

# Extract data
extractor = GFTADsDataExtractor("path/to/your/data/folder")
df = extractor.process_all_documents()
output_file = extractor.save_extracted_data(df, 'excel')

# Create visualizations
visualizer = GFTADsVisualizer(df=df)
visualizer.save_visualizations()

📊 Usage Examples

Extract Data from PDFs

from data_extractor import GFTADsDataExtractor

# Initialize extractor
extractor = GFTADsDataExtractor(r"c:\path\to\your\gftad\folder")

# Process all documents
df = extractor.process_all_documents()

# Save results
output_file = extractor.save_extracted_data(df, format='excel')
print(f"Data saved to: {output_file}")

Create Visualizations

from visualizer import GFTADsVisualizer

# Create visualizer
visualizer = GFTADsVisualizer("path/to/extracted_data.xlsx")

# Generate overview dashboard
overview = visualizer.create_overview_dashboard()
overview.show()

# Create timeline visualization
timeline = visualizer.create_activity_timeline()
timeline.show()

# Save all visualizations
visualizer.save_visualizations("output/folder")

Generate Summary Report

# Generate comprehensive summary
report = visualizer.generate_summary_report()
print(f"Total activities: {report['total_activities']}")
print(f"Average confidence: {report['avg_confidence']:.2f}")

📁 Output Files

The system generates several types of output files:

Extracted Data

  • gftads_extracted_data_YYYYMMDD_HHMMSS.xlsx - Main data file with all extracted information
  • gftads_extracted_data_YYYYMMDD_HHMMSS.csv - CSV version for broader compatibility
  • gftads_extracted_data_YYYYMMDD_HHMMSS.json - JSON format for programmatic access

Visualizations

  • overview_dashboard_YYYYMMDD_HHMMSS.html - Interactive overview dashboard
  • activity_timeline_YYYYMMDD_HHMMSS.html - Timeline visualization
  • objectives_analysis_YYYYMMDD_HHMMSS.html - Objectives analysis
  • confidence_analysis_YYYYMMDD_HHMMSS.html - Confidence score analysis
  • wordclouds_YYYYMMDD_HHMMSS.png - Word cloud visualizations
  • summary_report_YYYYMMDD_HHMMSS.json - Comprehensive summary report

🔧 Configuration

Customizing Keywords

You can customize the keywords used for entity extraction by modifying the setup_keywords() method in the GFTADsDataExtractor class:

self.keywords = {
    'time_indicators': ['by', 'before', 'after', 'during', ...],
    'action_verbs': ['develop', 'implement', 'establish', ...],
    'organizations': ['FAO', 'OIE', 'WHO', ...],
    # Add your custom keywords here
}

Adjusting Confidence Scoring

Modify the calculate_confidence_score() method to adjust how confidence scores are calculated based on your specific requirements.

🐛 Troubleshooting

Common Issues

  1. PDF Extraction Fails

    • Ensure PDFs are not password-protected
    • Try different PDF processing libraries if one fails

  2. spaCy Model Not Found

    python -m spacy download en_core_web_sm

  3. Memory Issues with Large PDFs

    • Process documents in smaller batches
    • Increase system memory or use a machine with more RAM

  4. Poor Entity Extraction Quality

    • Adjust keywords in the configuration
    • Improve preprocessing steps
    • Consider using more advanced NLP models

Performance Tips

  • Parallel Processing: For large document collections, consider implementing parallel processing
  • Caching: Use caching for repeated analyses
  • Preprocessing: Clean and preprocess text for better extraction quality

📈 Advanced Usage

Custom Entity Extraction

You can extend the entity extraction by creating custom extraction methods:

def extract_custom_entities(self, text):
    # Your custom extraction logic here
    return extracted_entities

Integration with Other Systems

The extracted data can be easily integrated with:

  • Business Intelligence tools (Power BI, Tableau)
  • Database systems (SQL Server, PostgreSQL)
  • Web applications and APIs
  • Machine learning pipelines

🤝 Contributing

Feel free to contribute to this project by:

  • Adding new visualization types
  • Improving entity extraction algorithms
  • Enhancing the user interface
  • Adding support for additional document formats

📄 License

This project is provided as-is for educational and research purposes.

📞 Support

For questions or issues, please check the troubleshooting section above or refer to the inline documentation in the code files.