Job Scraper

A Python project for scraping and extracting structured data from job postings. The project uses Playwright for web scraping, trafilatura for content extraction, and spaCy for natural language processing.

Features

• Asynchronous web scraping with Playwright
• Content cleaning and extraction with trafilatura
• Structured field extraction using spaCy
• Support for known job platforms (Workday, Greenhouse, etc.)
• FastAPI server for API access
• Command-line interface for batch processing
• Comprehensive error handling and logging

Installation

1. Clone the repository:

git clone <repository-url>
cd job-scraper

2. Create a virtual environment and activate it:

python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

3. Install dependencies:

pip install -r requirements.txt

4. Install Playwright browsers:

playwright install chromium

5. Download spaCy model:

python -m spacy download en_core_web_sm

Usage

Command Line Interface

Process job postings from URLs and save results to a JSON file:

python main.py --urls "https://example.com/job1" "https://example.com/job2" --output jobs.json

Options:

• --urls: List of job posting URLs (required)
• --output: Output JSON file path (default: jobs.json)
• --headless: Run browser in headless mode (default: True)
• --server: Run as FastAPI server instead of processing URLs

API Server

Start the FastAPI server:

python main.py --server

The server will be available at http://localhost:8000 with the following endpoints:

• POST /extract: Extract job data from URLs

  {
    "urls": ["https://example.com/job1", "https://example.com/job2"],
    "headless": true
  }

• GET /health: Health check endpoint

API documentation is available at:

• Swagger UI: http://localhost:8000/docs
• ReDoc: http://localhost:8000/redoc

Project Structure

job_scraper/
├── api/
│   └── api.py           # FastAPI server
├── cleaner/
│   └── cleaner.py       # Content cleaning with trafilatura
├── extractor/
│   └── extractor.py     # Job data extraction with spaCy
├── scraper/
│   └── scraper.py       # Web scraping with Playwright
├── main.py              # Main script and CLI
├── requirements.txt     # Project dependencies
└── README.md           # This file

Performance Considerations

• The project uses the lightweight en_core_web_sm spaCy model for efficiency
• Browser instances are managed asynchronously to handle multiple URLs concurrently
• Content extraction is optimized for job postings
• Platform-specific selectors are used when available for faster extraction

Error Handling

• Comprehensive error handling at each stage of processing
• Detailed logging with rotation and retention policies
• Failed URLs are reported in the output with error messages

License

MIT License