Skip to content

README.md

Latest commit

 

History

History
487 lines (337 loc) · 30 KB

README.md

File metadata and controls

487 lines (337 loc) · 30 KB
Raw

proxy-scraper-checker

The TUI during a run: per-protocol progress, the fastest proxies so far and logs

A proxy scraper and checker written in Rust.

It collects HTTP/SOCKS4/SOCKS5 proxies from any number of sources, verifies each one really works, and writes the survivors out with response times, geolocation and network ownership attached.

Features

  • Async Rust, shipped as a single binary for Windows, Linux, macOS, Android, FreeBSD, NetBSD, illumos and Solaris. Nothing to install.
  • Proxies are pattern-matched out of raw text, HTML or JSON, so sources need no cleanup as long as host and port stay together. scheme://user:pass@host:port and CIDR ranges are understood, from URLs or local files.
  • Every proxy has to fetch a real URL in full to survive, so ones that connect and then stall get dropped. Results are deduplicated across sources.
  • Each working proxy comes with its response time, and with its exit IP when check_url returns one. ASN and city-level geolocation are looked up in offline MaxMind databases, so there are no per-proxy API calls.
  • The interactive TUI follows the run through scraping, checking and saving, with per-protocol results, the fastest proxies so far, speed and time left. It rearranges itself for narrow screens, so it also fits a phone held upright. Without a terminal it prints plain logs, with a progress line every 30 seconds of checking, so it also runs under cron and in Docker.
  • Output is JSON with metadata, plus ready-to-paste plain text lists.

Related

If you want proxies without running anything, monosans/proxy-list is updated regularly using this tool.

Safety warning

Checking opens hundreds of simultaneous connections to untrusted hosts. Your ISP may read that as abusive traffic, and cheap routers can drop connectivity when their NAT table fills. Consider a VPN, and lower max_concurrent_checks if a run destabilizes your network.

Quick start

Every option is documented inline in config.toml. The binary reads config.toml from the current directory; set PROXY_SCRAPER_CHECKER_CONFIG to point it somewhere else. Colors follow the terminal's theme and turn off with NO_COLOR.

Pre-built archives are produced by CI from the latest commit on main and published as the latest release, see Verifying downloads. Open the section for your system.

Windows

  1. Download the archive for your processor:

    Processor Download
    64-bit Intel or AMD (almost every PC) Download
    ARM64 (Snapdragon and other ARM laptops) Download
    32-bit Download

    Not sure? Open Settings → System → About and look at "System type": x64-based processor means 64-bit Intel or AMD, ARM-based processor means ARM64, and 32-bit operating system needs the 32-bit build whatever the processor.

  2. Extract the archive to a dedicated folder.

  3. Edit config.toml.

  4. Run proxy-scraper-checker.exe. If SmartScreen shows "Windows protected your PC", click "More info", then "Run anyway".

macOS

  1. Download the archive for your processor:

    Processor Download
    Apple Silicon (M1 and newer) Download
    Intel Download

    Not sure? Apple menu → About This Mac shows "Chip: Apple M…" on Apple Silicon and "Processor: … Intel …" on Intel.

  2. Extract the archive to a dedicated folder.

  3. Edit config.toml.

  4. Run it from Terminal in that folder:

    ./proxy-scraper-checker

    If macOS refuses because it cannot verify the developer, clear the download quarantine and run it again:

    xattr -d com.apple.quarantine proxy-scraper-checker
Linux

  1. Run uname -m and download the archive from the matching row. The glibc build suits most distributions (Ubuntu, Debian, Fedora, Arch and so on). Each glibc download notes the oldest glibc it runs on (ldd --version prints yours). Take the musl build on Alpine, on an older glibc, or if the glibc build refuses to start.

    Several rows share i686, armv7l, armv6l or riscv64. On i686, take the row without SSE2 only if grep -w sse2 /proc/cpuinfo prints nothing. On armv7l and armv6l, the system has hardware floating point if /lib/ld-linux-armhf.so.3 (glibc), /lib/ld-musl-armhf.so.1 or /lib/ld-musl-armv7.so.1 (musl) exists; otherwise take a row without hardware floating point. The Thumb-2 build also needs neon in /proc/cpuinfo. On riscv64, take the RVA23 build only if the processor implements the RVA23 profile.

    Processor uname -m glibc musl
    64-bit Intel or AMD x86_64 Download (2.17+) Download
    64-bit ARM (Raspberry Pi 3 and newer on a 64-bit OS, ARM servers) aarch64 Download (2.17+) Download
    32-bit ARMv7 (Raspberry Pi 2 and newer on a 32-bit OS) armv7l Download (2.17+) Download
    32-bit ARMv6 (Raspberry Pi 1, Zero, Zero W) armv6l Download (2.17+) Download
    32-bit Intel or AMD i686 Download (2.17+) Download
    64-bit RISC-V riscv64 Download (2.29+) Download
    LoongArch loongarch64 Download (2.36+) Download
    64-bit PowerPC, little-endian ppc64le Download (2.32+) Download
    64-bit PowerPC, big-endian ppc64 Download (2.39+) Download
    32-bit PowerPC ppc Download (2.39+) —
    IBM Z s390x Download (2.17+) —
    64-bit SPARC sparc64 Download (2.39+) —
    32-bit ARMv5 armv5tel Download (2.23+) Download
    32-bit x86 without SSE2 (Pentium III, Athlon XP and older) i686 Download (2.17+) Download
    32-bit ARMv7 without hardware floating point armv7l Download (2.27+) Download
    32-bit ARMv6 without hardware floating point armv6l Download (2.17+) Download
    32-bit ARMv7 with NEON, Thumb-2 build armv7l Download (2.23+) —
    64-bit RISC-V with the RVA23 profile riscv64 Download (2.39+) —

  2. Extract the archive to a dedicated folder.

  3. Edit config.toml.

  4. Run it from that folder:

    ./proxy-scraper-checker
Android / Termux

Install Termux from F-Droid, not Google Play (why?).

  1. Install with one command:

    bash <(curl -fsSL 'https://raw.githubusercontent.com/monosans/proxy-scraper-checker/main/termux.sh')

    Running it again updates the binary and resets config.toml.

  2. Edit the config in a text editor:

    nano ~/proxy-scraper-checker/config.toml

  3. Run it:

    cd ~/proxy-scraper-checker && ./proxy-scraper-checker

The script picks the right archive by itself. To fetch one by hand, match the output of getprop ro.product.cpu.abi:

getprop ro.product.cpu.abi Download
arm64-v8a Download
armeabi-v7a Download (without NEON)
armeabi Download
x86_64 Download
x86 Download
FreeBSD

  1. Run uname -m and download the archive from the matching row:

    Processor uname -m Download
    64-bit Intel or AMD amd64 Download
    64-bit ARM arm64 Download
    32-bit Intel or AMD i386 Download

  2. Extract the archive to a dedicated folder.

  3. Edit config.toml.

  4. Run it from that folder:

    ./proxy-scraper-checker
NetBSD

  1. Download the 64-bit x86 archive. It needs NetBSD 10 or newer.

  2. Extract the archive to a dedicated folder.

  3. Edit config.toml.

  4. Run it from that folder:

    ./proxy-scraper-checker
illumos

  1. Download the 64-bit x86 archive.

  2. Extract the archive to a dedicated folder.

  3. Edit config.toml.

  4. Run it from that folder:

    ./proxy-scraper-checker
Solaris

  1. Run uname -p and download the archive from the matching row. It needs Solaris 11.4 or newer.

    Processor uname -p Download
    64-bit x86 i386 Download
    SPARC sparc Download

  2. Extract the archive to a dedicated folder.

  3. Edit config.toml.

  4. Run it from that folder:

    ./proxy-scraper-checker
Docker on Windows

The Docker image logs to stdout instead of showing the interactive TUI.

  1. Install Docker Desktop.

  2. Download the archive for your processor:

    Processor Download
    64-bit Intel or AMD (almost every PC) Download
    ARM64 (Snapdragon and other ARM laptops) Download

    Not sure? Open Settings → System → About and look at "System type": x64-based processor means 64-bit Intel or AMD, ARM-based processor means ARM64.

  3. Extract it to a folder and edit config.toml.

  4. Build and run from that folder:

    docker compose build
docker compose up --no-log-prefix --remove-orphans

Results land in out inside that folder; output.path is ignored in Docker.

Docker on Linux / macOS

The Docker image logs to stdout instead of showing the interactive TUI.

  1. Install Docker Compose.

  2. Run uname -m and download the archive from the matching row:

    Processor uname -m Download
    64-bit Intel or AMD (including Intel Macs) x86_64 Download
    64-bit ARM (Apple Silicon Macs, Raspberry Pi on a 64-bit OS) aarch64, arm64 on macOS Download
    32-bit ARMv7 armv7l Download
    32-bit Intel or AMD i686 Download
    64-bit PowerPC, little-endian ppc64le Download
    64-bit RISC-V riscv64 Download
    IBM Z s390x Download

  3. Extract it to a folder and edit config.toml.

  4. Build and run from that folder. The build arguments make the files in ./out belong to your user:

    docker compose build --build-arg UID=$(id -u) --build-arg GID=$(id -g)
docker compose up --no-log-prefix --remove-orphans

Results land in ./out; output.path is ignored in Docker.

Build from source

  1. Install the Rust toolchain, see https://rust-lang.org/tools/install/

  2. Clone the repository:

    git clone https://github.com/monosans/proxy-scraper-checker.git
cd proxy-scraper-checker

  3. Build and run it with the TUI:

    cargo run --features tui --release --locked

    Leave out --features tui to get plain log output instead of the interactive interface.

The binary lands in target/release/proxy-scraper-checker on Linux/macOS, or target\release\proxy-scraper-checker.exe on Windows.

Verifying downloads

Every archive is built by GitHub Actions from the public source and published by the CI workflow straight from GitHub's runners, never from someone's computer:

  • Each release links to a build provenance attestation that GitHub signs through Sigstore. It lists the SHA-256 of every archive together with the commit and the workflow run that built it, and only that workflow run could have produced the signature. Whatever went into a build, the source and the workflow included, is therefore in the public history of this repository.
  • Releases are immutable: once published, their files cannot be replaced.

To check a downloaded archive without installing anything:

  1. Compute its SHA-256: Get-FileHash <archive> in PowerShell, shasum -a 256 <archive> on macOS, sha256sum <archive> on Linux and Termux, sha256 <archive> on FreeBSD.
  2. Open the "Build provenance" link in the notes of the latest release and find the same hash next to the archive name. The page also shows the commit and the workflow run it was built from. Only the latest release is kept; attestations of older builds stay on the attestations page.

To check the signature itself, use the GitHub CLI:

gh attestation verify <archive> \
  --repo monosans/proxy-scraper-checker \
  --signer-workflow monosans/proxy-scraper-checker/.github/workflows/ci.yml

The Docker images are attested the same way, and the prebuilt Docker archives pin the image by digest, so docker compose build uses exactly the image CI built.

Keys

Key Action
q, ESC Stop the run and save the working proxies found so far. Again to quit
Ctrl-C Stop the run and leave the TUI at once. The save still happens
↑, PageUp, k Scroll the logs up
↓, PageDown, j Scroll the logs down
End Back to the latest log lines

Output

out/
├── proxies.json          working proxies with metadata, compact
├── proxies_pretty.json   the same data, indented
└── proxies/
    ├── all.txt           every proxy, with its protocol prefix
    ├── http.txt          one file per enabled protocol, no prefix
    ├── socks4.txt
    └── socks5.txt

A text line is host:port, or username:password@host:port for a proxy that needs credentials, with protocol:// in front of either in all.txt.

The JSON files hold an array of entries like this one:

{
  "protocol": "socks5",
  "username": null,
  "password": null,
  "host": "1.2.3.4",
  "port": 1080,
  "timeout": 0.42,
  "exit_ip": "5.6.7.8",
  "asn": {
    "autonomous_system_number": 12345,
    "autonomous_system_organization": "Example Networks"
  },
  "geolocation": {
    "country": { "iso_code": "US", "names": { "en": "United States" } },
    "city": { "names": { "en": "Chicago" } },
    "location": { "latitude": 41.85, "longitude": -87.65, "time_zone": "America/Chicago" }
  }
}

timeout is how long the whole request took, in seconds: connection, request and the complete response. That is also what sort_by_speed orders by. exit_ip is the address the target site actually saw, so comparing it with host spots proxies that forward through somewhere else.

exit_ip, asn and geolocation are only filled in when check_url returns an IPv4 exit address, so pointing it at an ordinary web page leaves all three null. A database that fails to download doesn't stop the run: the copy from an earlier run is used, and without one its field stays null. geolocation mirrors the GeoLite2 City record (abbreviated above); fields the database has no data for are omitted, and names are English only.

Stopping a run early writes out the working proxies found so far, whether from the keys above or with Ctrl-C or SIGTERM when there is no TUI. A run that ends with no proxies to write leaves the previous run's files as they were.

With sort_by_speed = false the text files list the same proxies in the same order on every run, so committing them to a repository shows a diff only when the list changed.

Sponsors

Want your name in this section? Support the project and it goes here.

Support this project

Star the repository so other people can find it. If you are interested in sponsoring, DM me on Telegram.

License

MIT

The binaries contain third-party code under its own licenses, listed in LICENSE-THIRD-PARTY. CI generates it for every build, and it comes with LICENSE in every release archive and next to the binary in /app of the published Docker image. An image you build yourself includes it only if the file is in your checkout.

The ASN and geolocation databases are not part of the binaries: the program downloads them at run time. They are GeoLite Data, covered by the GeoLite End User License Agreement.

This product includes GeoLite Data created by MaxMind, available from https://www.maxmind.com.