0.1.1: Restrict Crawl Scope and Filter Locale Query Variants - #6
Merged
Conversation
Add site2skill/url_filter.py with is_url_allowed() function that: - Restricts crawl scope to descendants of the starting URL - Rejects URLs with only localization query params (hl, lang, locale) - Allows URLs with content-switching query params (version, tab, etc.) - Supports custom excluded query key sets for future configurability Add test_url_filter.py with 22 tests covering all specified scenarios. Co-authored-by: laiso <39830+laiso@users.noreply.github.com>
Co-authored-by: laiso <39830+laiso@users.noreply.github.com>
Copilot
AI
changed the title
[WIP] Update over-crawl prevention plan for document URLs
Add URL filter for crawl scope restriction and localization query exclusion
Mar 7, 2026
Created demo.md using showboat with executable code blocks demonstrating: - CLI help output - Full example converting HTML to a skill with --skip-fetch - Output skill structure (SKILL.md, references/, scripts/) - Converted markdown with YAML frontmatter - URL filter module (localization query rejection, path scoping) - Test suite execution (26 tests passing) Co-authored-by: laiso <39830+laiso@users.noreply.github.com>
Copilot
AI
changed the title
Add URL filter for crawl scope restriction and localization query exclusion
Add URL crawl-scope filter and showboat demo document
Mar 7, 2026
laiso
changed the title
laiso
marked this pull request as ready for review
March 7, 2026 03:20
laiso
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
To prevent excessive crawling by
site2skill, the crawl scope during fetching has been linked to the starting URL. This ensures that only the starting URL and its sub-paths are targeted, rather than the entire domain, while excluding localization-specific queries such ashl,lang, andlocale.In conjunction with this, the version has been updated to
0.1.1and a changelog has been added.Changes
fetch_site()--include-directories,--accept-regex, and--reject-regextowgethl,lang, orlocaleare excluded by default.fetch_site.0.1.1.CHANGELOG.md.Why
The previous implementation was only restricted by
--domains=<host>, which made it easy for large documentation sites likedeveloper.android.comto spread from the starting page to related pages on the same host or to localized URLs.With this change, the crawling boundary logic has been connected to the execution path of
fetch_site(), ensuring that only the intended scope is retrieved.Example
Starting URL:
Expected behavior:
are included.
are excluded.
On the other hand, general queries that may switch content are not completely rejected.
💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.