Bump github.com/go-git/go-git/v5 from 5.19.1 to 5.19.2 - #18
Open
dependabot[bot] wants to merge 10 commits into
Open
Bump github.com/go-git/go-git/v5 from 5.19.1 to 5.19.2#18dependabot[bot] wants to merge 10 commits into
dependabot[bot] wants to merge 10 commits into
Conversation
Instagram feed posts (/p/, /tv/) were routed to yt-dlp, which cannot
download photos: it fails with "There is no video in this post". In
production that meant 354 of 407 /p/ captures failed (87%), and because
the viewer picked the default tab by type preference without checking
status, every one of them landed visitors on a red "Archive Failed" page
while a working screenshot sat one tab over.
Add a gallery-dl archiver as the image-world counterpart to yt-dlp. It
covers ~300 sites; this routes the post-shaped URLs of 15 of them
(Instagram, X/Twitter, Reddit, Tumblr, Bluesky, Flickr, Imgur,
DeviantArt, ArtStation, Pixiv, Pinterest, Newgrounds, VSCO). Profile and
feed URLs are deliberately excluded — gallery-dl would pull down an
entire account. Instagram reels stay on yt-dlp, which handles them well.
Output is a streamed ZIP holding every downloaded file, gallery-dl's raw
per-file metadata sidecars, and a normalized metadata.json (author,
caption, date, likes, tags, per-file dimensions). A new viewer renders it
as the original post, served from /gallery/:shortid/{list,file/*}.
Rename the "youtube" archive type to "yt-dlp" so a stored artifact names
the tool that produced it. That string is in ~100k rows, in permalinks,
and in API clients, so the old name keeps resolving forever: requests and
job args are normalized on the way in, lookups match both spellings, and
a startup migration rewrites existing rows. The migration moves
archive_items and queued river_job args in one transaction — the cleanup
worker pairs an item with its job by comparing those two values, so
renaming only the items would make every in-flight item look orphaned and
get force-failed while its job was still queued.
Notable behavior choices, with reasoning in the code:
- No --sleep-request override. gallery-dl already waits a randomized
6-12s between Instagram API calls, and because that flag is a root
config key it replaces the per-site defaults rather than acting as a
floor; any value below Instagram's own would make throttling more
likely, not less.
- gallery-dl's exit status is a bitmask, not an enum, so a run can report
several causes at once (4 extraction, 8 challenge, 16 auth, 64 no
extractor). Decoded accordingly.
- Exit 0 does not imply files were written, so success requires at least
one downloaded media file.
- gallery-dl is installed into the same Python environment as yt-dlp: its
default Instagram video path hands DASH manifests to yt-dlp as an
importable module.
Incidental fixes found along the way:
- A failed archive no longer wins the default tab over a completed one.
- Screenshots that fell back to JPEG were served as image/webp.
- itch inherited the 2-minute browser timeout; give it its own.
- s3SeekableReader.Seek dropped its open body even when seeking to the
current offset, costing one ranged GetObject per ~32KB for random
access reads.
- saveArchiveData could return without closing the archiver's reader,
leaking the producing goroutine and its temp directory.
- An unreadable cookies file was fatal at startup; it now warns. Losing
logged-in archiving is bad, losing the whole server is worse.
arker runs as PID 1 and never wait()s on its dead headless_shell children, so they accumulate as defunct processes - about 1k PID slots per day in prod against the container's ~18.9k pids cgroup limit. At the cap, fork fails and archiving breaks. The previous fix (docker run --init on the hand-run container, 2026-07-21) did not survive the move back to Coolify-managed deploys; baking tini into the image works regardless of how the container is launched.
The metadata mapping was derived from Instagram's schema and quietly
produced wrong or empty fields elsewhere. Verified against real captures
from production: Imgur albums and Bluesky posts both rendered with no
author and no like count, and Bluesky showed the wrong text entirely.
Three distinct problems:
- Bluesky uses "text" for the post body and "description" for an image's
alt text. Reading description first put alt text on the page where the
caption belongs. Caption resolution now prefers text/content/caption and
falls back to description, which is still correct for Instagram since it
has no "text" key.
- Sites differ on where the post record lives. Imgur describes the single
image at the top level and keeps the album's title, URL, uploader and
vote counts in a nested "album" object, so every post-level field came
back empty. Lookups now consult known container objects, and the
ambiguous keys ("id", "url") prefer the container so a post links to the
album rather than to one image's CDN URL.
- The poster is modelled as flat fields on some sites and a nested record
on others (Bluesky author{handle,displayName}, Imgur album.account).
resolveGalleryAuthor handles both, and avoids rendering "someone
(someone)" when handle and display name are identical.
Also widen the approval count beyond Instagram's vocabulary: likeCount,
upvote_count, point_count and favourites all mean the same thing to a
reader.
galleryString is now a plain top-level lookup; unwrapping nested people is
resolveGalleryAuthor's job, so callers control whether the top level or a
container wins rather than getting an implicit fallback.
Tests use sidecars captured from the real production archives.
Imgur albums carry upvote_count and favorite_count side by side, and the key order checked favourites first — so a post with 366 upvotes and 0 favourites reported 0 likes. Order the vocabulary by closeness to "likes" instead, since the first match wins and sites expose several at once.
Archives had no visual representation anywhere: the dashboard was a wall
of URL text, and the past-archives dropdown and API returned timestamps
only. Captures now carry a 480x270 JPEG preview, served from
/thumb/{shortid}[/{type}].
Thumbnails are deliberately not an archive type. Type names are permanent,
user-facing identifiers that render as viewer tabs and appear in
permalinks; a preview image is none of those things. They live instead in
four columns on archive_items (thumbnail_key/width/height/status), so the
capture-level preview is derived by walking items in the same preference
order the viewer uses to pick a default tab. The preview therefore matches
what a visitor sees when they follow the link.
JPEG rather than WebP. The only WebP encoder in the tree, nativewebp, is
lossless-only: encoding a downscaled screenshot with it produced a ~490KB
"thumbnail" against ~35KB for JPEG at the same dimensions, and it panics
outright ("too many bits for the given value") on some low-colour-count
inputs, which is exactly what a mostly-blank page decodes to. x/image/webp
is decode-only, which is all that is needed to read stored screenshots back;
it was already in the module graph and is now a direct dependency.
Generation takes two paths, because the cost is wildly different:
- New captures derive the thumbnail inside ScreenshotArchiver from the
image it has already decoded. That decode happens regardless, so the
preview costs one downscale and no extra browser round-trip.
- Archives captured before this change are backfilled by a River job that
the /thumb handler enqueues the first time somebody actually views one.
Work is paid off in the order it is needed rather than as a bulk sweep.
Backfill is never done inline in the request. A full-page screenshot
reaches 3000x20000 (~240MB decoded), and one dashboard render asks for
hundreds of thumbnails at once, so inline generation would take the
process down. The job runs on the existing high_priority queue, which is
bounded at max(2, MaxWorkers/2) and so also caps concurrent large decodes.
River's ByArgs uniqueness collapses the burst: verified against the running
server, 40 concurrent requests for a missing thumbnail created exactly one
job.
thumbnail_status distinguishes "not attempted" from "ready" from
"unavailable". Without that last state the lazy path would re-enqueue an
impossible job (unsupported type, oversized, undecodable bytes) on every
page view. Sources over 40 megapixels are rejected on their header via
image.DecodeConfig, before any decode is committed to.
No thumbnail failure can fail an archive. The archive is the product and
the preview is not; failing an otherwise-good capture over a cosmetic
artifact would burn a River attempt and re-run the whole download.
Supporting changes:
- Archiver.Archive now returns an archivers.Result struct instead of five
positional values, so adding this artifact (and the next one) does not
churn every archiver's signature. Thumbnail bytes are []byte rather than
an io.Reader on purpose: the main artifact's reader is already a live
process or a pipe with delicate close semantics, and a second one with
the same rules would be another way to leak a blocked goroutine.
- Thumbnail keys carry an upload nonce like archive keys, because the
production bucket forbids overwrites and deletes. Regeneration writes a
new object and repoints the row.
- /thumb always returns an image, falling back to a deterministic SVG
placeholder with a short max-age so a refresh picks up the real one.
A broken <img> across hundreds of dashboard rows is worse than a neutral
tile, and callers can render a card unconditionally.
- GET and HEAD both registered, matching /archive/:shortid/:type; caches
and link-preview crawlers probe with HEAD first.
Verified end to end against a real server: archiving hackclub.com produced
a 4.2MB WebP screenshot and a 35.7KB thumbnail with the correct top crop,
AutoMigrate created all four columns and the index, and clearing the
columns and re-requesting drove the full backfill path through
x/image/webp decoding of the stored screenshot.
Adds a Thumbnails section covering the data model, the two generation paths, and the constraints that are easy to violate: never generate inline in a request, always mark permanent failures unavailable, and never let a thumbnail failure fail an archive. Several existing claims had drifted from the code and were actively misleading: - zstd compression is described in four places, but it was removed in 087f7f9 and klauspost/compress is not in go.mod. Objects are stored exactly as the archiver wrote them. An agent trusting this would look for a decompression step that does not exist. - The Archiver interface was documented as Archive(url, writer), which was already wrong before this change and is now further off. Replaced with the real signature and a note on Result. - Archive types listed "YouTube", which was renamed to yt-dlp, and omitted gallery-dl entirely. - The project tree referenced internal/workers/worker.go and internal/archivers/browser_utils.go, neither of which exists. The real files are archive_worker.go / cleanup_worker.go and pwbundle.go / utils.go. Storage was listed as fs.go alone despite s3.go, direct.go and memory_storage.go sitting beside it. - Three of the six listed test files (archiver_test.go, login_text_test.go, vimeo_test.go) do not exist. Rather than replace one stale list with another, that section now describes where tests live and the two conventions worth knowing (in-memory SQLite for DB tests, httptest-driven gin engines for handler tests), and points at go test ./... instead. - The Storage interface notes now record that there is no delete method and that the production bucket is locked, since that constraint drives the nonced-key pattern used throughout.
archive.selfhosted.hackclub.com is NXDOMAIN — it does not resolve at all, so the documented git clone URL and SSH command both fail outright. The header at the top of the file already carried the right values (archive.hackclub.com, and the cloudflared SSH host), but two later sections still had the old ones. Also records two things that cost real time when debugging prod: container names rotate on every Coolify deploy, and one-off tools must not be run inside the app container because a deploy kills it mid-run.
Bumps [github.com/go-git/go-git/v5](https://github.com/go-git/go-git) from 5.19.1 to 5.19.2. - [Release notes](https://github.com/go-git/go-git/releases) - [Changelog](https://github.com/go-git/go-git/blob/main/HISTORY.md) - [Commits](go-git/go-git@v5.19.1...v5.19.2) --- updated-dependencies: - dependency-name: github.com/go-git/go-git/v5 dependency-version: 5.19.2 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com>
dependabot
Bot
added
dependencies
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bumps github.com/go-git/go-git/v5 from 5.19.1 to 5.19.2.
Release notes
Sourced from github.com/go-git/go-git/v5's releases.
Commits
3eeb238Merge pull request #2277 from go-git/checkout-v5008a78fgit: worktree, make the filesystem wrapper a symlink-safe boundary2263fb5Merge pull request #2268 from go-git/renovate/releases/v5.x-go-golang.org-x-t...77b7625build: Update module golang.org/x/text to v0.39.0 [SECURITY]85ea767Merge pull request #2267 from go-git/renovate/releases/v5.x-go-golang.org-x-n...198675abuild: Update module golang.org/x/net to v0.56.0 [SECURITY]4a0e66dMerge pull request #2254 from pjbgf/v5-dotgit-ref-name-containment3b306efstorage: dotgit, align reference-name safety with refname_is_safef3d0cc1storage: dotgit, reject path traversal in reference names979cfe9Merge pull request #2262 from joshblum/joshblum/to-slash-v5Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting
@dependabot rebase.Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR:
@dependabot rebasewill rebase this PR@dependabot recreatewill recreate this PR, overwriting any edits that have been made to it@dependabot show <dependency name> ignore conditionswill show all of the ignore conditions of the specified dependency@dependabot ignore this major versionwill close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)@dependabot ignore this minor versionwill close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)@dependabot ignore this dependencywill close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)You can disable automated security fix PRs for this repo from the Security Alerts page.