Fix streamed EPG ingestion and atomic profile saves - #185
Merged
Conversation
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
|
|
||
| // The authoring UI may change profile metadata, canonical channel choices, and | ||
| // output entries together. Save them atomically, then rebuild the guide once. | ||
| router.put('/api/output-profiles/:slug', requireAuth, async (req, res) => { |
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
There are confirmed performance/operational issues (notably getGuideData allocating all programmes when unscoped, plus cache TTL timer behavior on dynamic TTL updates) and the new atomic save path needs structured error identifiers to preserve row-level feedback.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
This PR addresses two core scalability/usability issues in the IPTV Proxy: (1) bounded-memory EPG/XMLTV ingestion (including gzip streams) with programme indexing for guide queries, and (2) atomic “Save Output Profile” behavior via a single HTTP/MCP operation to avoid repeated refresh cycles.
Changes:
- Add streaming XMLTV decoding/parsing utilities with decoded-size limits and gzip support, and use them in EPG ingestion.
- Build and use an in-memory programme index to answer
/api/guide(and MCP guide reads) without reparsing the full merged XMLTV document. - Introduce an atomic output-profile save API (HTTP
PUT /api/output-profiles/:slug) and MCP tool (save_output_profile), and update the admin UI + tests accordingly.
File summaries
| File | Description |
|---|---|
| test/unit/xmltv-stream.test.js | Adds unit coverage for chunked parsing, gzip decoding, size limits, and a representative “large feed” scenario. |
| test/unit/cache-manager.test.js | Adds coverage for byte-bounded cache eviction behavior. |
| test/integration/output-profile-routes.test.js | Verifies atomic save behavior across profile metadata, canonical edits, and output entries. |
| test/integration/mcp.test.js | Adds integration coverage for the new save_output_profile MCP tool. |
| server/xmltv-stream.js | Introduces streaming XMLTV read/limit utilities (decodeXmltvStream, readXmltvRecords). |
| server/mcp.js | Adds save_output_profile MCP tool and surfaces it in workflow metadata. |
| server/epg.js | Switches ingestion to streamed XMLTV reading, adds byte-bounded EPG cache, and introduces programme indexing for guide queries. |
| server/canonical.js | Adds atomic HTTP save route for output profile edits + refresh/invalidate behavior. |
| libs/output-profile-service.js | Implements transactional saveOutputProfile() to apply UI edits atomically. |
| libs/cache-manager.js | Adds max-bytes cache bounding and background expiry purging. |
| admin/src/App.vue | Replaces many sequential PATCH calls with a single atomic PUT save request. |
Review details
- Files reviewed: 11/11 changed files
- Comments generated: 5
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+2481
to
2484
| const json = await response.json(); | ||
| if (!response.ok) { | ||
| throw new Error(buildApiErrorMessage(json, 'Failed to save output profile')); | ||
| } |
Comment on lines
+65
to
+70
| // Expiry must not depend on a later read of the same cache key. The | ||
| // timer is unref'd so a cache never keeps the service/test process alive. | ||
| this.cleanupTimer = | ||
| this.ttl > 0 | ||
| ? setInterval(() => this.purgeExpired(), Math.max(1000, Math.min(this.ttl, 60000))).unref() | ||
| : null; |
Comment on lines
+399
to
+410
| if (result?.error === 'canonical-not-found') { | ||
| return res.status(404).json({ error: `Canonical channel not found: ${result.canonicalId}` }); | ||
| } | ||
| if (result?.error === 'binding-not-found' || result?.error === 'guide-binding-not-found') { | ||
| return res.status(404).json({ error: `Channel binding not found for canonical channel ${result.canonicalId}` }); | ||
| } | ||
| if (result?.error === 'entry-not-found') { | ||
| return res.status(404).json({ error: `Output profile entry not found for canonical channel ${result.canonicalId}` }); | ||
| } | ||
| if (result?.error === 'invalid-entry') { | ||
| return res.status(400).json({ error: `Invalid output profile entry for canonical channel ${result.canonicalId}` }); | ||
| } |
Comment on lines
+195
to
+197
| let programmes = tvgId | ||
| ? guideProgrammeIndex.get(tvgId) || [] | ||
| : Array.from(guideProgrammeIndex.values()).flat(); |
Comment on lines
+62
to
+66
| const closeTag = `</${activeType}>`; | ||
| const openingEnd = buffer.indexOf('>'); | ||
| const selfClosing = openingEnd !== -1 && /\/\s*$/.test(buffer.slice(0, openingEnd)); | ||
| const closeIndex = selfClosing ? -1 : buffer.toLowerCase().indexOf(closeTag); | ||
| const endIndex = selfClosing ? openingEnd + 1 : closeIndex === -1 ? -1 : closeIndex + closeTag.length; |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #172
Fixes #179
Validation