drm: fixes for compositor-less embedded use: cold start, SHM robustness, cursor support (legacy/software/scaling) - #794
Open
mirko wants to merge 6 commits into
Open
Conversation
added 6 commits
July 18, 2026 21:13
On a display with no prior modeset (no fbcon/fbdev emulation, no firmware POST pass - e.g. a bootloader handing straight over to bare KMS), the drm platform failed to initialize: - The CRTC was picked only by matching an encoder's CURRENT binding (encoder->crtc_id), which is 0 on cold start, aborting with "no crtc for encoder found". Prefer the target connector's own encoders and fall back to the first CRTC an encoder can drive (possible_crtcs). This also fixes picking another output's encoder whose CRTC cannot drive the chosen connector. Note that the incomplete display setup resulting from the old selection is also what left e.g. radeon DCE4.1's hardware cursor fetch uninitialized (cursor rendered as garbage) - a full cold-start modeset cures that as well. - init_cursor() treated a failing initial cursor-plane set as fatal, but at platform-setup time the CRTC is usually not lit yet - the renderer performs the first modeset on the first frame commit - so the initial upload may legitimately fail. Log and carry on instead: the pointer-motion handler re-sets the plane on every move and succeeds once a mode is active. The paths that remain fatal now emit a warning naming the failing step. Signed-off-by: Mirko Vogt <foss@mirko.in>
Three related fixes for the modeset renderer's SHM path, found running WPE on software rendering (llvmpipe/kms_swrast): - Never store the renderer in the buffer resource's user_data: for SHM buffers user_data is owned by libwayland's shm implementation - it IS the wl_shm_buffer. Overwriting it made wl_shm_buffer_get() return the renderer object from the second attach onwards: garbage width/stride, spurious "deferred resize pending" warnings, and a crashing copy loop. Keep the owning renderer in the buffer_object instead. - Release the exported SHM buffer right after copying it into the dumb buffer instead of parking it until the frame retires: holding it keeps an external reference on the wl_shm pool, deferring client-side pool resizes into stale mappings. - Harden the copy: bail out on a NULL data pointer, clamp to the bo's dimensions, and skip frames whose exported buffer carries impossible geometry (dispatching frame-complete so the client's pacing survives). Persistently bogus exports terminate the process for a clean supervisor restart rather than continuing on corrupted state. Signed-off-by: Mirko Vogt <foss@mirko.in>
Legacy (non-atomic) drivers - radeon, older nouveau - expose no universal cursor plane, only primary planes, so the plane-based cursor could never work there. Fall back to drmModeSetCursor()/ drmModeMoveCursor() on the CRTC, re-uploading on pointer motion since a modeset silently disables the legacy cursor with no notification. The cursor image also rendered wrong on hardware cursors: - Several drivers only display cursor buffers of exactly the size advertised by DRM_CAP_CURSOR_WIDTH/HEIGHT (typically 64x64) and read tightly packed rows regardless of the BO's (aligned) pitch. Allocate 64x64 and place the image top-left with a fixed 64-pixel stride. - KMS blending expects premultiplied alpha; the image data is straight alpha. Premultiply during conversion - otherwise pixels with color > alpha wash out additively and only the dark outline stays visible. Signed-off-by: Mirko Vogt <foss@mirko.in>
COG_PLATFORM_DRM_CURSOR=sw composites the built-in pointer image into the scanout dumb buffer instead of using a hardware cursor plane, for setups where hardware cursors are unavailable or undesirable. The cursor is baked into every frame after the SHM copy; on pure pointer motion it is moved in place on the live scanout bo (single map session restoring a save-under patch and blending premultiplied SRC-OVER at the new position, throttled to display rate to avoid raster flicker). Only CPU-writable dumb buffers from the SHM path are ever touched; with dmabuf (hardware-rendered) frames the software cursor stays hidden. "1" keeps the existing hardware plane/legacy behaviour. Signed-off-by: Mirko Vogt <foss@mirko.in>
cursor.x/y are unsigned, so `cursor.x += get_dx()` followed by `if (cursor.x < 0)` cannot work: the comparison is always false, and a negative position wraps around to a huge value which the upper bound check then pins to the opposite edge. Moving the pointer past the top or left screen edge teleported it to the bottom or right edge, from where it could be moved endlessly "through" the screen; the right and bottom edges behaved because their clamp is real. Accumulate the motion in floating point and clamp at zero before storing into the unsigned fields. Signed-off-by: Mirko Vogt <foss@mirko.in>
The cursor is drawn at the fixed physical size of its 16x16 artwork, so on a scaled view (--device-scale) it shrinks relative to everything else - at 2x it looks half its intended size. Enlarge the artwork by the device scale factor (nearest neighbour, rounded to the nearest integer) when building the cursor: up to 4x for the hardware cursor, bounded by its 64x64 buffer, and the same factor for the modeset renderer's composited software cursor, whose buffers are sized for the maximum. Signed-off-by: Mirko Vogt <foss@mirko.in>
Author
|
Partially related to: Igalia/WPEBackend-fdo#206 |
mirko
added a commit
to mirko/video
that referenced
this pull request
Jul 18, 2026
Six upstream-ready patches against 0.18.5 (developed and verified on a Dell Wyse 5010: radeon PALM/DCE4.1, both llvmpipe/SHM and r600/dmabuf paths): - 001: cold-start initialization. CRTC selection required an existing encoder binding (none exists without a prior fbcon/firmware modeset), kms_screen_create crashed on modeless connectors, init_cursor used the possibly-dead screens[0]. The incomplete display setup from the old CRTC selection is also what left DCE4.1's cursor fetch uninitialized - initially misdiagnosed as a hardware/kernel bug. - 002: SHM exported-buffer handling. Root cause of a crash family: the renderer stored itself in the buffer resource's user_data, which for SHM buffers IS libwayland's wl_shm_buffer - from the second attach on, wl_shm_buffer_get() returned the renderer as buffer metadata. Plus early buffer release (avoids deferred pool-resize stale mappings) and copy hardening. - 003: legacy hardware cursor (drmModeSetCursor) for non-atomic drivers without a universal cursor plane, with correct 64x64 tightly-packed premultiplied cursor image. - 004: software cursor option (COG_PLATFORM_DRM_CURSOR=sw), composited into SHM frames with in-place motion updates; dmabuf-safe. - 005: cursor.x/y are unsigned, so the existing 'if (cursor.x < 0)' lower clamp was dead code; moving past the top/left edge wrapped the position around and the upper clamp teleported the cursor to the opposite edge. Clamp in floating point before the unsigned store. Present upstream (master) as well. - 006: scale the 16x16 cursor artwork by the view's device scale factor (nearest neighbour; up to 4x within the 64x64 hardware cursor buffer, same factor for the modeset renderer's software cursor), so the pointer keeps its apparent size next to a --device-scale'd UI. Makefile: pin the complete meson option set of 0.18.5 (platforms drm/headless/wayland, wpe_api 2.0, libmanette, plugin path). PKG_RELEASE 9. Patchset already proposed upstream: Igalia/cog#794 Assisted-By: Claude Fable + Opus Signed-off-by: Mirko Vogt <mirko-openwrt@nanl.de>
mirko
added a commit
to mirko/video
that referenced
this pull request
Jul 18, 2026
Six upstream-ready patches against 0.18.5 (developed and verified on a Dell Wyse 5010: radeon PALM/DCE4.1, both llvmpipe/SHM and r600/dmabuf paths): - 001: cold-start initialization. CRTC selection required an existing encoder binding (none exists without a prior fbcon/firmware modeset), kms_screen_create crashed on modeless connectors, init_cursor used the possibly-dead screens[0]. The incomplete display setup from the old CRTC selection is also what left DCE4.1's cursor fetch uninitialized - initially misdiagnosed as a hardware/kernel bug. - 002: SHM exported-buffer handling. Root cause of a crash family: the renderer stored itself in the buffer resource's user_data, which for SHM buffers IS libwayland's wl_shm_buffer - from the second attach on, wl_shm_buffer_get() returned the renderer as buffer metadata. Plus early buffer release (avoids deferred pool-resize stale mappings) and copy hardening. - 003: legacy hardware cursor (drmModeSetCursor) for non-atomic drivers without a universal cursor plane, with correct 64x64 tightly-packed premultiplied cursor image. - 004: software cursor option (COG_PLATFORM_DRM_CURSOR=sw), composited into SHM frames with in-place motion updates; dmabuf-safe. - 005: cursor.x/y are unsigned, so the existing 'if (cursor.x < 0)' lower clamp was dead code; moving past the top/left edge wrapped the position around and the upper clamp teleported the cursor to the opposite edge. Clamp in floating point before the unsigned store. Present upstream (master) as well. - 006: scale the 16x16 cursor artwork by the view's device scale factor (nearest neighbour; up to 4x within the 64x64 hardware cursor buffer, same factor for the modeset renderer's software cursor), so the pointer keeps its apparent size next to a --device-scale'd UI. Makefile: pin the complete meson option set of 0.18.5 (platforms drm/headless/wayland, wpe_api 2.0, libmanette, plugin path). PKG_RELEASE 9. Patchset already proposed upstream: Igalia/cog#794 Assisted-By: Claude Fable + Opus Signed-off-by: Mirko Vogt <mirko-openwrt@nanl.de>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This series makes the DRM platform more reliable on embedded appliances that boot straight into cog with no wayland compositor, no fbcon modeset before launch - and on GPUs outside the atomic-modesetting mainstream (e.g. r600). It was developed and deployed on an x86 kiosk fleet spanning Intel (i915, atomic KMS) and radeon (legacy KMS, DCE4.1) hardware, plus pure software rendering (llvmpipe/kms_swrast).
Notes:
Disclaimer: I used Claude by Anthropic to help with debugging and fixing addressed issues.