[Wayland] Editor Process Hangs on Close

[GeneralProtectionFault]

Tested versions

Reproducible on master to date, basically (currently c614335)

System information

Operating System: Artix Linux x86_64 KDE Plasma Version: 6.7.4 KDE Frameworks Version: 6.28.0 Qt Version: 6.11.1 Kernel Version: 7.1.6-artix1-1 (64-bit) Graphics Platform: Wayland Processors: 12 × AMD Ryzen 5 5600X 6-Core Processor Memory: 96 GiB of RAM (94.2 GiB usable) Graphics Processor: NVIDIA GeForce RTX 3060 Manufacturer: ASUS

Issue description

Reproducible on master to date.

This appears to only happen when setting the Prefer Wayland option, virtually certain it's Wayland-specific.
Often, I'd see Redot hung in my process manager, several copies if I didn't clean it up. This happens to the Project Manager and Editor alike (clear difference as one Redot instance would take up ~2-300 MB, while the editor, even of a fresh project, takes up around 1 GB (memory).

I purposely opened & closed the dev build and got some GDB back traces.

Editor Hang:

[New LWP 4369]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/libthread_db.so.1".
0x00007f750389fdf2 in ?? () from /usr/lib/libc.so.6
(gdb) bt
#0  0x00007f750389fdf2 in ?? () from /usr/lib/libc.so.6
#1  0x00007f7503893c18 in ?? () from /usr/lib/libc.so.6
#2  0x00007f7503893f53 in ?? () from /usr/lib/libc.so.6
#3  0x00007f7503899035 in ?? () from /usr/lib/libc.so.6
#4  0x00005640d7a9af7c in std::thread::join() ()
#5  0x00005640d6dcadc9 in Thread::wait_to_finish (this=0x564101852a30) at core/os/thread.cpp:94
#6  0x00005640d1c5beeb in WaylandThread::destroy (this=0x564101852a30) at platform/linuxbsd/wayland/wayland_thread.cpp:4927
#7  0x00005640d1c3bb8f in DisplayServerWayland::~DisplayServerWayland (this=0x5641018527d0) at platform/linuxbsd/wayland/display_server_wayland.cpp:2233
#8  0x00005640d1c3edc9 in memdelete<DisplayServer> (p_class=0x5641018527d0) at ./core/os/memory.h:152
#9  0x00005640d1c7dbda in finalize_display () at main/main.cpp:425
#10 0x00005640d1ca877f in Main::cleanup (p_force=false) at main/main.cpp:5212
#11 0x00005640d1baf9dc in main (argc=1, argv=0x7ffe6f32b358) at platform/linuxbsd/godot_linuxbsd.cpp:124

Project Manager Hang:

Project Manager hang:
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/libthread_db.so.1".
0x00007f06c0c9fdf2 in ?? () from /usr/lib/libc.so.6
(gdb) bt
#0  0x00007f06c0c9fdf2 in ?? () from /usr/lib/libc.so.6
#1  0x00007f06c0c93c18 in ?? () from /usr/lib/libc.so.6
#2  0x00007f06c0c93c61 in ?? () from /usr/lib/libc.so.6
#3  0x00007f06c0d0ee9a in read () from /usr/lib/libc.so.6
#4  0x00007f06c0c8e27e in _IO_file_underflow () from /usr/lib/libc.so.6
#5  0x00007f06c0c90aaf in _IO_default_uflow () from /usr/lib/libc.so.6
#6  0x00007f06c0c81d77 in _IO_getline_info () from /usr/lib/libc.so.6
#7  0x00007f06c0c80977 in fgets () from /usr/lib/libc.so.6
#8  0x000055bd669a4e48 in OS_Unix::execute (this=0x7ffc49813c80, p_path=..., p_arguments=..., r_pipe=0x7ffc4980fce0, r_exitcode=0x7ffc4980fcc0, 
    read_stderr=false, p_pipe_mutex=0x0, p_open_console=false) at drivers/unix/os_unix.cpp:903
#9  0x000055bd64e63059 in handle_crash (sig=11) at platform/linuxbsd/crash_handler_linuxbsd.cpp:114
#10 <signal handler called>
#11 0x000055bd64f0c6e3 in WaylandThread::screen_get_data (this=0x55bd7a3d22f0, p_screen=2) at platform/linuxbsd/wayland/wayland_thread.cpp:4191
#12 0x000055bd64ee720f in DisplayServerWayland::screen_get_scale (this=0x55bd7a3d2090, p_screen=2)
    at platform/linuxbsd/wayland/display_server_wayland.cpp:662
#13 0x000055bd64eab488 in DisplayServer::screen_get_max_scale (this=0x55bd7a3d2090) at servers/display_server.h:373
#14 0x000055bd67461012 in EditorSettings::get_auto_display_scale () at editor/settings/editor_settings.cpp:1888
#15 0x000055bd64f4f03e in Main::setup2 (p_show_boot_logo=true) at main/main.cpp:3341
#16 0x000055bd64f4cd08 in Main::setup (execpath=0x7ffc498157e0 "/home/q/Documents/Repos/redot-engine-gpf3/bin/redot.linuxbsd.editor.dev.x86_64", 
    argc=0, argv=0x7ffc498142c0, p_second_phase=true) at main/main.cpp:2910
#17 0x000055bd64e62964 in main (argc=1, argv=0x7ffc498142b8) at platform/linuxbsd/godot_linuxbsd.cpp:109

Very similar, but not identical, so I'm throwing both in here.

There's even a comment in the code towards the end of /platform/linuxbsd/wayland/wayland_thread.cpp:

// Dispatches events until a frame event is received, a window is reported as
// suspended or the timeout expires.
bool WaylandThread::wait_frame_suspend_ms(int p_timeout) {
	// This is a bit of a chicken and egg thing... Looks like the main event loop
	// has to call its rightfully forever-blocking poll right in between
	// `wl_display_prepare_read` and `wl_display_read`. This means, that it will
	// basically be guaranteed to stay stuck in a "prepare read" state, where it
	// will block any other attempt at reading the display fd, such as ours. The
	// solution? Let's make sure the mutex is locked (it should) and unblock the
	// main thread with a roundtrip!
	MutexLock mutex_lock(mutex);
	wl_display_roundtrip(wl_display);

	if (is_suspended()) {
		// All windows are suspended! The compositor is telling us _explicitly_ that
		// we don't need to draw, without letting us guess through the frame event's
		// timing and stuff like that. Our job here is done.
		return false;
	}

...lending more weight to this being a Wayland threading issue.

Steps to reproduce

• Set Prefer Wayland in options
• Open your favorite process/system monitor (filter for "Redot" to make it easy)
• Open the engine & a project, close. For me, a few times is usually enough to see it hung, ymmv.

Minimal reproduction project (MRP)

N/A (Not project-specific)