Apple/Metal Client Roadmap #1911
Replies: 2 comments 3 replies
My proposal in #1782 is to encode platform information in the manifest names. This way, if a deprecation ever happened, the manifest could be deleted, indicating a dead end, or a dummy "your mac is too old" app could be shipped down in that manifest. If a shard operator builds universal binaries, they can stick the same app bundle into all of the different manifests. Or just turn the platform naming off and live on the wild side. |
|
I think encoding the same universal binary in each of the architecture manifests is the right move. That even covers when a user moves their installation from an Intel Mac to an ARM Mac (during something like Mac setup migration.) The error flow would need some adjustment. Right now Mac users who get onto a shard without a corresponding Mac manifest just get a generic can't find manifest error. That error should be improved to be something like the shard not supporting your client's platform. |
|
I didn't put it on the list - but there is also the code signing part of this. We might have established that we will not sign the client ourselves on behalf of all the shards. (We might need to sign it for Gehn.) I'm not sure all the other shards will want to sign up for a $99 Apple dev account, and there is low risk to our own account because any binary modification breaks the signing. But if we have trouble with deployment we could revisit that later. That does mean signing needs to occur somewhere between our Github build process and the manifest process. But I don't know where that signing process should live. I would guess it should be part of a tool like UruManifest? If UruManifest is grabbing builds from our repo - it will need to sign them as part of it's gather process or during staging. Universal binary creation could go into that same processing stage as well if we didn't want to do it as part of GitHub actions. The problem with both is that those actions would need to be performed on a Mac. I'm not aware of a way to perform an architecture lipo or a code sign on anything but a Mac. But maybe @dpogue knows about some tooling that I don't. :) |
I think LLVM ships a Codesigning I think needs to happen on a macOS machine, especially because IIRC macOS codesigning is actually notarizing which is a slightly different process than codesigning for iOS/visionOS? |
|
Right. I forgot the build will need to be uploaded to Apple for notarization. The build still needs to be code signed - but then uploaded to Apple and notarized as a follow on task. Having an upload + async wait step would add a lot of overhead to UruManifest. I don't have any great ideas there. |
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Writing up a discussion post with the current state of the Mac client, along with roadmap for the Metal renderer and iOS client. I'll update this post as things change.
Mac Client
The Mac client is currently building for both Apple and Intel based Macs. It requires a Metal 2 compatible GPU, but optional Metal 3 renderer features are working their way in. For hardware - we test on 2013 Macs and newer. The oldest GPU on the test bench is an Intel HD 4000. There is more detail on the testing configurations in #1534.
There is only one rendering feature currently open, along with two build/distribution issues.
Bump mapping
Metal bump mapping is ready for review in #1825. This PR emulates bump maps exactly as they were in the Windows pipeline. The lack of bump maps is the last currently known render difference in the Mac client.
Universal Binaries
We compile the Apple Silicon and Intel builds separately. Ideally - we should combine them into a single universal binary. Apple provides tooling to do this. Our Github Actions workflow just needs to be adapted to stitch the builds together.
Manifest creation
The update manifests don't support a Mac client right now. There is a PR on UruManifest to add support for the Intel Mac client to the manifest system:
Hoikas/UruManifest#12
This PR would need to be updated to support a universal Mac build instead. I'd also like to know what our policy is if we drop macOS versions or processor architectures. Xcode seems to be forcing us to drop deployment support for older versions of macOS. And Intel Mac support may not exist forever in the tooling. We should be prepared for what happens when a shard starts delivering client updates that drop support for some Macs.
Metal renderer
Work on the Metal renderer has mostly shifted to optimization. Plasma is often still CPU bound.
Vsync and scheduling improvements
Apple has made improvements to how frames are scheduled for Metal based rendering. #1877 and #1860 are both focused on replacing the game loop with Apple's new callback driven draw architecture. We have to supply enough detail to the new classes that the operating system can monitor our rendering and adapt our scheduling based on performance and other conditions.
Bindless rendering
A major source of CPU usage is the amount of uniform and texture binding we need to do in order to render a frame. Argument buffers are one solution to this problem. Instead of binding uniforms and resources one and a time - you cache the bind states on the GPU. This is similar to UBOs in OpenGL - but argument buffers can also hold references to textures, samplers, and even other buffers. And unlike UBOs - argument buffers don't use bind slots, can be stored as an array, and don't have the same count limits as uniform bind points.
The bump mapping support - #1825 - was a test of argument buffers. Bump mapping was complicated to generalize with uniform bind points. So bump mapping is instead being encoded as an array of argument buffers.
As a next step - I'd like to bring the entire layer system into Metal as argument buffers. Layer descriptions and texture pointers would be cached as argument buffers. Each pass would be cached as an array of the argument buffer layers. This would remove the core pass/layer loop when drawing a span and remove a lot of CPU overhead. This must be done carefully. Some features like popover layers and piggybacks can invalidate layer state.
I have an initial branch with this working - but am waiting for the Metal bump mapping PR to clear first. There are also differences between Metal 2 and Metal 3. In Metal 2 I need to use an encoder object to create the argument buffers. In Metal 3 I can just create POD structs in C++ and send them directly over to the shaders as buffers.
Argument buffers could also enable features like unlimited layers per pass - but it's not clear this would be useful.
Character skinning improvements
Character skinning is also a big CPU drain - especially in large multiplayer settings. I've looked at moving this to a GPU compute pipeline. I've also looked at threading. The way Plasma exposes structures the data and operations is making optimization difficult. With the current shape a lot of small and sparse jobs get dispatched instead of larger or more dense jobs. I'm still looking at options.
iOS client
The iOS client inherits the Metal renderer directly from macOS - so there are no rendering differences. The rendering feature set is the same.
One minor change was that software decompression of DXT textures was added as an optional feature in the Metal pipeline. This was added in #1778 for supporting older Apple chipsets that don't support hardware DXT compression.
SwiftUI client
We wrote the client front end in AppKit for the Mac. But we'd have to write a new front end for iOS. There is a branch that implements a SwiftUI client that can run on iOS. This also integrates Swift into our builds for Apple platforms. I've been doing a lot of experimentation with Swift/C++ interop.
This work has also exposed a lot of issues with code duplication in the client code. I'm working on separating out cross platform code from Mac only code and reorganizing the client to have more reusable classes.
The SwiftUI client is not merged and no PR is open yet. I'm not planning on migrating the Mac client to SwiftUI at this time.
KI input
KI input on iOS is broken. The iOS version uses the GameController framework for keyboard input instead of the macOS event system. GameController only sends us USB HID codes. It does not send us actual characters appropriate for chat or the console. We'd need to figure out a solution for character input.
Distribution
The big blocker for the iOS client is the App Store. Shards would not be directly able to patch an iOS client because Apple controls distribution. Every shard would have to submit to the Mac app store and that seems unsustainable.
I don't think we have any solutions to this problem - and it could block iOS distribution completely.
Vision Pro
There is a native Vision Pro build. One reason I was eager to get a Metal pipeline into place was for wherever Apple went with VR. I'm not actively working on this build right now. It builds off of the SwiftUI client work. It was not planned for full VR. I was looking at adding stereoscopic rendering - and it was intended to be played as a flat window with a mouse and a keyboard. But the contents in the window would look 3D.
All reactions