Transformer model performance on Apple Neural Engine

[MarcelGruenauer]

I downloaded two networks:

• kata1-tf3-b11c768-s11001M-d5973M.bin.gz
• kata1-zhizi-b28c512nbt-muonfd2.bin.gz

On a MacBook Air M5, the 28-block convolutional network gets about 5x the number of visits per second than the 11-block transformer network.

This may be because the Apple Neural Engine (ANE) backend is optimized for convolutional networks, not transformer networks, but I'm not knowledgable about neural networks in general, just a user.

So I wanted to know which network is stronger, given equal wall clock time.

I ran 116 matches between the 28-block network and the 11-block network, with pondering off and ten seconds per move. The 11-block network uses GPU+ANE; the 28-block network only uses ANE. These configs were determined by benchmarking both networks.

There are some games that have identical positions up to move 55 or even 75, but not that many.

Running the matches took about five days. Here are the results.

The convolutional network won more games, so I won't switch to the transformer network. AI-Sensei now uses transformer networks, so I assume that other backends runs them more efficiently.

Also both networks win about 53% of games as White.

I can supply the games and config files, if you want.

Results:

Games: 116

Engine Wins
b11    49
b28    67

Color Wins
Black  55
White  61

Engine Wins by Color
           Black White
b11 wins      23    26
b28 wins      32    35

Actual Thinking Time
b11   10.245 s/move over 11815 moves (121041.631 s total)
b28   10.041 s/move over 11824 moves (118727.409 s total)

Variation
Pairwise shared opening prefix: max 55 moves, median 1.0, average 3.5 over 6670 pairs

Longest Shared Opening Prefixes
 55 moves shared between games 014 and 016
 50 moves shared between games 011 and 075
 48 moves shared between games 031 and 065
 45 moves shared between games 026 and 064
 44 moves shared between games 026 and 098
 44 moves shared between games 052 and 114
 44 moves shared between games 064 and 098
 44 moves shared between games 072 and 112
 43 moves shared between games 059 and 085
 43 moves shared between games 072 and 110

Shared Positions
position shared at game 015 move 75 and game 079 move 75 (75 stones)
position shared at game 015 move 67 and game 027 move 67 (67 stones)
position shared at game 027 move 67 and game 079 move 67 (67 stones)
position shared at game 015 move 63 and game 091 move 63 (63 stones)
position shared at game 027 move 61 and game 091 move 61 (61 stones)
position shared at game 079 move 61 and game 091 move 61 (61 stones)
position shared at game 014 move 55 and game 016 move 55 (55 stones)
position shared at game 011 move 50 and game 075 move 50 (49 stones)
position shared at game 031 move 48 and game 065 move 48 (48 stones)
position shared at game 026 move 45 and game 064 move 45 (45 stones)