Repository navigation
Synapse prefers the most recent notary response for a key over the one with the highest validity #20358
Description
Description
When fetching keys to verify a event, synapse seems to prefer the last stored entry over the one for the key with the highest validity, causing errors to validate events when the notary that replies last, is serving a cached / outdated result.
>>> store = hs.get_datastores().main
>>> d = store.get_server_keys_json([("v2.chrrreeeeesss.com", "ed25519:YhTtqaku")])
>>> d
<Deferred at 0x770110ff5720 current result: {('v2.chrrreeeeesss.com', 'ed25519:YhTtqaku'): FetchKeyResult(verify_key=<nacl.signing.VerifyKey object at 0x770105696ee0>, valid_until_ts=1791023409553)}>
here we can see that the key that was returned has a valid_until_ts of 1791023409553
however, in my database there is a earlier response for the same key with a higher valid_until_ts:
synapse=# SELECT from_server, ts_added_ms, ts_valid_until_ms
FROM server_keys_json
WHERE server_name = 'v2.chrrreeeeesss.com' AND key_id = 'ed25519:YhTtqaku'
ORDER BY ts_added_ms;
from_server | ts_added_ms | ts_valid_until_ms
-----------------------+---------------+-------------------
usbpc.xyz | 1779422969518 | 1779825190708
matrix.org | 1779422969534 | 1779985573784
feline.support | 1779422969537 | 1779985573784
aguiarvieira.pt | 1779422969583 | 1780027769023
littlevortex.net | 1779422970189 | 1780027769200
tchncs.de | 1790066522126 | 1790434468552
unredacted.org | 1790066522213 | 1790671321484
t2bot.io | 1790066522723 | 1790671322625
t2l.io | 1790066522772 | 1790671322645
element.io | 1791051504134 | 1791293744653
maunium.net | 1791051504153 | 1791638934867
mgamers.com | 1791051504596 | 1791656304485
bridges.codestorm.net | 1791987290714 | 1792575348070
bi-vibes.com | 1791987300698 | 1791023409553
we can see that the latest response from bi-vibes.com is chosen over the one from bridges.codestomr.net despite the response from bridges.codestorm.net being the only one with a sufficiently large validity,
leading to the following issue when trying to verify a given event event:
Matrix-Codestorm matrix-synapse[518687]: 2026-10-14 16:15:10,707 - synapse.federation.federation_client - 442 - WARNING - sentinel - event id $B7Mi9iCa1PiCLlU-QsQAujUgG6D5hX5bpX979RNXDxg: unable to verify signature for sender domain v2.chrrreeeeesss.com: 401: Failed to find any key to satisfy: _FetchKeyRequest(server_name='v2.chrrreeeeesss.com', minimum_valid_until_ts=1791950306948, key_ids=['ed25519:YhTtqaku'])
Steps to reproduce
- configure 2 notaries in your config
- server A first, then server B
- server B will serve a outdated / cached result for example key
- server A will serve a key response that will satisfy the request
- make sure server B's response arrives after server A's response
- synapse will fail to verify the event even though the currently is a key in the db that checks all boxes
Homeserver
codestorm.net
Synapse Version
1.162
Installation Method
Debian packages from packages.matrix.org
Database
PostgreSQL / No / No / No
Workers
Multiple workers
Platform
Debian LXC on Proxmox
Configuration
No response
Relevant log output
/Anything else that would be useful to know?
would suggest to prefer the key response with the highest minimum_valid_until_ts instead of the answer that came in last.
Activity