Rare sync message failures

[KaloyanTanev]

🐞 Bug Report

Description

In big enough clusters we observe sync messages failing in 0-3% of the cases.
It might be false positive but needs to be checked.

Has this worked before in a previous version?

I think it was always the case in clusters with >1000 validators.

[pinebit]

Follow the internal conversation in Slack about this problem and viable solutions.

[KaloyanTanev]

The pitfall of the failed sync messages was the difference in the recognised head from different VCs. Given that VCs obtain their head root by different means, we cannot guard the sync message data fetching behind a consensus, as we do with other duties.

That said, the only sensible way of achieving sync message data consensus is the nodes to agree on the same head at 1/3 slot time. It did seem like a tons of work, something that does not seem suitable for round robin and QBFT. It can in practise be achieved via round robin and QBFT, however, there it is highly likely for performance to drop. Currently in a 7-node if there is 1 slow node, always seeing an old head, the rest 6 nodes will manage to gather threshold sigs on the new head. And if we supply deterministic head with round robin + QBFT, in this same cluster in 1/7 sync messages there will always be a late head and our vote will be discarded. Moreover, we will need to provide this agreed upon head on both GET /eth/v1/beacon/states/{state_id}/root and the SSE head subscription.

Currently we are trying to improve the attestation head votes for a lot of clusters. This is mainly related to BN deployments and configurations. With this effort we will be improving the sync message success as well.