SwapDEX Chain-Report for the incident on Friday, March 11th

Summary

On March 10th, 2022 a data center managed by Contabo GmbH in Duesseldorf experienced an outage due to an internet fiber cable being physically cut during construction work in proximity of the data center.

The outage affected SwapDEX validator slots at block 322.904 and lasted for 14 hours and 32 minutes. During that time 296 of 300 validator slots managed by the SwapDEX community received an “unapplied slash” in era 92.

The unapplied slash means that it can be reverted through governance action, which is exactly what is happening at the moment.

Customer Impact

All slashes in eras 92 to 104 are in the process of being reverted due to the force majeure nature of the outage. For the avoidance of doubt, any future slashes due to a centralized VPS management system by community operators won’t be reimbursed. We highly recommend our community revisit its VPS architecture and ensure an appropriate degree of decentralization.

What Happened

On March 10th, 2022 a data center managed by Contabo GmbH in Duesseldorf experienced an outage due to an internet fiber cable being physically cut. The outage caused an immediate connection interruption of approx. 60% (180 e.a.) of all SwapDEX validator slots.

The SwapDEX network protocol detected the subsequent unresponsiveness as the validators failed to send the “I’m online” heartbeat signal. This signal is submitted on-chain automatically every session. Furthermore, the network protocol kept track of the payable actions performed (blocks produced in BABE, uncles references, validity statements) which acts as an additional measure for responsiveness.

The SwapDEX network protocol classified the simultaneous failure of approx. 60% of the validators as an attack since an outage of that magnitude threatens the network’s ability to produce blocks. This could be observed in block production times peaking around the 60 seconds mark (usually 6 sec).

Validators that lost connection or failed to perform a payable action due to the subsequent network instability were therefore perceived as a threat to the network and received slash penalties of 7% of all SDX bonded on their validator slots (self and nominator bond) and the loss of all nominators.

Due to validator slots trying to revalidate prior to the data center being fully online again the network protocol conducted further slashes in a span from era 93 to 104 leading to a total amount of 7.2M SDX worth $324,000 USD slashed.

Lessons Learned

  • Need to spread validator hosting across many providers and datacenters across the globe.
  • Need to motivate the community to invest in some higher spec VPSs to bear the load in case of outages.

What’s next

  • The council is going to propose motions to reimburse slashes covering era 92 to 104

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store