Incident Report: Bridge stopped by validators
This blog post will provide updates on the situation and expected timelines for resolution as soon as they are available. Updates will also be shared via the official Vega Twitter and Discord channels.
Date & Time
2024/02/17 15:40 (GMT)
Status
[Monitoring]
Classification
Under investigation (Major/Critical)
For full details of how incidents are classified, please see this blog.
Description
On the 17th February at 11:18 (GMT) the Vega to Ethereum bridge was stopped by validator actions following reports from the community of a potential intentional exploit and/or manipulation of markets targeting one or more liquidity providers.
This means that both deposits to, and withdrawals from the Vega network are not possible at this time. This includes withdrawals that were started on Vega but not completed.
Given the timing of the reported attack, we do not believe that any funds from this attack have left the network, as any withdrawals will have been subject to the 24 hour withdrawal delay present in the alpha version, and are now on hold while the bridge is stopped.
The Vega team and impacted community members are investigating these events and will provide updates along with any recommendations (such as reverting certain transfers), and if bugs are found, fixes will be made available.
The validators are expected to re-enable the bridge once they are happy that the issue is resolved, or if they decide that no further action should be taken.
Note that any resolutions, as well as control of the bridge are outside of the control of the Vega team, which can only investigate using the publicly available chain data and make recommendations.
The bridge stop transaction can be seen here:
https://etherscan.io/tx/0xde22bcbaa2408265d9835b1b05eec99b9e3713a3ec4e367f9efbdd1e77a695d0
There is no immediate risk to any funds held on the Vega mainnet, as the bridge has been stopped. There is no risk to user’s staked VEGA tokens due to this incident. Staked tokens are never under the control of the Vega network and are always held only in users’ Ethereum wallets.
Since this incident involves validator actions and the possibility of a patch release to Vega, validators and node operators are requested to be on high alert in case of the need to coordinate a deployment of the protocol between themselves if the project team publishes an updated release.
This blog post will provide updates on the situation and expected timelines for resolution as soon as they are available. Updates will also be shared via our usual Twitter and Discord channels.
Updates
2024/02/23 19:30 (GMT)
Manipulator withdrawals stopped and funds redistributed
Vega validators and community signalled their agreement with the reports recommendation to return the funds from these withdrawals to those keys that lost money as a result of the incident, but not to touch the funds held by other unrelated users participating in the market who happened to be net gainers. The Vega team assisted by preparing and testing the code for a patch to implement this.
Following this, a quorum of validators voted on-chain to upgrade mainnet to include the patch and execute these actions. This was completed at approximately 10:00 UTC. The patch, and the actions within it can be reviewed here.
Validators also signed the required Ethereum transactions to burn the withdrawal nonces for three withdrawals that were identified as being from the party responsible for the market manipulation. The effect of this was to permanently cancel these withdrawals. The transactions are linked below:
Withdrawal 1: https://etherscan.io/tx/0xabd3be3eda413d351bc4b5e1aeda22fe67fbe096ddf3b506c9f67b8c65b0b302
Withdrawal 2: https://etherscan.io/tx/0xeebd252ec78095fceba989c367cb8bc3cc4898f4165273460ad76ab63b65179e
Withdrawal 3: https://etherscan.io/tx/0xf3e7a3a748b12003d44016d5b9666b661803a394d712a63666e5497d9421d51a
At approximately 5:45pm UTC the transaction to restart the bridge was sent and withdrawals and deposits are now being processed again.
It is possible to verify the solvency of the bridge by confirming that the total balances from across all accounts on the Vega chain (API docs) matches the total balance of each asset in the ERC20 bridge asset pool contract (docs, balances on Etherscan).
The validators and the project team will continue to monitor the network and notify the community of any further issues. Please reach out on Discord if you encounter problems with deposits or withdrawals.
Markets remain suspended by governance
As the markets on Vega were suspended by community governance, they remain in an auction state in which normal trading is not possible.
Given that some parameter changes were recommended to reduce the chance of a repeat of the events of the past week, and that Palazzo Mistero was deployed with many new features, it is expected that the community will vote to both update market configurations and resume trading on these markets in the near future.
You can follow along with the discussion around these proposals on the forums.
2024/02/20 19:40 (GMT)
The Vega project team have completed their investigation and report into possible manipulation and market abuse in the LDO/USDT-PERP market from 14/02/2024 to 17/02/2024 that led to the Ethereum ERC20 bridge being stopped by a quorum of validators at the request of the community.
The team concluded that there is clear evidence of premeditated and coordinated price manipulation of the market with the intention of extracting profit at the expense of other network participants. They further concluded that the manipulator took significant steps to conceal their identity by using a large number of keys, and that they were not acting as a good faith researcher or bounty hunter given, among other things, their immediate attempt to withdraw the excess gains.
The recommendation of the Vega team is therefore that the community should not allow the withdrawal of gains made by the manipulator and their associated keys. Instead the team have recommended splitting the amounts of these withdrawals pro-rata between the keys that lost funds as a result of the price manipulation and market abuse.
Some of these keys are owned by LPs that take significant risk, and made large losses during these events. It is important that the community recognise the value such LPs bring and their importance to the success of the project, and the team believes that the community should demonstrate this by using governance (either through the stake delegated to validators, or an on-chain vote) to return funds to those that lost them.
The team recommends reading the entire report, found here.
2024/02/19 20:00 (GMT)
The Vega team continues to investigate and prepare its report on the events of the past week related to the LDO/USDT-PERP market.
In the interim, a patch release (0.73.14) has been published. This patch will suspend all markets at the time of the protocol upgrade and additionally set the funding margin scaling factor to 0 for each market. This is in order to ensure that similar manipulation or exploitation of markets is not possible in the time before the community is able to review market parameters and update them through governance; if necessary.
If validators choose to deploy this patch, it will mean that trading is not possible until markets are reopened via governance.
The team recommends deploying this patch ASAP in order to protect the current markets and participants on the network.
There is no change from the previous update to the expectation that the Ethereum ERC20 bridge will likely remain stopped until the community, including validators, are able to read and process the report following this incident.
2024/02/18 16:00 (GMT)
The Vega project team are currently still investigating the events related to this incident and are working on a detailed report and any associated recommendations. This report will be made available to the community as soon as possible.
At this time, the team expects that validators will not decide to re-enable the Ethereum bridge until after this report and the associated data have been made available and the community has reached consensus on how to proceed. Therefore it remains that both deposits to, and withdrawals from the Vega network are not possible at this time.
