Improving and Decentralizing Chainlink’s Feature Release and Network Upgrade Process

Chainlink’s accelerated growth to meet increasing demand from the DeFi ecosystem has led us to quickly launch many new Decentralized Oracle Networks for Price Reference Data. At the same time, we’ve been rapidly upgrading these networks to meet the demand for additional features and functionality. In the spirit of transparency, we wanted to alert our community to a pricing anomaly that has been corrected, with only a small overall effect, and assure you that we are continually working to improve our processes.

We take security very seriously and as part of our monitoring service, yesterday we identified a pricing anomaly on one of our oracle networks for the XAG/USD price pair. We acted quickly and the price pair was updated to broadcast the correct information. This pricing anomaly caused a few minutes of downtime for the single affected user on this price pair and affected less than forty thousand dollars.

Importantly, this pricing anomaly was the result of human error and was not an oracle issue—the security and redundancy of our oracles and data sources remain intact. The XAG-USD contract was updated by a highly redundant architecture of 7 different data aggregation providers which were being queried by 9 independent node operator teams of DevOps/security experts distributed around the world. None of the data aggregators nor the node operators were compromised in any way during this anomaly. The human error occurred while seeking to improve the XAG/USD network by releasing additional data reliability features requiring an update to the smart contract’s configuration about the specific job IDs being run by node operators servicing the XAG/USD oracle network, leading them to incorrectly request a gold price (XAU) instead of the silver price (XAG).

We plan to address this issue in the near term by accelerating an increase in multi-signature signers for the release of new features and updating of oracle networks. We’ve always had multi-signature implemented in our contracts, and will now be going on to increase the number of signers, eventually moving to an approach that includes each oracle network’s larger users as signers for that oracle network’s feature upgrades. By increasing our multi-signature signers as a mechanism for decentralizing the feature release and network update process, we can largely eliminate this category of issue.

We sincerely apologize to the affected user and the larger community for any stress this pricing anomaly may have caused. We’re also glad to say that the affected user has committed to continue using the Chainlink network.