Hello to the Secret community from SCRT Labs!
This post covers major recent and upcoming network improvements in response to high on-chain demand that has led to some scaling issues experienced by Secret users. Read to learn about our four biggest recent network improvements, SCRT Labs’ work on upcoming upgrades, and why we’ve never been more excited about Secret Network’s potential as a universal privacy hub for Web3!
Over the past six months, the Secret Network ecosystem has exploded with a variety of different projects and added hundreds of new users. In our industry, this is generally known as “a good problem to have”. For users, however, it can cause many actual issues as network contributors and infrastructure race to meet new levels of demand.
Periods of high demand on any blockchain often cause UX issues, especially when many users are all rushing to complete the same action on-chain. Secret is no exception: recently we’ve seen airdrop claims, popular NFT drops, and high market volatility lead to perceived slowness or unresponsiveness.
Secret Network features never-before-built technology for protecting data privacy. This functionality is what allows developers to deploy applications that aren’t possible on any other blockchain. It empowers users with privacy protections that we believe should be required in Web3.
Still, since our technology is new and different, we are constantly solving problems that have not been solved elsewhere. With that in mind, Secret Network itself has not halted or broken. Many of these challenges were to be expected with increased demand. Our main goal at this time is to address this colossal growth by implementing short-term and long-term improvements focused on network scalability and a dramatically improved UX.
We’ve taken major steps in a protracted amount of time in order to address our users’ valid concerns because we want to make sure using the chain is as efficient and as cost-effective as possible.
Upgrades So Far
Four immediate improvements have been recently implemented. We’ve already seen positive results with some of the latest high-traffic events (including NFT mints):
- Query nodes
- Improved Infrastructure for Keplr
- On-chain proposals to better control block times
- Adjusting gas prices in Keplr
Query nodes
During times of high network demand, like an exciting NFT drop, users reported that they could not view their balances on Keplr.
We analyzed the issue and discovered that Secret Contracts cannot process transactions and queries in parallel. Moreover, queries cannot process in parallel. The stringency of default LCD connection timeouts unfortunately served to compound the issue.. That’s why nodes that were serving a lot of queries had a hard time keeping up with blocks and returned stale results or simply timed-out. This resulted in an unacceptable user experience for everyone on the network.
To combat these issues, we created a special type of node specifically for serving queries. We slightly modified Tendermint, removed some locks contention and enabled rocksdb data storage. Early results show great promise:
Bright blue – Cluster A serving 20k queries per hour at an average of 1.79s per query
Red – Cluster B serving 20k queries per hour at an average of 1.62s per query
Dark blue – A single query node serving 20k queries per hour at an average of 0.213s per query
For more info see https://github.com/scrtlabs/SecretNetwork/releases/tag/v1.2.5
Improved Infrastructure for Keplr
It’s important that wallet software remains responsive even when an app or the network has issues. Keplr is the wallet standard for the Cosmos blockchain, so SCRT Labs has collaborated with other Cosmos ecosystem partners in order to fix a problem affecting the entire network with this specific wallet.
Keplr has historically used a single infrastructure provider for their Secret Network nodes. This has led to situations where the infrastructure provider could not keep up with the peak loads from both Keplr and other network apps, leading to outages. Even worse, since both Keplr and the apps suffered, these situations appeared to be a network wide defect.
To address this issue, we and our partners have implemented a new infrastructure solution for Keplr. This solution involves global load balancing, caching optimization and smart rate limiting. These fixes reduce the load that hits the nodes themselves.
Implementation of this improved Keplr infrastructure along with our new query nodes have led to a 30-50% reduction of traffic, resulting in faster and more acceptable response times.
On-chain proposals to better control block times
In the past couple of months, we saw the release of two new apps that utilize heavy cryptographic operations:
- Shinobi Protocol uses ed25519 signature verification to verify Secret Network block and secp256k1 message signing to sign Bitcoin transaction
- Shade Protocol uses secp256k1 signature verification to process their cross-chain SHD token airdrop.
These two apps on their own caused block times to stretch from the target block time of 6s up to 90s or more. This highlighted the fact that our gas metering mechanism should be better adjusted to reflect real world resource consumption (namely, processing time).
To fix the core issue requires a software change and a hard fork. In the meantime, some community members suggested lowering the block gas limit in order to get more consistent block times.
In response to this request, we submitted an on-chain proposal to lower the block gas limit from 10M to 4M: https://www.mintscan.io/secret/proposals/76.
However, the limit was found to be too restrictive when trying to upload a new NFT contract. To correct the situation, the ChameleonVille team proposed to increase the block gas limit to 6M: https://www.mintscan.io/secret/proposals/80
Adjusting gas prices in Keplr
Transactions on Secret Network are usually processed by the Keplr wallet which overrides the gas price that’s provided by the app.
Keplr has not at this time updated their gas prices to be reflective of the current market – the last time these values were updated was in 2020. This causes users to pay fees that are way above what’s needed to get their transaction accepted. The market has drastically changed since these prices were set. The network has evolved and validators have lowered their accepted gas prices. To reflect this reality, we submitted a PR to Keplr to lower the default gas price by 2.5x. The changes should take effect within the next few days.
We at Secret Labs would like to express a huge thanks to all of the community members who researched the currently accepted gas prices by validators!
Improving Future Upgrades
Fortunately for users and developers, these are just first steps towards a much more scalable Secret – and we can now also look forward to a better process for handling upcoming network upgrades which will introduce further improvements.
Back in November 2021 during the Supernova upgrade, we also added the x/upgrade module, which allows submitting a software upgrade proposal. If a software upgrade proposal is accepted, the chain halts at the height/time that’s specified in the proposal. To complement this, cosmovisor is a process manager for Cosmos SDK chains and can be used to do automatic chain upgrades when a software upgrade proposal is accepted.
As a result, the flow of future upgrades should look like this:
- A software upgrade proposal is submitted with a specified upgrade height/time of X
- The proposal is accepted
- At height/time X the chain halts in anticipation for installation of new binaries
- Node runners replace the old binaries with the new ones and re-start their nodes
- Nodes that are set up using cosmovisor as a process manager can install the new binaries using cosmovisor anytime before height/time X. Then when it’s height/time X, they can sit back while cosmovisor replaces the binary and restarts the node
Some implications of this new upgrade method:
- No need to export the state and import it with the new binaries – faster upgrades
- No tax event for SCRT stakers – staking rewards will remain untouched (because there’s no export & import of state)
- The chain-id will remain secret-4
- The block height will not reset
User experience for validators is important too!
Backport crypto APIs from CosmWasm v1
While we are working on adding CosmWasm v1 to Secret Network, a quick step we can take to improve the performance of v0.10 contracts is to backport the crypto APIs from CosmWasm v1. These new APIs delegate heavy cryptographic computations outside the WASM engine, which is good for performance for a few reasons:
- Secret Network’s WASM engine is slower compared to vanilla CosmWasm. Back in 2020 the vanilla CosmWasm’s WASM engine didn’t play nice with SGX, so we opted to go with a slower yet SGX-compatible WASM engine.
- The crypto crates that are used today for doing these computations are usually big which in turn makes the WASM binaries big too.
- The WASM engine meters every opcode that is executed. This overhead is compounded with heavy computations.
The new functions that will be available to contracts:
deps.api.ed25519_verify()
deps.api.ed25519_batch_verify()
deps.api.secp256k1_verify()
deps.api.secp256k1_recover_pubkey()
deps.api.ed25519_sign()
deps.api.secp256k1_sign()
Note that ed25519_sign() & secp256k1_sign() are unique to Secret Network, as Secret Contracts allow generation and usage of private cryptographic keys!
The addition of these new API functions means faster execution of permits for future SNIP20 tokens, SNIP721 tokens & cross-chain airdrops. It also means massive performance gains for apps like Shinobi Protocol that heavily utilize ed25519 signature verification and secp256k1 message signing.
Our intention is to upgrade the chain with these new APIs sometime in April.
Replace our WASM engine
When initially developing secret contracts we had to make a choice on which WebAssembly (WASM) engine to use. We decided to go with wasmi – a slower, older engine that was made for SGX (the core technology that enables privacy) – rather than undertaking long development to adapt a more performant engine. It was a good decision at the time, as it allowed us to launch faster, though now we will have to catch up the tech to match adoption.
(Source: https://medium.com/wasmer/benchmarking-webassembly-runtimes-18497ce0d76e)
Our intention is to replace our current WASM engine, Wasmi, with the vanilla CosmWasm’s WASM engine, Wasmer. This is going to be a difficult task, since as you can imagine the engine that runs our smart contracts is tightly integrated with many system components. Still, we’re confident that we will be able to make this transition, and we’re aiming for a Q4 release for this feature.
What’s Next?
These network improvements are all part of our vision for Shockwave, announced in early January of 2022. With more products and tools being built, more creators launching on Secret Network, and more supporters and partners than ever helping to ensure our global growth, the future has never been brighter for Web3 privacy.
Secret Network
If you’re a developer, you can also read through our latest announcements re: the new $225M Secret Ecosystem Fund, which contain details on how to connect with the SCRT Labs team and the rest of our developer ecosystem. Additionally, check out Secret Network grants and get started building today!
If you’re anyone who is passionate about ensuring Web3 users have the data privacy protections they need and deserve, become a Secret Agent. It is our mission to make sure the decentralized web we’re building is one that truly empowers – and one that is accessible to all.
Thank you for your support – the Shockwave is just starting! 🤫
Onwards and upwards!
