It has been quite a year for the Secret Ecosystem! Our community launched an independent blockchain in February, and since then, Enigma’s dev team worked really hard to build a compute module in parallel with Confio building CosmWasm. Now, Secret Contracts are going live (on September 15), following a network upgrade driven by on-chain governance.
Secret Network
We are deeply grateful to Ethan Frey and Simon Warta for all their cooperation and support! Thus far, it has been a great experience working together, and we will continue building momentum as we pursue our common goals in the Cosmos ecosystem and beyond!
The Cosmos SDK is a solid platform for building and customizing high-quality blockchains using Tendermint for consensus. As we initially explored the Cosmos ecosystem, we found a particularly exciting project, CosmWasm, which enables running Wasm modules as contracts on a Cosmos blockchain. We could see this project was gaining traction, and we decided to base our solution on it and maintain compatibility with it as much as we can.
Having participated in the Cosmos ecosystem for the better part of a year, we are thrilled to share this detailed post about our collaboration with the CosmWasm developers, who are building and testing a smart contract module, named x/wasm, for the Cosmos SDK. Building in tandem with CosmWasm, Enigma introduced the x/compute module for privacy-preserving “Secret Contracts,” written in the Rust programming language, based on CosmWasm, running across the network of validator nodes equipped with Trusted Execution Environments (TEEs).
As a first step in becoming familiar with the Cosmwasm project, we launched our testnet in March with smart contracts running CosmWasm! We have a ton of respect for the CosmWasm team, and we understand collaboration is the best strategy for long-term development projects. Because of this, we are contributing to upstream development and helping as much as possible.
If you are not familiar with CosmWasm, you can learn more about their contributions to the Cosmos here:
CosmWasm’s implementation of the contract engine (virtual machine) is written in the Rust programming language, under a layer of Go integrated with the Cosmos SDK. This was great for Secret Network because Enigma’s former Discovery testnet used Rust compiled to Wasm. As an extension of this approach, even the contracts themselves will be written in Rust (at first), although the contract runtime does not come without limitations. For example, you don’t have access to system libraries, networking or the OS. Plus, you can’t upload contracts that contain floating point operations in their compiled binary.
Furthermore, as our solution required the contract engine to run inside a secure enclave, we knew that we would have to make adjustments and modifications to their implementation. Most importantly, the Wasm engine they chose, Wasmer, is not suitable for running inside TEEs. This meant that we had to reimplement the execution environment that runs the Wasm modules using a different Wasm engine. We found Wasmi to be the best appropriate option for us. Read more about that design choice:
guy
Approach
- Run a CosmWasm contract in wasmi (same imports/exports)
- Run wasmi inside a TEE
- Expose high-level API compatible with cosmwasm-vm and swap them
SecretWasm Architectural Design
One level below the node and Cosmos SDK module, are FFI bindings into the rust library, compiled into a dynamic-link library (dll) and linked into the go binary. This calls into `cosmwasm_vm`, which manages the wasm code. Here you may see the high-level operations that can be performed on the Wasm modules. For Secret Network, the Wasm runtime is in the TEE, so we forward the operations that the node requests from `cosmwasm-vm` into the SGX enclave, where the private computations finally execute.
This design can be summarized by this comparison of the original components of the CosmWasm implementation side-by-side with our custom components:
Here is the pull request in which Reuven started to implement for his replacement of wasmer with a wasmi runtime running inside a secure enclave. Overall, this entailed modification of x/wasmd, along with go-cosmwasm and cosmwasm-vm (renamed to cosmwasm-sgx-vm), to interface with our solution running inside of Intel Software Guard Extensions (SGX).
This Week in Rust ~ Quote of the Week, July 16, 2019: “Rust is 5 languages stacked on top of each other, except that instead of ending up like 5 children under a trenchcoat, they end up like the power rangers.” — @reuvenpo
Core contributor of Secret Network, Reuven Podmazo is an expert Rust programmer on Enigma’s dev team. After trying to integrate CosmWasm into Secret Network, learning about its implementation details, and becoming familiar with its features and issues, he began working with the CosmWasm team. While the entire team had a part in this collaboration, Reuven worked most directly with the Confio team. At the same time, the other team members worked on infrastructure, tooling, and the custom details of encryption and decryption in our forks of the CosmWasm repositories. Without delving into specifics of memory management for distributed state machines, the following is a summary of Enigma’s collaboration with CosmWasm.
Major Pull Requests
- added interpretation and unwrapping of wasmer_runtime_core::error::RuntimeError
- added correct handling of out-of-gas events in cosmwasm-vm
- adjusted gas consumption logic in case of errors in go-cosmwasm
- unlimited buffers in FFI
After writing the implementation groundwork, Reuven opened these sub-PRs:
Summary of Contributions
Collaborating with Simon Warta and Ethan Frey, the core developers of CosmWasm, we managed to patch a memory leak in the input arguments of functions exported from the go-cosmwasm dll and implement better panic handling and out of gas errors [1] [2].
There was an implicit limitation where large values could be written to storage, but when reading them back out, they were reported as missing if above 2000 bytes. Aiming to solve this problem, Simon opened an issue about memory allocation (closed here; length limit for the result of ExternalStorage.get was increased to 128 KiB). Reuven proposed a more robust allocation flow, which the CosmWasm team eventually implemented:
CosmWasm
The proposed solution required a certain cyclical reference to be created between one of the custom CosmWasm components, and one of the objects inside the Wasmer engine. The original implementation of the solution suffered from a dangling pointer bug that repeatedly crashed the process. The trick was to put the Wasmer component on the heap, so that pointers to it would not be invalidated before the object pointing to it was released. Reuven opened this pull request to demonstrate the idea.
CosmWasmAfter finalizing the 0.8 release, Reuven gained enough trust from the CosmWasm team and was given permission to make changes directly in cosmwasm repositories. He then declared, “I hereby solemnly swear I am up to no good I will not push to master, or merge any PRs.”
CosmWasm approved the changes, which started on the gas issue, and largely covered the case of sending errors back from Go to Rust and Rust to Wasm in the callbacks. In order to fully address the bugs, vulnerabilities, and missing features, Reuven had to spend a lot of time on restructuring their error handling strategy in many contexts. This issue goes together with this one to implement end-to-end error passing. All the work on error passing was required to cleanly pass the information about gas running out, which is implemented in the Cosmos SDK as a Go runtime panic, through the multiple layers of the system.
Context Switching
While studying the way gas is charged for actions in the system, Reuven noticed the case where an operation would use up more gas than was available for the computation was not handled at all, not in the Rust layers, but also not in the Go layers interfacing with the Cosmos SDK itself. Turns out when an operation in the SDK would run out of gas, the SDK would simply throw a panic, which it would then catch after unwinding through the x/wasm call stack. This caused some unique challenges, as our module was not implemented in Go. In general, panics across language boundaries are not defined. If a contract runs with a low gas limit and performs a large storage access, the memory space of that process would get corrupted because the Rust language does not expect that to happen. The solution was adding panic detectors at all the language boundaries, and passing along information about various kinds of events (similar to running out of gas, and general crashes) between the different runtimes.
Additionally, while discussing issues around this area in the code, we noticed that the contract may also overspend. There was no feedback between the contract runtime and the cosmos SDK during the contract execution, regarding the costs of the operations inside and outside the contract’s runtime. This would cause cases where the contract kept running for a while after using more resources than it was allocated, This led to a series of pull requests that allowed the contract runtime to track both the gas used in the SDK by accessing storage, for example, as well as the gas used to run the contract itself, and stop execution as soon as required.
The CosmWasm team said all the improvements Reuven made along the way really help the code quality in general, and they invited him to be a core contributor!
How do you do gas metering with wasmi?
Metering is really just a question of how much and if it actually matters. While wasmer has built-in gas metering (for some configurations, wasmi has no such concept). This meant that we had to find a solution for injecting gas metering into the users’ Wasm modules (a.k.a. Secret Contracts) inside the enclave, before executing them. We inject gas metering into the wasm code using pwasm-utils:
CosmJS / SecretJS
The CosmWasm team is working on CosmJS, “the Swiss Army knife to power JavaScript-based clients for Cosmos.” In collaboration with Confio, Tom Langer and Assaf Morami led a collaboration to explore how CosmJS might be adapted for Secret Network. We only encrypt messages between users and enclaves, and the contracts receive messages in plaintext, so we haven’t changed any contract interfaces. Data enters the enclave before the contract, so we need encryption capability in the client to send encrypted blobs instead of JSON from the client to a node. Enigma developed SecretJS for that purpose, allowing users to interact with a REST API exposed by Secret Nodes.
Further Collaboration
We are thrilled to be collaborating with Confio, the team behind CosmWasm, and building our compute module and SecretJS in parallel with their groundbreaking wasm module for the Cosmos ecosystem. Now, Secret Network is able to run CosmWasm contracts inside secure enclaves across a distributed network of Secret Nodes. In other words, Secret Network can execute contracts in a parallel universe. Using Secret Contracts, users can encrypt input client-side (on their own device) and send it to the network. The encrypted inputs are only decrypted in the TEE to run the computation and update the contract state. All of this happens without any network participant observing encrypted inputs – full privacy!
We believe that Secret Contracts provide crucial features powering open economies throughout Cosmos, the internet of blockchains. Secret Network is designed for interoperability with Ethereum, Cosmos Hub, and the recently announced Confio DeFi network powered by CosmWasm. More and more individuals are beginning to understand how the programmable privacy features of Secret Tokens provide an important differentiator of the Cosmos universe.
Learn More
Create your first Secret Contract, and start building your own Secret Apps!
Secret Network
Please let us know any ideas or questions. All are welcome to join our dev committee on Discord. We have weekly meetings to discuss various projects related to improving documentation and overall experience for developers. You can fill out this form to get involved, and see our project board for suggested tasks!
To Infinities and Beyond 🛸🛸🛸
To discuss Secret Network and Secret Apps, visit our community channels:
Website | Forum | Twitter | Discord | Telegram
