OpenZeppelin is a well-known and trusted company in the blockchain world. It provides secure open-source smart contracts and libraries, which accelerate the blockchain development worldwide. For more than two years, developers from all over the world have been using OpenZeppelin’s v4 contracts to build applications. The v4 contracts were maintained, well-documented, and audited over a long period. However, On the 5th of October, a new journey began with the release the v5 of OpenZeppelin contracts. In this article, we will explain - with an example - why it is not safe to upgrade existing deployments from OpenZeppelin Contract v4 to v5.

TLDR

The v5 of the contracts includes significant changes and improvements. Some contracts and libraries have been restructured, while others have been removed. However, the most crucial reason for the incompatibility is that the v5 contracts of OpenZeppelin integrate a significant change to the storage layer. The contracts in v5 adopt the namespaced storage layout, which is defined in ERC-7201. This change implies that the new logic of v5 contracts will query existing data and write new data in incorrect storage slots, making an upgrade from v4 to v5 not safe. Developers have several options to update an existing contract, such as reinitializing the contract, deploying it as a new proxy, or choosing to continue with updated v4 contracts. Each option has its own unique challenges and requirements.

Prerequisites

We consider the knowledge of upgradeable smart contracts and rudimentary knowledge of the Ethereum Virtual Machine (EVM) storage as a requirement for understanding this article. The following tutorials give a good overview about upgradability of smart contracts:

• https://www.youtube.com/watch?v=JgSj7IiE4jA
• https://www.youtube.com/watch?v=kWUDTZhxKZI
• https://www.youtube.com/watch?v=YJZV9uiDbJI

For the EVM storage layer we recommend the following articles:

• https://steveng.medium.com/ethereum-virtual-machine-storage-layout-beb9a72a07e9
• https://www.adrianhetman.com/unboxing-evm-storage/
• https://medium.com/@Knownsec_Blockchain_Lab/knowsec-blockchain-lab-depth-understanding-of-evm-storage-mechanism-and-security-issues-50509acea373

Comparing how state variables are stored in OpenZeppelin upgradeable contracts in v4 versus v5

The Ethereum Virtual Machine (EVM) stores state variables in a key-value store with 256-bit keys and 256-bit values. These state variables are stored in a compact, continuous manner. Therefore, the order of the state variables in a smart contract determines their position in the storage.

OpenZeppelin v4 upgradeable contracts primarily use the default approach for storage management. The storage slots for state variables are determined by their sequential order in the contract. Therefore, base contracts often include a ‘storage buffer’ — defined as uint256[49] __gap; — to reserve storage slots. This will allow future versions of that contract to use up those slots without affecting the storage layout of child contracts. This approach of storage handling complicates contract management and upgradeability by increasing the risk of mistakes and storage conflicts, where new upgrades may unintentionally overwrite existing storage slots.

To simplify upgradability and minimize storage conflicts, OpenZeppelin v5 adopted the namespace storage layout, which has been proposed officially as ERC-7201 on June 20, 2023. The ERC-7201 recommends using pseudorandom locations derived from a namespace as the basis for new storage trees in base contracts. The root slot for storage is calculated using a specific formula. This approach does not require changes on the EVM level and can be implemented by grouping state variables within a struct and using assembly language to access specific storage slots directly, as shown in the example:

In the example smart contract x and y are state variables. However, they are not stored on the first and second slots in the storage as default. They will be stored in the storage tree under the given root MAIN_STORAGE_LOCATION = 0x183a…b500, which is calculated using the suggested formula in ERC-7201.

When a v4 upgradeable contract is deployed on the network, the state variables of the contract are initialized, stored, and managed using the default storage pattern based on their order in the contracts. However, if the implementation contract is upgraded using v5 contracts, a critical issue occurs. The new v5 contracts are designed to interact with storage in a different way, using the namespaced storage layout introduced in ERC-7201. As a result, these updated contracts will not recognize or access the original storage slots used by the old contract. This mismatch in storage access will lead to dangerous and incorrect states where variables may not reflect the intended values, and functions may behave unpredictably, leading to loss of funds or compromised security.

Unsafe Upgradability Example

The following ExampleV4 contract uses OpenZeppelin v4 and imports the following contracts Initializable, OwnableUpgradeable, and UUPSUpgradeable. It sets the value of exampleStateVariable, the address of the owner, the address of implementation contract, and the status of the contract to Initialized during the initialization process when it is deployed as proxy.

To check the storage keys of the state variables, we can deploy it as proxy using Remix IDE and then debug it using the Debugger.

As shown in the picture, slot 0x000…000 (slot 0) has the value 0x01 marking the contract as initialized. At Slot 0x000….033 (Slot 51) the, address of the owner of the contract is stored. At 0x000…0c9 (Slot 201) the value 7 of the exampleStateVariable is stored (Note: The variables are not sorted at slots 0, 1, 2 due to the buffer gaps between the contracts). At Slot 0x36089…82bbc the address of the implementation contract stored (Note: The slot of the implementation contract address was already hardcoded in v4 in a similar way to the namespaced storage layout to prevent conflicts).

For demonstration purposes, we will update the implementation contract to ExampleV5 using OpenZeppelin v5 contracts, without introducing new logic. To execute the upgrade, we must import the new v5 contracts, make minor modifications to the original contract to align with the new structure, and then deploy it. Then, we invoke the upgradeTo function on the proxy contract, providing the address of the new implementation contract as an argument.

After upgrading the contract we can call the two state variables in the proxy contract to check their values.

The value of exampleStateVariable is now 1 and not 7 because the upgraded contract reads the value from the first storage slot, which was the same slot of the initialized status. Furthermore, the owner is the zero address 0x000…000 because the new implemented contract following v5 of OpenZeppelin looks for the address of the owner at another storage slot. Inspecting the v5 contract, we will find that the owner value is read from slot 0x9016…9300, which is empty.

The same issue applies to the Initialization status of the contract. It is read from slot 0xf0c5…6a00 and not from 0x000…000 as it was in v4 contracts

This means our very simple updated contract has incorrect values, is not initialized, and has the zero address as Owner.

At this point, anyone would be able to initialize it again giving the address of owner to control the contract. Even if the original owner still managed to initialize the contract before anyone else the contract will stay in an unsafe state. To explain this we will re-initialize and debug the storage of the upgraded contract using Remix.

As displayed on the above storage snapshot, the original (v4) storage slots at 0x000…0c9, 0x000…033 are not empty. Therefore, new state variables added to the contract could unintentionally use these slots, creating invalid values.

Going Forward with Existing Contracts

For developers, finding the best way to update existing contracts is crucial. There are several options for updating deployed v4 upgradable contracts, some of these options are:

• Reinitializing the Contract: This process involves clearing the storage and migrating the data within the contract. While this approach might be feasible for simple contracts, for complex contracts with dynamic data types such as arrays, strings, and mappings, the storage becomes overly complicated, making migration within the contract impractical, if not impossible.
• Deploying as a Proxy: Similar to the Social Migration approach for non-upgradable contracts, this method involves deploying the new contract as a new proxy with a new address, followed by migrating the data and users to this new contract. Depending on the contract type and existing data, this process can become complicated. Additionally, it requires users’ involvement.
• Updating using v4 Contracts: Likely the most advisable approach is to continue using v4 contracts. This method eliminates the need for data migration or storage restructuring. Developers need to maintain their existing OpenZeppelin v4 contracts, extending or updating only the necessary components for their specific needs when required.

Summary

In Ethereum, upgradeable contracts are complicated due to potential storage conflicts. The introduction of a new storage layout aims to make the upgrade process easier and cleaner. However, old upgradable v4 contracts will face issues with the new layout, as their state variables could be dislocated, leading to an invalid contract state. Therefore, a native upgrade from v4 contracts to v5 is considered as not safe and is not recommended, requiring careful selection of strategies for updating v4 Upgradeable Contracts.

Why Upgrading OpenZeppelin Smart Contracts from Version 4 to Version 5 is Unsafe was originally published in 51nodes on Medium, where people are continuing the conversation by highlighting and responding to this story.

Due to its immutability and censorship resistance, the Blockchain seems to be a suitable place to manage identities and credentials. However, querying the data on the blockchain in a trusted and efficient way is one of the most challenging issues every Web3 developer has to face while developing a DApp. Therefore, this article describes a simple approach to revoke verifiable credentials and a decentralized and efficient way to index and query those revoked credentials using the Graph protocol.

We consider the knowledge of Self-Sovereign Identity (SSI) and rudimentary knowledge of the Ethr DID method as a requirement for understanding this article.

Ethr DID and Credentials

The Ethr DID method uses Ethereum addresses as identifiers. By default, the DID of the issuer/subject does not require an on-chain registration and it is constructed from the Ethereum address and controlled by its private key. This feature minimizes the effort of integrating Identities in many on-chain use cases.

When an off-chain verifiable credential is created, the issuer entity must set the “issuer” property to its Ethr DID and the “credentialSubject.id” property to the Ethr DID of the subject. Example:

"verifiableCredential": {
    "@context": [..],
    "issuer": {
      "id": "did:ethr:0xf1232f840f3ad7d23fcdaa84d6c66dac24efb198"
    },
    "issuanceDate": "2022-07-2T12:00:00.000Z",
    "credentialSubject": {
      "id": "did:ethr:0x435df3eda57154cf8cf7926079881f2912f54db4",
      "degree": "Bachelor Of Arts"
    },
    "proof": { ... }
  }

After that as usual the issued credential will be used by the subject to generate a verifiable presentation. The verifier will verify the presentation and accept or reject it depending on its business logic.

Credential Revocation

Verifiable Credentials are similar to certificates, some of them have an expiration date and some don’t. However, in both cases, the issuer and the subject must be able to revoke them. To enable this critical requirement, we need a tamper-proof revocation data registry that is always available and cannot be censored by any of the mentioned parties. This can be achieved using the Blockchain. Such a registry can be implemented as a simple smart contract that stores a reference of the credential and a reference of the entity who revoked it.

The revocation process:

• Get the hash value of the issued verifiable credential using the keccak256 hash function after converting the credentials to bytes. This value will be used as the on-chain reference to the credential
• Send a transaction to the revocation smart contract to call the revoke method revoke(bytes32 digest)giving the extracted hash value as input
• The entity that revoked the credential is the address of the invoker and it can be matched with the issuer or subject of the credential based on their DIDs

The additional important key that will help use indexing those revocations is that when a credential is revoked, the revocation smart contract triggers an event that includes the keccak256 hash of that credential and the address of the entity that revoked it.

In the Github Repository, we prepared a simple project that:

• Deploys a revocation smart contract that stores references to the credential and the entity that revoked it.
• Uses the did-jwt-vc and ethr-did libraries to issue 3 off-chain verifiable credentials by 3 different issuers and revoke them on-chain

Please check the ./ssi-contracts folder and follow the instructions in the README file. After following the steps you should have the address of the revocation registry smart contract deployed on the Goerli network. The 3 revoked transactions can be found on Etherscan using the address of the deployed contract.

Why do we need to index the revoked Credentials?

In a verification scenario, the verifier gets a verifiable presentation that includes 1 to n verifiable credentials from the subject. In addition to the normal validation process of the credentials, the verifier must also check the on-chain status of the credential at any time easily and efficiently. To do that the verifier could request the status of each credential by calling the methodrevoked(address issuer, bytes32 digest) in the revocation registry contract. However, this approach of getting the information is limited, costly, and not efficient for many reasons, some of which are:

• It requires knowledge of the Blockchain and Web3 technology
• It is slow and limited because of the latency, number of requests, node capability
• It requires running a blockchain node or relaying on a node provider
• Duplicate requests for the same credentials are sent over and over again
• It requires the verifier to be online all the time

And this is when indexing comes into play to provide an easy, fast and simple way to iterate and query data. Indexing is especially needed in the Blockchain area where its underlying design makes it difficult to search for relevant data.

But How can we index the Blockchain?

One approach is iterating all transactions and checking if the transaction interacts with our revocation contract. This method is very slow and resource-intensive. The other approach which is used by most developers is to subscribe to events that are triggered by the target contract. This method is much easier, supported by tools and libraries like web3js, and is more resource-saving. Now we come to the second important question of who should do the indexing? There are some possible answers:

1. The verifier himself. But it is not a good solution because each verifier needs to build up and maintain its own logic and infrastructure. Furthermore, the server of the verifier that is responsible for the indexing is a Single-point-of-failure

2. External ingestion service that indexes the data on behalf of the verifier. Such a service could be very useful. However, it creates one of the most critical issues which is the lack of TRUST. Verifiers need to trust the ingestion service and relying on it eliminates the advantage of using the Blockchain

3. Decentralized indexing protocol like The Graph protocol, which enables decentralized applications to query complex data from the blockchain without having to develop and operate proprietary indexing servers or use a centralized service provider. This is the preferred solution in most use cases because:

• It is distributed and decentralized. Therefore it removes the Single-point-of-failure and the missing trust issue
• It does not require advanced Web3 knowledge. Developers use only GraphQL requests to query the data from already implemented and indexed Subgraphs
• The technology behind it has been designed specifically for the Blockchain. Therefore, it is quick and easy to query and filter
• Cost-efficient because indexers are competing with each other to provide the best service for the lowest price
• Can be used by many verifiers around the world with minimal latency because the nodes are distributed and public

The creators of the Graph protocol recognized the indexing and querying problem at the end of 2017 and started working to solve it by defining and implementing a decentralized indexing protocol. The introduction page of the Graph protocol offers detailed information about the protocol and you can also join the free and very useful courses provided by The Graph Academy.

Develop and Deploy a Revocation Subgraph

A Subgraph defines which data should be indexed from a blockchain, and how it should be stored. In our simple example, the core data the verifier requires is the hash of the credential and who revoked it. To make things easier for verifiers we could also add the timestamp and block number.

To keep the article simple we will deploy the subgraph on a hosted service. Hosted service can be seen today as a good and easy place to start with the development of Subgraphs because it is free to use and reduces the deployment and testing complexity of Subgraphs. Developers can at any time migrate their Subgraphs to the decentralized network.

The following steps to develop and deploy the Subgraph are described in detail with all the related files in this repository :

• First, initialize a new sample project using the graph cli
• Then modify some of the configurations like adding a startBlock and checking the address of the contract
• After that comes the main part of the development which is to define the entities and implement the handle method that will be executed when an event is emitted by a smart contract
• In the end, build the project and deploy it on the hosted service using the graph cli and an access token

When the Subgraph is deployed and synchronized we can query the indexed data using simple GraphQL requests.

Example: open https://api.thegraph.com/subgraphs/name/<github-name>/credential-revocation-graph in browser and run the following query to get the hashes of all credentials that are revoked by Issuer-A from ./ssi-contracts

The results should contain only a single revoked credential unless you modified the scripts inside ./ssi-contract or used an already existing contract for this query. You can also compare the results with the logged data of the executed script in./ssi-contract

Summary

In this article, we described one of many approaches on how to revoke issued credentials. This outlined approach benefits from the Ethr DID characteristic and it depends on the Blockchain as an immutable registry accessible for all parties. We also talked about the likely best way to index and query revoked credentials without relying on centralized services. And at the end, we showed how to develop a simple Subgraph that could run on the Graph protocol decentralized indexing network and index the required data.

51nodes GmbH is a provider of crypto-economy solutions based in Stuttgart, Germany.

51nodes supports companies and other organizations in realizing their Blockchain projects. 51nodes offers technical consulting and implementation with a focus on smart contracts, decentralized apps (DApps), integration of blockchain with industry applications, and tokenization of assets.

Indexing and Querying Revoked Verifiable Credentials was originally published in 51nodes on Medium, where people are continuing the conversation by highlighting and responding to this story.

This article serves as an overview of current Zero-Knowledge Proof (ZKP) implementations in the crypto space and discusses what to expect from ZKP as an exciting cryptographic method in the upcoming months and years. Specifically, we provide a snapshot of some of the most interesting projects and how ZKP improves important properties of today’s blockchains’ infrastructure, tooling, and applications. In an earlier article on ZKPs written at the end of 2020, we have taken a closer look at the core principles of Zero-Knowledge Proofs, their usage in Verifiable Credentials, and the state of available implementations back then. Now, in 2022 ZKPs are increasingly used as a solution for some of the blockchains’ scalability issues. We have also started to see improvements in its usage for privacy enhancements, though the topic of privacy seems to be in an earlier stage than scalability.

Primer on ZKPs

Essentially, a ZKP is all about trust between multiple parties. In simpler terms, this means one party is able to prove to a second party that it holds a certain piece of information without disclosing the relevant information to the receiving party. For instance, proving to the cashier that you are older than 21 without revealing your actual age. To get a better idea of ZKPs core principles, please refer to this ELI5 video.

ZKP Infrastructure and Tooling

ZKP solutions discussed in this section focus on improving whole blockchain networks or on providing specialized tools for those. For blockchain networks, ZKPs can help with keeping the ledger size consistent or with providing an overall more efficient approach to Distributed Ledger Technology (DLT). SDKs and tools can in return be used for writing ZKP logic and for compiling this logic into ZKP circuits.

Mina Protocol

Mina Protocol is a layer 1 protocol that takes blockchain infrastructure to a new level by basing blockchain interactions on ZKPs. While other scaling solutions aim to decrease transaction size and cost, Mina has a more holistic approach. Mina is a succinct blockchain, with a constant size of about 22KB.

With Mina, ZKPs are directly integrated into smart contracts and so-called zkApps that can be built using those ZKP-enabled smart contracts. zkApps manage their state off-chain (mostly synchronously) and then store a proof of their state on-chain once the computation is finished. Thus, Mina and zkApps allow the building of highly efficient applications. Fortunately, zkApps are built using Typescript not requiring learning a special-purpose programming language like Solidity with Ethereum.

StarkNet/StarkEx

StarkNet is a layer 2 network for Ethereum using the “StarkEx Protocol” for providing faster and more cost-effective transactions and increased privacy. Instead of directly sending transactions to layer 1 Ethereum, StarkEx uses zkRollups for creating proofs for the transaction on layer 2 and then stores those proofs in batches on layer 1.

The StarkEx Protocol is defined in five different components, i.e., Application, StarkEx Service, SHARP, Stark Verifier, and StarkEx Contracts of which some are acting on-chain and others off-chain.

The high-level architecture above shows how users will use an Ethereum-based application that sends transactions to the StarkEx Service. This service uses StarkWare’s Cairo language for creating zero-knowledge programs that prove the application-relevant state. After a proof has been computed off-chain it is stored on-chain using StarkEx Contracts. The Stark Verifier can then be used to verify the proofs by either checking the state of the StarkEx Contract or by using the SHARP service in case one needs to prove the states of multiple different applications at once.

DuskNet

DuskNet is a business-oriented ZKP-based blockchain with fast transactions for building privacy-preserving smart contracts and confidential tokens for the financial sector that respect confidentiality agreements. Driving factors for DuskNet are mostly privacy concerns and compliance with GDPR rules and the essential need of companies to keep information secure.

To provide these properties in a blockchain network, DuskNet uses PLONKs that are in general faster than bulletproofs but in need of a trusted setup. For more information on PLONKs, I can recommend this blog from Vitalik.

One interesting use case of DuskNet is the XSC Security Token Standard, which provides permission management for an asset's lifecycle. The ledger records all transactions, but the access rights of token holders are not lost once a token holder fails on a transaction or the access keys are lost. This property is an important requirement of securities law.

Nightfall 3

EY’s Nightfall 3 provides a secure and privacy-preserving solution for transacting ERC-20 tokens as well as ERC-721 tokens at low cost. The successor of EY’s Nightfall called Nightfall 3 aims to improve the performance of such transactions even more while simplifying the developer experience. In addition, Nightfall 3 provides the ability to transfer ERC-1155 tokens.

Nightfall’s performance improvements are achieved by combining the existing ZK solution with optimistic rollups, creating a ZK-optimistic rollup hybrid. In this scheme, ZK transactions are grouped and then sent to the ledger as an optimistic rollup.

Aleo

Aleo promises to be the “first decentralized, open source platform to enable both private and programmable applications”. To further encourage the benefits Aleo uses the example of DEXs like Uniswap. A DEX on Aleo would keep the number of tokens you own disclosed or hide from where you got those tokens in the first place. Furthermore, all of this privacy is enabled without removing the ability to integrate with data from public blockchains.

Aleo comes with a variety of tools that should help new developers to build applications using Aleo.

Leo — Aleos programming language inspired by JavaScript, Rust, and Scala for writing ZKP applications.

Aleo Studio — Aleos IDE for writing applications with Leo.

Aleo Package Manager — For publishing the packages and applications written with Aleo Studio.

snarkOS — A decentralized OS for running Aleo. SnarkOS contains important logical components for writing ZKP applications and for proving states publicly.

ZKP Applications

Once you have decided on a certain infrastructure, may it be a ZKP-based blockchain network or a non-ZKP network you can build your own application. This is where ZKPs get interesting for the common user as the benefits are becoming more and more obvious at this level.

Hyperledger Aries

Aries describes itself as “a shared, reusable, interoperable tool kit designed for initiatives and solutions focused on creating, transmitting, and storing verifiable digital credentials.” We discussed Hyperledger Aries and their use of Hyperledger Indy’s “Anoncreds” in our earlier article. Interesting to note at this point is that we can see the first integrations of BBS+ signatures (signature-based ZKPs) that enable selective disclosure.

Combined with the W3Cs JSON-LD credentials, Hyperledger Aries could be one of the first movers in providing the ability to use ZKP-enabled W3C credentials in messaging, resulting in a more private and secure exchange of information between users or even devices.

Iden3

Iden3 is an open-source project aiming to provide a new and decentralized solution for digital identities. Based on ZKPs, Iden3 powers really neat use cases like anonymous logins and reputation proofs while not requiring users to disclose their actual identity. Although it is in the early stages of development, the use cases of an open-sourced and community-driven digital identity solution for end users seem manifold.

Iden3 developed its own Circom language and the Circom 2.0 compiler that allows creating ZK-Snarks (another type of ZKPs) on a more abstract and not mathematical level. This way, Circom allows for easy creation and integration of ZKPs. The figure below shows how Circom and SnarkJS can be used in combination.

Loopring

Loopring is an open-sourced decentralized exchange using an Automated-Market-Maker(AMM) and zkRollups to provide a fast and cheap method for token exchanges and payments. Loopring and similar applications could very well introduce a significant improvement for multiple fields in the crypto industry like Decentralized Finance (DeFi) and NFT trading.

One of Loopring’s latest announcements includes the hosting of GameStop’s upcoming NFT marketplace. This might introduce the advantages of ZKPs to a broader audience.

dydx

dydx is another decentralized exchange mostly focusing on perpetual trading. While dydx was initially launched on Ethereum’s layer 1 it introduced trading as a layer 2 solution for Ethereum in mid-2021 through zkRollups. The solution used for enabling zkRollups is StarkWare’s StarkEx (described above). Using the layer 2 solution instead of the layer 1 solution provides various improvements to trading like instant off-chain settlement. For a more detailed comparison of the benefits of the layer 2 approach please see this blog.

Sorare

Sorare uses ZKPs in an app for building your own soccer teams and trading your player cards. Like dydx, Sorare introduced its scaling solution of choice in the middle of 2021 by integrating the StarkEx Protocol. While there's nothing too special about Sorare, I think it is a great example of where the crypto (consumer) space is heading.

In a blog entry, Pierre Duperrin explained the reasoning behind Sorare’s decision to use Ethereum layer 2 scaling solutions that use ZKPs instead of other scaling solutions. The key argument for ZKPs seems to be the scalability trilemma. While there are other scaling solutions out there, ZKP-based solutions provide the advantage of not compromising on security or decentralization while increasing scalability.

Conclusion

ZKPs help to improve the scalability and privacy of a lot of existing solutions in the crypto space — for instance decentralized exchanges. Furthermore, ZKP can be a key enabler for the Web3.0 by improving privacy — not just on a transaction level but also on a network level. Users will be able to interact almost anonymously while still being able to provide the necessary information to participate in networks and applications.

Tools like Hyperledger Aries and networks like DuskNet will further enable businesses to utilize ZKP technology in privacy-preserving solutions. I can also think of scenarios where businesses will be able to integrate private solutions with public solutions. For example, using ZKP-enabled digital identities to participate in open markets privately. Personally, I am confident to say that crypto’s public sector (especially DeFi) will experience a lot of growth in 2022 and early 2023 through ZKP-based scalability improvements. This can already be seen by examples like Sorare or Loopring but will only accelerate once more applications integrate ZKPs.

51nodes GmbH is a provider of crypto-economy solutions based in Stuttgart, Germany.

51nodes supports companies and other organizations in realizing their Blockchain projects. 51nodes offers technical consulting and implementation with a focus on smart contracts, decentralized apps (DApps), integration of blockchain with industry applications, and tokenization of assets.

Improving Scalability and Privacy of Blockchains: 2022 Update on Zero-Knowledge Proofs was originally published in 51nodes on Medium, where people are continuing the conversation by highlighting and responding to this story.

Today, one doesn’t even have to know anything about Solidity programming or blockchains at all to create a product or service that is built on top of a blockchain. Nowadays there are services like Tatum and Moralis, providing almost everything you need to instantly start developing your product or service instead of managing all the “behind the scenes” things on your own.

This post explains how to get started with blockchain development using the platforms of Tatum and Moralis. I discuss pros and cons and what you can build with them today.

Introduction

The two main features of Moralis and Tatum are a Web API and an SDK for application development based on a public blockchain. With the Web API we can retrieve all the necessary data from a blockchain and with the SDK we can easily use code to interact with the blockchain to make a transaction or call a function of a smart contract. The blockchain data retrieved by the Web API can be accessed with the SDK as well. Without using such a service a developer must run his own full node to know the blockchains state in order to make a correct transaction based on the current state.

Moralis and Tatum work on multiple public blockchains like Ethereum, Binance Smart Chain, and Polygon. Thus, one can retrieve data from all these blockchains using a single Web API. While Moralis “only” covers 6 blockchains, the Tatum API covers around 40. Both APIs can be used to access the main network and a test network of a blockchain like for example Rinkeby or Ropsten for Ethereum.

Behind the scenes, services like Tatum and Moralis run a full node of every blockchain they support. Having access to all blockchains’ data, they are able to aggregate it so that one can access all users’ balances, token balances or NFTs in one place. The API can then be used within the respective SDK to help developers creating their product or service. The following picture shows the general architecture of Moralis.

Web API comparison of Moralis and Tatum

Using a Web API to get all necessary blockchain data saves developers a lot of time and money. Retrieving blockchain data via Tatum and Moralis is free of charge. Both APIs are able to:

• Get all Transactions an address has made.
• Get the current block of any chain.
• Get the native balance or balance of any ERC-20 token.
• Get all NFT information of a user or a contract.
• Get and write data from and to IPFS.

Both, Moralis and Tatum cover all “basic” API requests. Differences are listed in the following.

Requests specific to Moralis’ API

• Get all Events emitted at a specific smart contract.
• Get liquidity pair reserves for a Uniswap based exchange (DeFi).
• Resolve an unstoppable domain and return the address.

Good thing to know: The Moralis Web API differentiate between all Ethereum Virtual Machine (EVM) blockchains like Ethereum, Binance Smart Chain, Avalanche and Polygon and the new created Solana Web API for Solana. For Solana, one can “only” retrieve users’ current balance, transactions, NFTs and tokens at the moment.

Requests specific to Tatum’s API

• Generate a wallet and get the private key, the mnemonics and the address.
• Deploy a standard ERC-20, ERC-721 and ERC-1155 contract.
• Interact with virtual accounts and the off chain ledger — I will cover this later in more detail.

How to use the APIs

After you registered for Moralis you can see on the left side of the screen the Web3 API Button. Once clicked you can use the whole Web API within the browser (see figure below). The x-api-key and the whole curl request are shown when you execute a request. Using the API from outside the browser like for example with Postman or curl is also possible.

Using Tatum’s API works a bit different. After the registration you will be redirected to your dashboard. Here you can find your API keys — one for all main networks and one for all test networks.

To see the whole API you have to go back to the homepage to find the documentation of the API. Sadly, it is not possible to use the API within the browser. However, Tatum provides request code examples in 13 programming languages (see below).

Pricing

Both APIs have a very good free version that can be used immediately after registration. The following figures show the number of requests possible with the APIs, depending on the plan you may choose.

Web API conclusion

Most importantly first: both API’s from Moralis and Tatum are working great and as expected and I can recommend both to you. Moralis has a little bit less supported blockchains and features than Tatum but looks more modern on the other hand. With both API’s you have access for all “basic” requests and if you really need something advanced you should look at both API’s and compare if one of these API’s is already dealing with your problem. If so just choose it and try it out.

How to use Moralis’ blockchain SDK

Moralis describes itself as “the ultimate Web3 Development Platform” and is often said to be “the firebase of crypto”. Regarding help in blockchain software development, a JavaScript SDK is available. You can either use it as a plain npm module named “moralis” or you can use the “react-moralis” module to use the SDK within your React project. All the following paragraphs will focus on EVM blockchains but like already mentioned Moralis also support Solana.

Moralis server

Starting a new project with Moralis requires you to create a Moralis Server within your Dashboard.

Creating a Moralis server requires only to select a name, your region, main network or test network and which blockchains should be included and listened to. In the free version of Moralis, you can run 3 of these servers at the same time but you must revoke them manually after 7 days.

After you created your Moralis server, you can open either its details or its dashboard. Opening the details will show you your Server URL and your Application ID. These two keys have to be imported in your SDK to use it correctly. In the dashboard you can see all your additional databases Moralis is creating for you.

In the following paragraphs I will show use cases of these additional databases and cover some of the more advanced features you can do with your Moralis server that is helpful for Dapp development.

Authenticating users

The Moralis SDK provides a simple way to track all users of a Dapp. Either users can register themselves with a username and password or do it the web3 way by using Metamask. The following code example shows how to connect your Dapp with your Moralis server and how to authenticate users the web3 way.

const Moralis = require('moralis');

const serverUrl = "YOUR_SERVER_URL";
const appId = "YOUR_APP_ID";

Moralis.start({ serverUrl, appId });

Moralis.authenticate()

The authenticate function will open the user’s Metamask window and request a signature from the user by clicking the “Sign” button. Note that there is no gas fee involved in signing because it is not a transaction.

Now, here’s the cool part. Once a new user logged in, you can observe a new row added to your “User” table in the Moralis dashboard with the address of the user. Furthermore, the Moralis server creates additional tables like for example “EthTransactions”, where you can see all transactions made of all users on the blockchain networks you selected when creating the server.

Because your tables can get fast quite big, your whole database is capable of managing queries and filters in order to only get and show the desired data.

Listening to events

One feature only Moralis provides, is listening to events of a specific contract. To this respect, open your server details and go to the “sync” tab. Now either choose to “follow” a specific address or a specific event. If you choose to follow an event, you only have to paste its name with parameter, its ABI as JSON and the contract address. After creation the data will synchronize automatically and the result can be found within a newly generated table in your server. A more detailed tutorial can be found here.

Cloud-based queries

Complex database queries can be run on the Moralis server in the cloud. For example, if you have a table with movie ratings with 1–5 stars and you would like to know the average film rating, you normally would first get all ratings and then calculate the average. To directly get the answer, just deploy and call the query as a JavaScript function within your Moralis server. For more information about the correct syntax check out Moralis’ documentation here.

Getting data with Moralis’ SDK

Let’s take a closer look at the actual JavaScript SDK and how to retrieve blockchain data. Moralis differentiates between their “normal” API for all EVM blockchains and their Solana API. The following paragraphs refer to the EVM blockchain API.

To get some data, there are 4 entry points:

• With Moralis.Web3API.native.functionName(options) we can access the Web API data within our SDK. Click here for more information.
• With the useMoralisQuery(‘tablename’) function we can get access to all the data within a specific table of our Moralis server. With the React SDK it should also be possible to enable live updates whenever your table you reference to updates. However, this didn’t work for me while testing.
• With the Moralis.executeFunction(‘options’) function we can interact with a smart contract directly in order to get the state of a public variable or the result of a read only function. Click here for more information.
• With the Moralis.Cloud.run(‘function_name’) function we can execute our cloud functions and get the result back.

Using these methods, you can retrieve all blockchain data required for building your Dapp. Of course, all these possibilities can be combined within your Dapp. For example, many requests like getting the specific ERC-20 balance of an address can be resolved using any of these four possibilities.

Creating transactions with Moralis’ SDK

Knowing how to get blockchain data, let’s look at how to make a transaction with the Moralis SDK. The general approach of doing a transaction with the Moralis SDK is by conducting the actual transaction via Metamask. This way, the user and the developer don’t have to deal with any private keys, because Metamask handles them. Metamask provides the SDK with the information, which network the user is currently connected to, so we don’t have to deal with any blockchain network code. The SDK creates a correct transaction with the currently connected network on its own. Thus, the user only has to click on a button within the Dapp and Metamask to accept the transaction.

Making a transaction with the Moralis SDK is a one liner of code. Only the parameters of a transaction, e.g., the transaction amount, the receiver and the contract address have to be passed as JSON parameter to the Moralis.transfer(options) function. Note the different parameter settings in the code examples below to create a transaction for respectively sending a native coin, an ERC-20 token or an NFT. For more information click here.

1) Transfer native coin options
const options = 
{type: "native", 
 amount: Moralis.Units.ETH("0.5"), 
 receiver: "0x.."
}

2) Transfer ERC20 Token options
const options = 
 {type: "erc20",
 amount: Moralis.Units.Token("0.5", "18"),
 receiver: "0x..",
 contractAddress: "0x.."
}

3) Transfer ERC721 Token options
const options = 
{type: "erc721",
 receiver: "0x..",
 contractAddress: "0x..",
 tokenId: 1
}

// Call the Moralis.transfer function with options as parameter
const transaction = await Moralis.transfer(options);

// You can wait until its confirmation and do sth. with the result
const result = await transaction.wait();

Interacting with smart contracts

For interacting with a function of a smart contract, you have to create an options variable as JSON string that includes the contract address, the function name, the ABI of the function, and the parameters. Now call the Moralis.execute(options) function with the options string as parameter. That’s everything it takes to create a transaction with the Moralis SDK as shown in the following code example.

let options = {
 contractAddress: '0x...',
 functionName: 'FUNCTION_NAME',
 abi: [ FUNCTION_ABI_AS_JSON ],
 params: {
     parameter1: value1,
    }
};

const tx = await Moralis.executeFunction(options);
const result = await tx.wait();

Moralis conclusion

Moralis appears modern and helps building a Dapp. The Web API works great and using the SDK, it becomes a one liner to handle user authentication, getting data from any blockchain or creating valid transactions. Also, the documentation is very well written and detailed. They also have a very active Youtube Channel.

A limitation of Moralis is that it only deals with reading from and writing to a blockchain. Creating your own wallet or deploying a new smart contract are not possible. Therefore, scripting can get complicated because to send a transaction you have to deal with Metamask in the first place. But if you are only looking for a SDK that helps you build the JavaScript/backend part of your Dapp then choosing Moralis makes a good choice.

How to use Tatum’s blockchain SDK

Tatum describes itself as “The ultimate blockchain development platform” that offers a unified framework for 40+ blockchain protocols to reduce the complexity of blockchain development. While Moralis specialized in building Dapps, Tatum is a more general tool with a lot of methods needed to deal with a blockchain. Methods for creating and managing your own wallets and private keys, deploying smart contracts and interacting with them are included in the Tatum JavaScript SDK and their API.

Maybe because of the broad scope of Tatum, it feels a little bit unfinished. They feature a lot of services and endpoints instead of specializing on one specific. Thus, their Web API documentation is so big that it takes more than 2–3 seconds to load and render the whole page. Also, a lot of things are missing completely, and some other things are not properly explained yet in the documentation. However, Tatum told me that they are working on it already. So as well as in the Moralis section of this article, I will focus in the following paragraphs on EVM blockchains only, since the Tatum API is most developed for these kind of blockchains.

In the following paragraphs, I will show you the most important things you can put to use already today with Tatum.

Initializing Tatum’s SDK

Tatum’s SDK is as well written for JavaScript and can be downloaded as npm module with the name ‘@tatumio/tatum’. Once installed you can import it within your project and use one of the hundreds of functions Tatum provides. Calling any of these functions is a one liner. But compared to Moralis you don’t pass all the options as a single JSON string, but rather each parameter is passed individually.

Generating a wallet

By creating a wallet from scratch, we can script any transaction we want because we know the private key and can sign transactions on our own. To create a wallet with the Tatum SDK you need to import the generateWallet function and the Currency Enum. Once you call the function with the desired cryptocurrency, like for example Bitcoin or Ethereum, you will get your 24-mnemonic words and your xpub for your newly generated HD-wallet.

With these two variables, Tatum can create new addresses with the corresponding private keys for your new wallet using the generateAddressFromXPub and generatePrivateKeyFromMnemonic functions, which you have to import as well. The following code demonstrates the whole process.

1) const {generateWallet, generateAddressFromXPub, generatePrivateKeyFromMnemonic, Currency} = require("@tatumio/tatum");

2) const btcWallet = await generateWallet(Currency.BTC, false);

3) const btcAddress = generateAddressFromXPub(Currency.BTC, false,    "YOUR_XPUB", "HD_WALLET_INDEX");

4) const btcPrivateKey = await   generatePrivateKeyFromMnemonic(Currency.BTC, false, "YOUR_MNEMONICS", "HD_WALLET_INDEX")

In the first line of code, we import the SDK and all required functions. In the second line, we generate a Bitcoin wallet and get the mnemonic and the xpub. In the third line, we create a new address from the xpub and in the fourth line, we create the private key for it. Because we are dealing with a HD-Wallet you must also pass the index of it as a parameter.

Tatum’s Key Management System (KMS)

To create a transaction, we need our newly created private key to be passed as a parameter to each function. Because it is never a good idea to send your private key across the internet, Tatum provides two approaches: (1) You can sign your transaction locally — this way you must have your private key stored locally on your computer, or (2) you can use the Tatum Key Management System (KMS). KMS is a cloud solution by Tatum, allowing to import your private keys encrypted with a password of your choice to let Tatum sign transactions for you within their service. This way, the private key doesn’t have to be sent across the internet because Tatum signs your transaction and only sends the signature of the transaction. For more information, click here.

Accessing blockchain data

Currently, the Tatum SDK does not support yet to get all the blockchain data being available via the API. So to access blockchain data via Tatum, one should consult the Web API docs and make a HTTP request. This method can be used to build your own Dapp. I recommend to create a separate function for each request, in which you pass all necessary parameters and do the actual HTTP request within this function. This way, you create your own little framework for one-liner requests to get any necessary data from the blockchain.

Interacting with a smart contract

Interacting with a smart contract doesn’t work yet with the SDK at the time of writing. Instead, you must use the API as described here.

Transfering coins and tokens

At least the transfer functions for native coins, ERC-20 tokens, NFTs and ERC-1155 tokens exist already in Tatum’s SDK and are documented. The following code example shows how to create an ERC-20 token transaction. For more information, check out the documentation.

import {sendEthOrErc20Transaction, ethGetTransaction, Currency} from '@tatumio/tatum';

const transaction = await sendEthOrErc20Transaction({
chain: Currency.'CHAIN_TO_USE', 
fromPrivateKey: 'YOUR_PRIVATE_KEY',
contractAddress: 'CONTRACT_ADDRESS',
digits: DIGITS_OF_TOKEN,
amount: "AMOUNT_TO_SEND--0.1 for example",
to: "TO_ADDRESS"
});

const transaction = await ethGetTransaction(transaction.txId);

Deploying a standard smart contract

Tatum’s SDK allows deploying standard smart contracts for EVM-compatible blockchains. Thus, one can deploy a smart contract with custom parameters for token standards listed below. Detailed information on parameters to use regarding using one of the token standards are linked to the list:

• ERC-20
• ERC-721 (NFT)
• ERC-1155

Virtual accounts

Virtual accounts are more or less the equivalent of a Moralis server. It’s a separate private ledger maintained by Tatum to enable, e.g., setting up webhook notifications when at a specific address a new transaction is observed. While for Moralis you must paste your address to synchronize and the webhook URL endpoint in your server’s dashboard, in Tatum you must set it up using the SDK or API. Moralis’ dashboard has great usability but Moralis provides no API for specifying webhooks. Tatum on the other hand has no dashboard but provides an API and SDK for the setup.

Unfortunately, Tatum’s SDK did not work properly during my experiments and I had to use the API again to test the webhooks. The steps for setting up a virtual account in Tatum and to subscribe to blockchain events are:

• Create a new wallet and save your ‘xpub’
• Create a new virtual account with the ‘xpub’ as one of the parameters. Only a private key corresponding to this ‘xpub’ is allowed to do a transaction within this virtual account.
• Connect the address to sync with your virtual account
• Initialize a subscription that listens to incoming events and send the data to the endpoint of your choice.

Tatum conclusion

Tatum’s SDK is a not a complete product at the moment, given its claims. Tatum aims at offering much more features than Moralis for interacting with a blockchain, especially when starting from scratch. Tatum is also more flexible by supporting more blockchains and already provides more usable features that can be helpful for scripting: Generating a wallet, managing keys, or deploying a new smart contract works great. The documentation is quite okay, but I would like to see Tatum to improve on that in the future. For functions not present in the SDK yet, you can use Tatum’s API instead, which works fine. To conclude, Tatum already provides a decent service and given the assumption that they will improve their documentation and SDK, I can recommend using Tatum’s SDK and API for developing a Dapp.

Other blockchain SDKs and APIs worth considering

Besides Moralis and Tatum, several other services exist that provide a Web API and a SDK for Dapp development. But none of them are as advanced as Moralis and Tatum and can be used only for some parts of your Dapp probably.

• Idexo: It is another service, offering an SDK for blockchain development that feels like the very small brother of Tatum. Idexo provides only an SDK and is very limited regarding the number of blockchains and features. When trying to evaluate it, I had problems with registration and login. What Idexo is most known for is their Telegram bot that allow users to mint an NFT within Telegram.
• The Graph: This is a Web API, focussing on complex and automated queries on Ethereum and IPFS.
• NOWNodes: They offer a Web API that is comparable to Tatum. The service supports also 40+ blockchains and a lot of endpoints are supported for each blockchain.
• Infura: Infura provides some tools and endpoints to interact with the Ethereum blockchain and IPFS.
• Cosmos: If you even want to build a custom blockchain, then the Cosmos SDK might be worthwhile. Creating an own blockchain gets almost as simple as deploying a new token with it.

Conclusion

It’s astonishing to experience how easy blockchain development has become due to tools and services like Moralis and Tatum. I recommend to use Moralis for developing a Dapp and to use Tatum for close interaction with different blockchains. I expect more to come as these services are evolving quickly. On top of that many other services exist to keep an eye on.

51nodes GmbH based in Stuttgart is a provider of crypto economy solutions. 51nodes supports companies and other organizations in realizing their Blockchain projects. 51nodes offers technical consulting and implementation with a focus on smart contracts, decentralized apps (Dapps), integration of blockchain with industry applications, and tokenization of assets.

Thanks to Achim Klein and Jan-Paul Buchwald for their help in the course of developing this article.

Getting Started with Blockchain Development: Pros and Cons of Tatum vs. Moralis was originally published in 51nodes on Medium, where people are continuing the conversation by highlighting and responding to this story.

Vor diesem Hintergrund streben Selbstsouveräne Identitäten (SSI) die Verknüpfung der Nutzerfreundlichkeit einer Google-ID mit einem hohen Grad an Nutzerkontrolle und Privatsphäre an.

Was genau ist SSI?

SSI bilden die neueste Entwicklungsstufe digitaler Identitäten. Eine Identität meint alle Attribute einer Person, Organisation, oder Gegenstands, die dieses Subjekt definieren. Attribute beschreiben die Charakteristika eines Subjekts. Eine digitale Identität meint hier die digitale Repräsentation dieser Attribute. Je nach Verwendungskontext können verschiedene Teilidentitäten benötigt werden, die nur einen Teil der Attribute umfassen.

Ein Credential ist ein fälschungssicherer und überprüfbarer Nachweis von Attributen eines Subjekts. Credentials werden in einem persönlichen digitalen Wallet gehalten. Das Subjekt ist der Holder der Nachweise, nachdem es diese von einem Issuer beschafft hat. Ein Issuer stellt Credentials nach einem Schema aus, das eine übergreifende Verwendung und allgemeingültige Interpretation erlaubt. Ein Verifier fordert, z.B. für den Zugriff auf einen Dienst, einen Nachweis einer Berechtigung (ein Credential) von dem zugreifenden Subjekt. Über ein Verifiable Data Registry werden dazu Identifikatoren ausgetauscht, die auf Subjekte, deren Credentials und deren Issuer verweisen können. Das Verifiable Data Registry enthält keine subjektbezogenen Daten.

Welchen Mehrwert bringt SSI?

SSI überwindet das inkonsistente und — für Nutzer als auch Diensteanbieter — aufwändige Identitätsmanagement in isolierten Datensilos und gleichzeitig die Abhängigkeit von bisherigen Single-Sign-On-Identitätsmanagern wie Google und Twitter, die die Anmeldevorgänge der Nutzer kontrollieren, auswerten und verwerten. Mit SSI haben Subjekte wie Personen oder Maschinen eine einzige Identitätsverwaltung (zumindest für einen gewissen Namensraum). Die einer Identität zugeordneten Berechtigungsnachweise müssen nur einmal eingeholt werden und können dann nach Bedarf passgenau für beliebig viele Anmelde- und Nachweisprozesse eingesetzt werden. Analog zur Plastikkärtchen-Identität können die Nachweise geprüft werden, ohne den Herausgeber zu involvieren.

Für Diensteanbieter vereinfachen sich Onboardingprozesse, das sogenannte Know-your-Customer und die Compliance zu den damit verbundenen Regularien. Zudem sind die bereitgestellten Identitätsinformationen aktuell. Durch die Vereinfachung der Onboardingprozesse werden diese auch mit einer höheren Wahrscheinlichkeit als bislang von den Nutzern abgeschlossen.

Nutzer von Diensten (beliebige Subjekte, auch Maschinen) können ihre Identität in Plug-and-Play-Manier einsetzen und ihre Identität einfach zwischen verschiedenen Diensten portieren. Das Verwalten und Aktuellhalten von dutzenden Nutzerkonten und das Merken der zugehörigen Passwörter entfällt, da nur noch ein Zugang zum persönlichen Wallet erforderlich ist. Konsequenterweise reduzieren sich auch IT-Sicherheitsprobleme, die durch die Verwendung des gleichen Passworts für verschiedene Nutzerkonten entstehen.

Welche Herausforderungen gibt es?

Einschränkend kann vermerkt werden, dass die Vorzüge und Nutzbarkeit einer selbstsouveränen Identität sich zunächst auf diejenigen Dienstanbieter beziehen, die auf einer Verifiable Data Registry präsent sind — ähnlich den mobilen Anwendungen auf den App Stores. In Analogie dazu ist ein SSI-Netzwerk umso wertvoller für seine Nutzer, je mehr Dienste darüber nutzbar sind. Schließlich ist hier ein Wachstums- und Konsolidierungsprozess zu erwarten, der über die Durchsetzung von bestimmten Angeboten entscheidet.

Ein kritischer Gedanke ist, dass mit dem Verifiable Data Registry eine ähnliche Abhängigkeit der Identitäts-Subjekte wie bislang zu den Single-Sign-On-Anbietern aus den Sozialen Medien entstünde. Dieser Gedanke führt jedoch in die Irre, denn das Verifiable Data Registry kann mit Hilfe von Distributed Ledger-Technologie in einem Blockchain-basierten Netzwerk betrieben werden. Mehrere dezentrale Rechenknoten, die von unterschiedlichen Akteuren betrieben werden können, würden das Netzwerk bilden. Oftmals sind hier auch Non-Profit-Organisationen wie Sovrin oder die Energy Web Foundation aktiv. Ein deutlicher Kontrast zu den bisherigen Single-Sign-On-Anbietern.

Ein ausschlaggebender Faktor für den Erfolg von SSI wird sein, dass die Issuer von Credentials das erforderliche Vertrauen herstellen können. Weiterhin wird, wie so oft, die Nutzererfahrung und die daraus resultierende Akzeptanz entscheidend sein. Lösungsanbieter wie Spherity nutzen eine Cloud, um ein nutzerfreundliches Identitätswallet bieten zu können, das auch Private Keys verwahrt und vor dem ungewollten Verlust der Identität schützt.

Wer treibt SSI voran?

Die Bundesregierung und die Europäische Union sind unter anderem Treiber von SSI. Sie fördern das Entwickeln der Konzepte, Technologie, Standards und Regulierungen bis hin zu Pilotierungen in verschiedensten Branchen und Bereichen.

Das Bundesministerium für Wirtschaft und Energie fördert z.B. das „Schaufenster sichere digitale Identitäten“. Eines der Schaufensterprojekte ist IDunion. IDunion arbeitet an der Schaffung eines offenen Ökosystems für dezentrale, selbstsouveräne Identitäten. Dazu wird mit Blockchain-Technologie ein verteiltes Identitätsnetzwerk aufgebaut. Das Projekt wird von 47 namhaften Forschungs- und Industriepartnern vorangetrieben. 51nodes ist Partner von IDunion.

Die Europäische Union finanziert z.B. das „European Self-Sovereign Identity Framework Lab“. 20 Teilprojekte kümmern sich um die Schaffung und Erweiterung eines Open Source-Softwareframeworks für SSI. Weitere 42 Teilprojekte entwickeln kommerzielle Komponenten and Dienste unter Nutzung des SSI-Frameworks.

Anwendungsbeispiel: Wie kann SSI der Energiewirtschaft helfen?

Vor dem Hintergrund der Energiewende stehen die Betreiber von Übertragungs- und Verteilnetzen für elektrischen Strom vor großen Herausforderungen und neuen gesetzlichen Pflichten. Über eine Million (privater) dezentraler Stromerzeuger müssen effizient und effektiv in bestehende energiewirtschaftliche Prozesse (z.B. der Regelleistung und des Engpassmanagements im Redispatch 2.0) integriert werden.

Aktuell sind energiewirtschaftliche Prozesse gekennzeichnet durch produktspezifische und teilweise aufwendige Registrierungsprozesse an verschiedenen Stellen. Beispielsweise kann man hier das Marktstammdatenregister der Bundesnetzagentur, die Regelleistungsplattform der Übertragungsnetzbetreiber und die Plattform connect+ zur Kooperation und zum Datenaustausch der Netzbetreiber nennen. Hinzu kommt eine geringe Motivation der Betreiber von kleinen Energieerzeugungsanlagen und der Sicherung der Aktualität der Daten an den unterschiedlichen Registrierungsorten.

Die Frage von erforderlichen Nachweisen betrifft z.B. die Art der Stromerzeugung (z.B. EEG-Anlage) bzw. für welches energiewirtschaftliche Produkt die Anlage eingesetzt werden darf und kann. Die Identifikatoren (Energy Identification Code) für Anlagen werden zentral durch den BDEW vergeben. Weiterhin gibt es Vorgaben zur Marktkommunikation durch den Regulator, die mit Hilfe einer Zertifikatinfrastruktur für Signierung und Verschlüsselung erfolgt. D.h. die Zertifikate müssen bei allen genannten Stellen eingepflegt und aktuell gehalten werden. In einem “Internet of Energy”, wenn Millionen von Endkunden mit ihren dezentralen Flexibilitätsanlagen (z.B. Elektromobilität, Heimspeicher, Wärmepumpen, etc.) auf den Energiemarkt drängen, kommen diese Systeme zur Abwicklung der Massenprozesse für Registrierungs- und Veränderungsanforderungen in der notwendigen Geschwindigkeit an ihre Grenzen.

Der Ansatz von SSI verspricht demgegenüber die Vision der Integration von Millionen von dezentralen Anlagen in die Prozesse und IT-Systeme der Netzwirtschaft in unaufwändiger Plug-and-Play-Manier, ohne dass Mehrfachanmeldungen und parallele Konsistenzhaltung in redundanten isolierten Identitätsmanagementsystemen weiter erforderlich wären. Damit sollten Hürden für Anlagenbetreiber sinken und die Prozesse der Netzwirtschaft können schlanker und reibungsärmer aufgestellt werden. Schließlich würde ein wichtiger Beitrag für das Gelingen der Energiewende geleistet.

Aktuell arbeitet 51nodes gemeinsam mit dem Übertragungsnetzbetreiber TransnetBW an der Erörterung von SSI im Kontext der Energiewirtschaft. Bislang konnten schon mehr als zehn potenzielle Anwendungsfälle für SSI identifiziert werden.

Interesse an SSI?

51nodes ist ein Anbieter von Lösungen für die Crypto Economy. 51nodes unterstützt Unternehmen und andere Organisationen bei der Umsetzung ihrer SSI- und Blockchain-Projekte. Sie haben Fragestellungen zu digitalen Identitäten und Anknüpfungspunkte zur Nutzung von SSI? Kommen Sie gerne auf uns zu!

Weiterführende Informationen zu SSI

(1) Self-Sovereign Identity: The Ultimate Beginners Guide (Webseite)
(2) Self-Sovereign Identity: Grundlagen, Anwendungen und Potenziale portabler digitaler Identitäten (Whitepaper des Fraunhofer FIT)
(3) Chancen der Self-Sovereign Identities aus Sicht von Unternehmen für das Identity & Access Management (Webseite)
(4) Ein Überblick über das SSI-Ökosystem (wissenschaftlicher Artikel)
(5) Eine technische Umsetzungsplattform für SSI in der Energiewirtschaft: Energy Web Decentralized Operating System (Technische Konzeptdokumentation)

Das Potenzial von Selbstsouveränen Identitäten am Beispiel der Energiewirtschaft was originally published in 51nodes on Medium, where people are continuing the conversation by highlighting and responding to this story.

Introduction

Smart contracts are codified contracts using rules and algorithms that can automatically trigger and incur electronic transactions of cryptocurrencies like Ether. The expected economic impact of smart contracts is very large as they allow for improving the efficiency of many existing business models and they also allow for completely new business models. For instance payouts of insurance contracts can be automated using smart contracts or autonomous cars could also pay tolls and parking fees automatically.

While smart contracts are already well known amendments from blockchain approaches like Ethereum, these approaches suffer from shortcomings like low throughput, and high transaction costs. For these reasons, the IOTA foundation had set out to first propose a new and more scalable transaction system based on the so called “Tangle”. Secondly, IOTA version 2.0 is currently under development, and it includes an approach to smart contracts. Smart contracts under IOTA 2.0 are promised to be built on an infrastructure that both scales well, and incurs low transaction costs.

Before the background of a promising new approach for smart contracts, I started to explore how to set up a private IOTA network for development and how to develop and deploy a smart contract on it. As the IOTA developer documentation is lagging it was quite a challenge to get a working solution. Thus, the main contributions of this article are, (1) a proper description how to set up the network and how to enable smart contracts, and (2) a fully working demonstration smart contract, which implements a simple prediction market in which multiple network participants can predict and bet on a certain outcome of an event.

The outline of this article is, I give (1) an overview of IOTA and the network’s smart contract integration, (2) a description of how to set up an environment for developing and testing IOTA smart contracts, (3) an implementation of a simple prediction market as a smart contract, (4) some notes and insights on IOTA and smart contracts, and (5) a conclusion.

The IOTA Network and Smart Contracts

IOTA is about a new kind of public and permissionless distributed ledger for exchanging value and data. The IOTA network has been designed to overcome the main bottlenecks of Blockchain-based distributed ledgers. Due to the organization of transactions (of value) in a chain, there is just one end to append new ones, which makes it slow. Thus, in contrast to the Blockchain-based approaches the distributed ledger of IOTA is organized in a different way. The Blockchain, which is central to Bitcoin or Ethereum for instance, is replaced by the Tangle. The Tangle connects transactions via edges in a directed (and acyclical) graph (see next figure). In contrast to the Blockchain, there are multiple nodes (representing transactions) on which new edges to new nodes, i.e., transactions, can be appended.

Beside the data structure, there are further key differences to classic Blockchain-based approaches. The consent mechanism requires no miners because all users help in validating transactions. Therefore, transactions can be essentially conducted with zero fees. The current IOTA network still requires a central coordinator, defining trusted transaction milestones, which other transactions need to reference to be also trusted. To achieve true decentralization, the planned update to IOTA 2.0 should overcome this limitation.

The key properties and key promises of IOTA are of being

• highly scalable by a new data structure allowing for parallel transactions
• requiring few resources and being suitable for devices and sensors
• having zero-fee transactions
• running fast transactions
• finally approving messages within seconds
• providing a distributed network, which is robust against attacks

These properties would enable a layer of trust and a very scalable and efficient messaging system for the “machine economy” with a very large amount of devices connected to the Internet. The current version of IOTA on the Mainnet is 1.5, which went live in April 2021. Version 1.5 is seen as an intermediate step in maturing the IOTA technology and proving its usability and practical value. This would form the basis for smart contracts that use the messaging capabilities of the IOTA network. Smart contracts are supposed to be introduced in version 2.0 the latest and might be even backported to 1.5. At the time of writing, smart contracts are subject to development.

The design for IOTA 2.0 foresees different layers (see figure above) that separate the general underlying messaging, transacting, and validating infrastructure from the application layer, which comprises also smart contracts, which would then use layer 1 capabilities to realize their features. A main reason for separating smart contracts into another layer is to not compromise the messaging functionality. Smart contracts run on Wasp nodes on layer 2 in connection with Goshimmer on layer 1, responsible for conducting transactions and messaging (see next figure).

Due to the separation of Wasp and Goshimmer, they need a technical capability to connect and interact with each other. Therefore, Goshimmer nodes contain a “txstream” plugin, which needs to be activated for connecting Wasp nodes. Smart contracts are developed in the Rust programming language. Rust is known to generate very performant and memory efficient executables while preventing the developer from many classes of errors by a helpful compiler and a well-designed language. To test smart contracts and run unit tests against smart contract functionality, the “solo” environment written in Go can be used.

Setup

The described setup defines a private IOTA 2.0 network you can run on your laptop. The network hosts a layer 1 Goshimmer node for messaging and a layer 2 Wasp node for running smart contracts. On this basis, the next chapter shows how to develop and deploy a smart contract to this network.

Environment

I used Ubuntu 20.04 LTS with latest updates and upgrades and

• installed git
• installed rust 1.53.0 for developing native IOTA smart contracts by
  curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
• installed wasm-pack 0.10.0 for compiling smart contracts into “WebAssembly” binaries by
  curl https://rustwasm.github.io/wasm-pack/installer/init.sh -sSf | sh

Installed dependencies for rocksdb required by Goshimmer as its underlying database engine:

• sudo apt-get install libgflags-dev
• sudo apt-get install libsnappy-dev
• sudo apt-get install zlib1g-dev
• sudo apt-get install libbz2-dev
• sudo apt-get install liblz4-dev
• sudo apt-get install libzstd-dev

Cloned and built Goshimmer 0.7.5 from the develop branch (in the version of 2021–08–16) according the following description:

• git clone -b develop https://github.com/iotaledger/goshimmer.git
• git checkout tags/v0.7.5
• cd goshimmer
• go build -tags rocksdb

Prepared Goshimmer for transaction handling
Save the config.json file to your Goshimmer directory to

• enable the txstream plugin, which allows Goshimmer to communicate with Wasp nodes
• disable the portcheck plugin

To check whether Goshimmer synchronizes its time and whether messaging works, open up the dashboard on your local machine by http://127.0.0.1:8081/dashboard. It needs to display

• TangleTime Synced: Yes
• Message: DBVBaNbSEsq8D1SuNd7ULLeSPTXLwQBwfW1agWKnFX23 (as an example) — that is, the message must not read 1111111111111111111111111 .. (containing only ones)

Before running Goshimmer, delete the message database of previous (erroneous) attempts ( if any) by rm -rf mainnetdb in the mainnetdb subdirectory of Goshimmer. The database will be automatically generated again with a fresh start of Goshimmer.

Also in case tangle time does not synchronize (see the dashboard on your local machine by http://127.0.0.1:8081/dashboard or the Goshimmer log: “can’t issue payload: tangle not synced”), shutdown Goshimmer, delete the message database, and restart Goshimmer.

Run Goshimmer in its directory as follows — after having created the ./assets/snapshotTest.bin file as described subsequently:

./goshimmer --autopeering.seed=base58:8q491c3YWjbPwLmF2WD95YmCgh61j2kenCKHfGfByoWi --node.enablePlugins=bootstrap,prometheus,spammer,"webapi tools endpoint",activity,snapshot,txstream --messageLayer.startSynced=true --autopeering.entryNodes= --node.disablePlugins=clock --messageLayer.snapshot.file=./assets/snapshotTest.bin --messageLayer.snapshot.genesisNode= --metrics.manaResearch=false --mana.enableResearchVectors=false --mana.snapshotResetTime=true --statement.writeStatement=true --statement.writeManaThreshold=1.0 --config=./config.json

Create a cli wallet

The cli wallet can send IOTA funds to a Wasp wallet, where it is used to deploy smart contracts. We generate an initial transaction with funds for our cli wallet.

Install the cli-wallet in a new directory

• wget https://github.com/iotaledger/goshimmer/releases/tag/v0.7.5 download cli-wallet-0.7.5_Linux_x86_64.tar.gz
• tar -xf cli-wallet-0.7.5_Linux_x86_64.tar.gz

Set reuse_addresses=true in the config.json of cli-wallet:

To create a new wallet run ./cli-wallet init , returning

IOTA 2.0 DevNet CLI-Wallet 0.2 GENERATING NEW WALLET … [DONE] ================================================================ !!! PLEASE CREATE A BACKUP OF YOUR SEED !!! !!! !!! !!! E7owJWtDBGSUAZUWQkn1kHG5zUy2PLQf6eEr3RoMCJs7 !!! !!! !!! !!! PLEASE CREATE A BACKUP OF YOUR SEED !!! ================================================================ CREATING WALLET STATE FILE (wallet.dat) … [DONE]

Note your SEED for allocating funds to this wallet.

We generate a custom genesis snapshot, with the transaction that allocates the funds.

Go to the Goshimmer installation directory and then to the following subdirectory ./tools/genesis-snapshot

Paste the seed of the previously generated cli wallet to the following command

go run main.go --token-amount 3500000 --seed E7owJWtDBGSUAZUWQkn1kHG5zUy2PLQf6eEr3RoMCJs7 --snapshot-file snapshotTest.bin

Now,

• go to your Goshimmer directory and inside of it run
• mkdir assets
• cp ./tools/genesis-snapshot/snapshotTest.bin ./assets/snapshotTest.bin to provide the generated snapshotTest.bin file to Goshimmer.

Setting up a Wasp node for smart contracts

I installed Wasp from the master branch in the state of 2021–08–03.

• git clone https://github.com/iotaledger/wasp.git
• check out a workable state of the code from the repository. I used this commit.
• go build -tags rocksdb
• go build -tags rocksdb ./tools/wasp-cli

We need to transfer funds to the Wasp wallet by creating the wallet in the first place by ./wasp-cli init

We need to get the address of the wallet by ./wasp-cli balance , returning something like

Address index 0 Address: 1Ah4cqMPdrDGx6Htapk7NZUxxcYHsP1C3oAugEYHVmACj

To send funds to this wallet, paste your address into this command and run it in the cli-wallet’s directory:

./cli-wallet send-funds -amount 40000 -dest-addr 1Ah4cqMPdrDGx6Htapk7NZUxxcYHsP1C3oAugEYHVmACj

Now, ./wasp-cli balance returns a balance of 40,000 IOTA.

Finally, configure wasp-cli to be able to connect to the local Goshimmer node and to form a committee of one local Wasp node by saving the wasp-cli.json file to the directory of your wasp-cli.

Deploying a chain

Smart contracts are deployed on a chain, which needs to be deployed first:

./wasp-cli chain deploy --committee=0 --quorum=1 --chain=predmarketchain --description="Prediction Market"

where

• committee=0 specifies to use one Wasp node only, which handles smart contracts.
• quorum=1 says one Wasp node is enough here — for development and testing

Now we have to provide funds to the chain by

./wasp-cli chain deposit IOTA:1000 --chain=predmarketchain ,

reducing the wasp wallet’s balance by 1,000 IOTA.

A Prediction Market Smart Contract

The private IOTA network is now used to develop and deploy a smart contract. I report the design and implementation of a simple prediction market in Rust, how to build and deploy it, and finally how to use it.

Design

A classic example of realizing a smart contract is given by a prediction market. A prediction market is a virtual electronic market allowing to predict outcomes of future events by placing a monetary bet on a certain outcome. Such events could be sports events, political events, future prices of stocks, or other events with uncertain future outcomes. For instance, the outcome of a political election could be subject to predictions on a prediction market. A simple binary question to be answered by prediction market participants could be “Will candidate/team A win?” — with possible outcomes being “yes” or “no”.

Our design of a prediction market for demonstration purposes is simple. We omit a book maker and a pricing mechanism. Formally, we do not pose a question with predefined possible outcomes. Instead, market participants can bet basically on any outcome of an event with an arbitrary amount of tokens until the time for predictions is over. Afterwards, the winning outcome is determined and winning bets placed on the correct outcome receive back their share on the overall amount of tokens placed in bets. Assume, in total 700 tokens were bet on “no” and 300 tokens in total were bet on “yes”, and “yes” is the actual outcome. A single bet on “yes” with 100 tokens receives (100/300)*(700+300) = 333 tokens, making a win of 233 tokens.

Realizing this design as a smart contract in the IOTA network allows to deploy one contract per question to be answered. The account deploying the contract is in control and has to specify the time until when bets can be placed on outcomes. The actual question to be answered has to be communicated in third party channels. Any network participant can then look up the contract and call a function to place a bet on an outcome by sending some IOTA from their wallet. Finally, after the time for predictions and bets has passed, the deploying account has to call a function to close the prediction market and to provide the actual outcome of the event and correct answer to the question. This triggers the evaluation of all bets with regard to the correct answer. Accounts with the correct answer receive the winning amount of IOTAs in a transaction.

Note that in a real world and more production-like scenario, one might consider using an oracle to provide the outcome of an event. Oracles can stream off-chain data (about events) into the Tangle, so smart contracts can use this data in their evaluations.

Implementation

Smart contracts for the IOTA network can be implemented in Rust and then compiled to a WebAssembly file.

Our demonstration smart contract implemented in Rust can be viewed and cloned from this repository. The smart contract first exposes three functions for (1) initializing a prediction market, (2) placing a bet on an outcome, and (3) closing the prediction market for determining winners. When the contract is loaded, the mentioned functions’ implementations are made publicly available under the first string’s name, e.g., “initmarket”.

The first function should be called by the account deploying the contract for initialization. Optionally, the function can set an end time for betting using the parameter BETENDUTC, which is a date-and-time string in ISO format, assuming time in UTC. In case the parameter is omitted, bets can be placed at any time until the closemarket function is called (see below).

The second function allows to place a bet on a certain outcome value of an event, provided as parameter BETVALUE, e.g. “yes”. The amount to bet is the amount of IOTA sent with the function call. Bets must be placed in time before the betenddatetime has passed, which was set on initialization of the market. To save incoming bets for future evaluation to determine winners, I use two structs, defined in the beginning. That is, I define a hash map, mapping a betting account’s id to a Bet struct, which defines the amount of tokens and the outcome value of the bet. These custom structs are used instead of the built-in map offered by the context object of the function because only a proper hash map allows iterating over all keys and elements stored. As custom objects are not accommodated by the state stored in the context of the function, we need to jsonify it to produce a string, which can then be stored in the state.

The third function closes the prediction market and is to be called by the contract owner. Calls by other accounts will fail. The function can only be run after the specified end time has passed for predictions, and the function to close the market can be called successfully only once. The function requires a BETVALUE parameter, specifying the winning outcome, e.g., “yes”. The function runs through the stored bets, determines winning bets and the amount of IOTA coins they receive, and sends the IOTA to the wallets of the winners.

Build

To build the smart contract, one can pull the accompanying repository on github with the following structure:

* Cargo.toml 
* src/lib.rs 
* pkg/ 
* target/

The full Rust code of the smart contract is contained in a file called lib.rs. The structure and the naming of files follows standard conventions for Rust. To define dependencies of the smart contract code, the Cargo.toml file reads as follows:

The most important dependency is wasmlib, which is the IOTA’s smart contract library, allowing to produce smart contracts as compiled WebAssembly files. Furthermore, serde is used for serializing custom state objects to json strings for storage in the smart contract’s state. chrono brings some date and time related functionality required to control the end of a prediction market.

To build the smart contract, run wasm-pack build in the directory where Cargo.toml resides. The other directories like pkg and target are created automatically in the build process. The compiled WebAssembly file is located in the pkg directory and named predictionmarket_bg.wasm.

Preparation

We proceed by deploying our simple smart contract compiled as a WebAssembly wasm file.

Note: please adapt the path to the wasm file if required.

./wasp-cli chain deploy-contract wasmtime predictionmarket "Prediction Market SC" ./prediction-market-smart-contract/pkg/predictionmarket_bg.wasm --chain=predmarketchain --upload-quorum=1 -d --address-index=0

outputing:

uploaded blob to chain -- hash: 6wVabTkRUUGrQzEj8s4yuPC8dfaHGLsoHLXsqvveSw4hPosted on-ledger transaction BChQcWEmMptqRM1z4C9ZffTQXPebg1MMiEYENuAnV7KV containing 1 request:
- Request 4RkBSF6BAHgfamyJnXFy4r1YoVU9wvZ9b6uvXUjt2VWAfF5
Waiting for tx requests to be processed...
Posted off-ledger request 2JdtBzxjP4Bc6Tj2SnZPSctoYDHxuCAvPWJwn6ufMShJ6Lw

Now, functions can be called on the contract. First, the same wasp-cli that deployed the contract needs to call the initmarket function. There are two possibilities:

a) do not specify a specific an end date for the prediction market to simplify testing and development by this call ./wasp-cli chain post-request predictionmarket initmarket --chain=predmarketchain

b) specify a specific end date and time for the prediction market. The iso format is used and UTC is assumed. In this way, all bets must be placed before this time and the market can be only closed after this time. Run ./wasp-cli chain post-request predictionmarket initmarket string BETENDUTC string "2021-09-08 23:00" --chain=predmarketchain

Before placing a bet, we check the wallet’s balance with ./wasp-cli balance

returning

Address index 0
Address: 1BAgmaSN1RYk5rbbxMK21CZo8t2zQ3EFeMwDPnMhQdQbs
Balance:
IOTA: 28653
------
Total: 28653

For the deployed prediction market, we assume two possible outcomes “yes” and “no” on which bets can be submitted. To place a bet with 10 IOTA on “no”, we run

./wasp-cli chain post-request predictionmarket bet string BETVALUE string no --chain=predmarketchain -t IOTA:10

and afterwards, by ./wasp-cli balance we see the wallet's balance reduced by 10 IOTA.

Address index 0
Address: 1BAgmaSN1RYk5rbbxMK21CZo8t2zQ3EFeMwDPnMhQdQbs
Balance:
IOTA: 28643
------
Total: 28643

We now introduce and prepare the setup for four more participants to the prediction market. Each participant requires another wasp wallet. Thus, we create four new subdirectories waspwallet2, waspwallet3, waspwallet4, waspwallet5 and copy wasp-cli and wasp-cli.json to those directories by repeating with regard to these directories

The last command initializes a new wallet.

In case the contract’s code is changed, it needs to be re-deployed using the first and main wasp-cli. You also need to re-deploy the chain on which the contract is deployed first. Due to the redeployment of the contract, the copied versions of wasp-cli.json (in waspwallet2 to waspwallet5 directories) miss the new address of the deployed predmarketchain chain. In this case, we need to get it back in there to be able to run bets on the same prediction market from another account. Either you noted the address of the chain when it was created, such as

activating chain nZBwoJi5q7KGk8D2cgm16PWrdM6aL2qTdCY27HHjZgrK.. OK.

or you perform a cat wasp-cli.json in the directory of your first wasp wallet, giving you some information like this (among other information):

{
  "chains": {
    "predmarketchain": "nZBwoJi5q7KGk8D2cgm16PWrdM6aL2qTdCY27HHjZgrK",
  },

Now you need to provide this address of the predmarketchain to the new wasp wallets’ configuration files. To do this, you can run the following command for the new wallets in the respective directories (e.g. in waspwallet2 through waspwallet5 in our example).

./wasp-cli set chains.predmarketchain nZBwoJi5q7KGk8D2cgm16PWrdM6aL2qTdCY27HHjZgrK

Please adapt the chain’s actual address nZBwoJi5q7KGk8D2cgm16PWrdM6aL2qTdCY27HHjZgrK to yours.

Finally, all four new wasp wallets need to be funded. First, find out their address by running (for each new waspwallet subdirectory)

waspwallet2/wasp-cli address

Then, provide the funds using cli-wallet (in the directory where it resides on your computer) by running

./cli-wallet send-funds -amount 40000 -dest-addr 1Ah4cqMPdrDGx6Htapk7NZUxxcYHsP1C3oAugEYHVmACj

and replacing the address 1Ah4cqMPdrDGx6Htapk7NZUxxcYHsP1C3oAugEYHVmACj with the actual address found by running ./wasp-cli balance before in the respective subdirectories (of waspwallet2 to waspwallet5).

Simulation of a Prediction Market

Now we are ready to place bets in the deployed contract’s prediction market on behalf of these four new participants. So, in the respective subdirectories we run say

• cd waspwallet2
• ./wasp-cli chain post-request predictionmarket bet string BETVALUE string yes --chain=predmarketchain -t IOTA:100
• cd ../waspwallet3
• ./wasp-cli chain post-request predictionmarket bet string BETVALUE string no --chain=predmarketchain -t IOTA:50
• cd ../waspwallet4
• ./wasp-cli chain post-request predictionmarket bet string BETVALUE string yes --chain=predmarketchain -t IOTA:200
• cd ../waspwallet5
• ./wasp-cli chain post-request predictionmarket bet string BETVALUE string yes --chain=predmarketchain -t IOTA:500

Finally the contract owner (with the first wallet) can close the prediction market by running in the directory of the first wallet

./wasp-cli chain post-request predictionmarket closemarket string BETVALUE string no --chain=predmarketchain

In this example, the actual outcome is specified to be “no”. When running this command from a different wasp wallet’s directory, we obtain a log output on the Wasp node

You are not authorised to close the prediction market - only contract creator is allowed to close the market.

When successfully closing the prediction market, the Wasp node’s log outputs

CLOSEMARKET is executed:
the winning value is: "no"
total amount of bets placed on "no" is 60 IOTA
total amount of bets placed on "yes" is 800 IOTA
total amount of bets over all values: 860 IOTA
1FbCCHv9if3xbnRg3wJ7SY1kBFSdhNFR6Ax6haw6PhYDL placed a bet on "no", which is a WIN
bet amount: 10 IOTA; won amount: 143 IOTA; of total amount placed a bet on 860; where total amount per winning value: 60
transferring won amount of IOTA to: 1FbCCHv9if3xbnRg3wJ7SY1kBFSdhNFR6Ax6haw6PhYDL
1FZtVTCi2GDuQ1oMGZqpT38akLpcMiMv6a8MVKNJYYdsr placed a bet on "yes", which is not a win
1F81pGLKLhPb5ANFSGWQ7UPMSnPdahNZaZkgrcyaFXvpu placed a bet on "no", which is a WIN
bet amount: 50 IOTA; won amount: 716 IOTA; of total amount placed a bet on 860; where total amount per winning value: 60
transferring won amount of IOTA to: 1F81pGLKLhPb5ANFSGWQ7UPMSnPdahNZaZkgrcyaFXvpu
17jdFbAhWwF79fBEia6A8AYMTmMYncipaFhDTjsqUbMfp placed a bet on "yes", which is not a win
1BAgmaSN1RYk5rbbxMK21CZo8t2zQ3EFeMwDPnMhQdQbs placed a bet on "yes", which is not a win
consensus/action.go:338 postTransaction: POSTED TRANSACTION: 4Aw6PzQGkk6MFzPVAXZkz8RGiPocYxuxhA9o7qgeDN7h, number of inputs: 2, outputs: 3
EVENT: state was synced to block index #11, approving output: [0]4Aw6PzQGkk6MFzPVAXZkz8RGiPocYxuxhA9o7qgeDN7h
STATE TRANSITION TO #11. requests: 1, chain output: [0]4Aw6PzQGkk6MFzPVAXZkz8RGiPocYxuxhA9o7qgeDN7h

Revisiting the bets, we had placed the following ones:

Bets number 1 and 3 were on “no” and the bet was on the actual outcome “no”. So the total amount of IOTA was 860, the bets on “no” were only 60 IOTA in total. The share of bet number 1 is 10/60 and share of bet number 2 is 50/60. So, wasp wallet 1 receives 1/6 of 860 IOTA, i.e, 143 IOTA. And wasp wallet 3 receives 5/6 of 860 IOTA, i.e, 716 IOTA. Note that when transferring funds, a minimum transaction of fee of 1 IOTA is deducted from the amount to be transferred.

Note that running the closemarket function of the smart contract a second time leads to an error message in the Wasp node’s log:

the prediction market was already closed

Limitations

There are some limitations of the presented prediction market

• Only one contract per chain can be deployed because the bets are not stored per contract identification in the chain’s state
• All bets are stored on-chain, so they are public
• Each account (given by a wasp wallet) can place only one bet per deployed prediction market contract
• The actual question asked by the prediction market and the possible outcomes have to be conveyed informally
• Bets are against other market participants — there is no market maker

Insights

Along the way of producing this article and experimenting with IOTA smart contracts, many small insights were gained. With the hope that these insights will be helpful, I share them in the following.

Transaction Time

Running transactions in the described setup on a laptop can take several seconds when waiting for the wasp-cli commands to finish. However, they can be run asynchronously when using the wasp-cli, i.e., commands can return immediately after issuing a command. Of course, transactions can run in parallel. When running scripted wasp-cli requests, one has to take care not to create too much load on the Wasp node because otherwise requests will fail with “time out”-errors. To evaluate transaction times in a production setup on the Devnet or Mainnet, further investigations are required.

Transaction Fees

In principle fees for transactions and deployments are configurable. However, there are some minimum fees that apply such as for deploying a chain (100 IOTA), deploying a smart contract, and posting state changing requests to a smart contract (1 IOTA). With default minimum fees in mind, transaction costs in the IOTA network should be very low compared to transaction fees in the Ethereum network. Note that chain owners could also increase transaction fees.

Production Readiness

On a first shot, nothing really worked when trying to set up a private IOTA network based on public documentation. That is, one has to find workable versions of the software, find a proper configuration and parameters, and find working ways for funding wallets and deploying contracts. The IOTA 2.0 software is under development and special notice has to be taken on which versions to run and which version of the different pieces (Wasp, GoShimmer, cli) to combine. The documentation in IOTA’s repositories is lagging behind the development process. However, the IOTA community is very helpful and can be reached on Discord. So, getting the network up and running depended digging into material on the github repositories, third party information (e.g., on youtube), own experiments, and asking questions on IOTA’s discord server. Once the IOTA private network and the smart contract are up and running, you are ready for testing and experiments. Further steps taken by IOTA to bring the smart contract implementation and IOTA 2.0 to production readiness can be looked up in their roadmap.

Developing Smart Contracts

Using Rust as a language for developing smart contracts might help producing error-free and stable smart contracts because Rust is well-known for its strict and sophisticated compiler which together with the strong typing and the language design eliminates whole classes of errors common in other languages. Also, the compiler provides helpful error messages. Therefore, using Rust can provide some advantages over the Solidity language, which is usually used for developing Ethereum smart contracts. However, Rust might be unfamiliar for smart contract developers being used to Solidity.

Conclusion

This article discussed smart contracts as a new and upcoming feature in IOTA 2.0. Because IOTA has a different approach to conduct and store transactions in a more parallel way with fewer resources than Ethereum, a higher scalability and lower fees are promises that come with this approach. Our experimental setup of a private IOTA network and simple prediction market IOTA smart contract shows that it is already possible to use IOTA smart contracts for development and testing. As the process of setting up the IOTA network and running smart contracts was a bit challenging, I believe this article and example code can provide useful input and support for other developers and the community interested in IOTA smart contracts.

51nodes GmbH based in Stuttgart is a provider of crypto economy solutions. 51nodes supports companies and other organizations in realizing their Blockchain projects. 51nodes offers technical consulting and implementation with a focus on smart contracts, decentralized apps (DApps), integration of blockchain with industry applications, and tokenization of assets.

Thanks to Majd Turfa and Jan-Paul Buchwald for their help in the course of developing this article, experiments, and the setup.

Exploring IOTA 2.0 Smart Contracts in a Private Network: Developing a Prediction Market was originally published in 51nodes on Medium, where people are continuing the conversation by highlighting and responding to this story.

A short time ago, we as 51nodes decided to host our own Kusama Validator. This article will help you to understand how you can set up your own Validator on Polkadot’s Canary Network. At the very least you will walk away with a basic understanding of how staking works in Polkadot and what responsibilities different roles have in the ecosystem. Let's start with the most important concepts Validators and Nominators.

Validators and Nominators

In Polkadot’s ecosystem, a node participating in the BABE Consensus is called a Validator. Such an entity has three key responsibilities:

• Produce blocks if asked to do so
• Vote over the blocks (more precisely the “chain of blocks”) that still have to be finalized
• Forward messages between Parachains

To allow your Validator to become part of the active set that is currently validating the chain, you will need to stake a minimum of ~4000KSM. Obviously, holding or acquiring such a vast amount of KSM is only possible if you’re quite wealthy, as 4000KSM are 1.28 million dollars as of writing this.

A Nominator is an entity holding vast amounts of KSM, these are most likely entities involved in Polkadot for a long time. The Nominator elects different Validators to Stake his KSM. In short, this role describes someone who does not run his own node. Instead, the entity stakes their KSM most profitably by assigning their stake to a validator. Please understand that staking is risky for the Nominator, if the Validator — for whatever reason — stops running, they both risk getting slashed. The slash would ultimately affect the Nominator's stake.

This mechanism incentivizes Nominators to choose Validators that charge a minimal commission fee and have a good reputation in running their nodes. Validators in return get incentivized to run their node as stable as possible. In return Validators can use low commission fees to attract Nominators.

Controller and Stash

As we now know the two most important roles in the consensus process we can talk about the accounts you have to set up to run your Validator. Two account types are of importance at this stage — The Controller and the Stash.

Controller and Stash Accounts are explained quite easily. While the Stash is the entity holding the majority of your funds, the Controller is the entity having control over the Stash and the actions you take — like starting or stopping your Validator.

Setting up a Secure Validator

To set up your Validator you can either choose the manual set-up or the Secure Validator Setup. As you can already guess from the subtitle we have chosen the second option.

Within the Secure Validator Setup, you will have two options for setting up the node. The first option is to set up your server and the Polkadot application in one run using Terraform. We decided to use the second option which is to set up your server yourself and then use Ansible to securely set up the Polkadot application.

For the following setup you will need the following:

1. At least one Debian-based machine (preferably Ubuntu 18.04 as used in Polkadot’s Documentation) with the following specs (min. 300GB Storage, 2–8GB Memory, 1–2 CPUs) hosted with the Provider of your choice.
2. Ansible(v2.8+) installed on your machine
3. A minimal of 3KSM depending on the amount you want to stake yourself. 0.3KSM will be needed for requesting an on-chain identity. 1–2 KSM should be held for upcoming transaction fees. The overhead can then be used to stake using your own validator. A higher amount of self stake raises trust in your validator as you project more trust on it yourself.

Preparing Your Node

Before deploying Polkadot to your remote machine you will need to prepare an additional user and ssh access for this user. Start with creating a new admin user and then create new ssh keys. Preferably you should also disable the root login as described in the ssh key instructions.

Running the Secure Validator Setup

If you successfully opened a shell to your remote machine, you can adjust the Ansible script for your needs. First, clone the repository:

git clone https://github.com/w3f/polkadot-validator-setup.git

Now start adjusting the ansible/inventory.sample file according to the instructions given in the setup guide. Below you can find an already edited example inventory file using placeholder values:

If you are done adjusting the inventory file to your needs you should be able to execute the playbook via the provided script:

chmod +x setup.sh
./setup.sh my_inventory.yml

The Ansible Playbook will now be executed and deploy the Polkadot service to your remote machine.

In case everything worked well your node should show up on the telemetry. It will now take some time until the node is in sync with the network. For us, this took nearly 7 days. Once finished you will only need to bond your KSM and set your session keys to enter your validator into the “waiting” list. From there on, it is a game of building trust into your Validator so that enough foreign stake is received to participate in the active validators pool. The more trust you generate in you and your node the more steady your rewards will become in the long term. To achieve more trust in your node you can also have a look at on-chain identities. For about 0.3KSM you can request a Registrar to verify your personal information which Nominators can then use to get more information on who is actually running the node.

Monitoring Your Validator

As keeping your validator up and running is key to running your node successfully, good monitoring is a must-have. For an initial layer of monitoring, you can use a Telegram Bot. But for the more sophisticated part of the monitoring, most people will use Prometheus and Grafana. You can set these up following the instructions given in Polkadot's documentation but beware that some metrics might not show up in the provided Grafana dashboard as not all metrics are delivered which is what happened to us. This is why we enabled the node exporter in the Secure Validator Setup which provides additional data and can be displayed in Grafana with dashboards like this.

Validator Stats and Nomination

In the PolkadotJS-UI under Validator Stats you can browse Validators and inspect stats like rewards or previous slashes. For example, search for ESgN4sdziBpAubx2pGJ1CapGxetE2E1zUbTQCaRFbxxrW2Y and you will find our Validator. Using the displayed information a Nominator can determine his risk with a Validator. He should now be able to nominate the validator as described in the documentation.

51nodes GmbH based in Stuttgart is a provider of crypto-economy solutions.

51nodes supports companies and other organizations in realizing their Blockchain projects. 51nodes offers technical consulting and implementation with a focus on smart contracts, decentralized apps (DApps), integration of Blockchain with industry applications, and tokenization of assets.

[1] https://wiki.polkadot.network/docs/en/learn-staking

Kusama Validator Node Setup was originally published in 51nodes on Medium, where people are continuing the conversation by highlighting and responding to this story.

Some weeks ago we published an article in which we refer to one of our GitHub projects that you can use to deploy a quorum network with a dynamic amount of nodes to your Kubernetes cluster. In this article, I will describe how to set up our newest addition to the Helm Chart — the Epirus block explorer.

Why we use Epirus

If you come across the task to choose a block explorer in 2021 you will most likely have 2–3 viable solutions of which some are licensed and some are free to use. Most Ethereum Clients will most likely favor BlockScout, an open-source block explorer with a wide variety of features. But, if it comes to quorum-based networks we discovered some quirks that do now align well with BlockScout. One example would be the BlockScout having problems parsing rafts block timestamp which is in nanoseconds instead of seconds. We discovered that those kinds of problems often arise with open-sourced explorers in combination with quorum-based networks. This could be due to the enterprise nature of quorum which ultimately could lead to fewer open-source contributions.

Based on those discoveries we decided to stick with the explorer that is covering most of quorums peculiarities and is also referred to in the official quorum documentation — Epirus.

How to deploy the Explorer

To deploy the Epirus Explorer to your cluster adjust the epirus values in the values.yaml file. Enable the ingress value to make it accessible locally or adjust the node value to change to which node Epirus will connect to.

After adjusting these values simply run helm install — if you don't have a running cluster — or helm upgrade if you already deployed a cluster on your machine to deploy/upgrade the charts.

helm install nnodes quorum -n quorum-network                       helm upgrade nnodes quorum -n quorum-network

Accessing the UI

After a successful deployment of the explorer, we can access the UI using the Cluster’s IP. Run the following command to get the IP and go to <cluster-ip>/dashboard.

kubectl cluster-info

If everything deployed and started successfully you should be welcomed by the Epirus Dashboard. Navigate to the different tabs on the left to get information on contracts, transactions, blocks, and the network itself.

51nodes GmbH based in Stuttgart is a provider of crypto-economy solutions.

51nodes supports companies and other organizations in realizing their Blockchain projects. 51nodes offers technical consulting and implementation with a focus on smart contracts, decentralized apps (DApps), integration of Blockchain with industry applications, and tokenization of assets.

Epirus Chain Explorer for Quorum was originally published in 51nodes on Medium, where people are continuing the conversation by highlighting and responding to this story.

Note: This article provides a general and slightly technical introduction to Parachains and indicates how to get started building a Substrate based blockchain which can be launched as Parachain in the Kusama & Polkadot ecosystem. It will explain some parts of Polkadot but it won’t get too much into specific details like staking, slashing or other important mechanics in the ecosystem. Check out the official wiki to learn more about Polkadot.

Status quo

First of all it is important to mention that the Parachain functionality is not live yet. This functionality is the last missing piece of the Polkadot whitepaper and there is a public rollout plan available that provides an overview about the progress. However, since August 2020 the Parachain functionality is being tested in the official Parachain Testnet Rococo which in V0 only ran with prototype code. In December 2020 Rococo V1 was launched and recently Plasm Network announced to be the first project that joined the Rococo V1 Testnet which surely is a big milestone as the codebase of Rococo V1 will also be used in Kusama and Polkadot. Meanwhile other possible future Parachains like KILT followed and announced their own Roadmap.

Kusama vs. Polkadot

Although Kusama can be seen as the pre-production environment for Polkadot you should be aware that Kusama might and probably will evolve independently from Polkadot. If your project is in an early stage or you want to experiment with Parachains in a real world environment before moving it into production you will probably prefer Kusama over Polkadot first. You might also be interested in testing specific features in Kusama which aren’t yet available on Polkadot.

Relay Chain & Parachains

The Relay Chain is the central chain of Polkadot. All validators of Polkadot are staked on the Relay Chain in DOT and validate for the Relay Chain. It has deliberately minimal functionality and its main responsibility is to coordinate the system as a whole, including Parachains. Other specific work is delegated to the Parachains that serve different purposes. All Parachains that are connected to the Relay Chain share the same security. Polkadot also has a shared state between the Relay Chain and all connected Parachains to ensure the validity of the entire system. Smart Contracts are not supported on the Relay Chain but can be supported by Parachains.

It’s expected that each Parachain will be as light and application specific as possible and serve a specific use case.

In The Path of a Parachain Block it is described in detail how a new Parachain Block is produced.

You may notice the term Parathread in the picture above. The current (optimistic) assumption is that the Relay Chain supports up to 100 Parachains and some of these Parachain slots will be permanently reserved to serve the network on system level (e.g. Governance). Thus there probably won’t be enough slots available for all projects which is why they need to be obtained in a so-called auction (more on that below). Whereas Parachains get a guaranteed high throughput by bonding DOT tokens, Parathreads follow a pay as you go model where a fee in DOT is being paid to validators for creating blocks. In some cases this even makes sense if there aren’t many or high frequent state updates expected (e.g. DNS, Oracles).

Parachains can become Parathreads and vice versa as they share the same technical base.

Parachain Slot Auction & Parachain Crowdloans

In order to become a Parachain in Kusama or Polkadot a slot must be obtained in a Parachain Slot Auction. Each slot duration is capped to 2 years and divided into 6-month lease periods. Parachains may lease more than one slot over time in order to extend their lease to Polkadot past the 2 year slot duration. Depending on the auction it might be possible that a slot is owned by 4 different Parachains during the 2 year slot duration.

Parachains don’t need to always inhabit the same slot. As long as a Parachain inhabits any slot it can continue as Parachain.

Parachain candidates can place bids in unpermissioned candle auctions that rely on verifiable random functions (VRFs) where the original mechanism has been slightly modified to be secure on a blockchain. The auction can be divided into two phases:

• Opening Phase
• Closing Phase

No matter in what phase the auction is, the bids are public. From the first block until the last block during the Closing Phase a winner will be determined randomly and nobody knows which block determines the winner until the Closing Phase is finished.

Projects will also be able to launch a Crowdloan Campaign with specific parameters which allows them to loan DOTs from the community to obtain a Parachain slot without having to bond all the required DOTs on their own.

In the video above core developer Shawn Tabrizi explains both variants very well.

Substrate

After giving an overview about the Polkadot ecosystem it’s time to introduce Substrate. It is a modular framework that enables you to create purpose-built blockchains by composing custom and/or pre-built components.

Substrate is the foundation of all blockchains in the Polkadot ecosystem and requires knowledge in Rust.

In the Substrate Developer Hub (link below) you can learn everything that is required to get started developing your own Substrate-based chain.

Official Substrate Documentation for Blockchain Developers · Substrate Developer Hub

A few important things to know about Substrate:

• The runtime of Substrate is referred to as the “state transition function” which contains the business logic that defines the behavior of the blockchain. You as a developer can define storage items that represent the state of the blockchain and functions that allow users to make changes to the state.
• Substrate ships with FRAME (Framework for Runtime Aggregation of Modularized Entities) which is a set of modules (called Pallets) and support libraries that simplify runtime development. Pallets are individual modules within FRAME that host domain-specific logic.
• To enable forkless runtime upgrade capabilities, Substrate uses runtimes that are built as WebAssembly (WASM) bytecode.

To learn more about how to perform a forkless runtime upgrade you can take a look at the official tutorial.

Pallets vs. Smart Contracts

For people heading over from other chains and ecosystems like Ethereum it might be obvious that application specific logic is always written in Smart Contracts which in turn are deployed on the blockchain. But as you already learned the idea of the Polkadot ecosystem is to have different Parachains that are as light and as application specific as possible.

There are basically 3 possibilities to add custom logic to a Substrated-based blockchain:

1. Write custom Pallets and include it in the runtime
   - https://substrate.dev/docs/en/tutorials/create-a-pallet/
   - https://substrate.dev/recipes/pallets-intro.html
2. Include the Contracts Pallet in the runtime and write contracts in ink! (Rust based eDSL)
   - https://paritytech.github.io/ink-docs
   - https://substrate.dev/substrate-contracts-workshop
3. Include Frontier (Pallets that serve as Ethereum compatibility layer) to the runtime and write contracts in Solidity
   - https://github.com/paritytech/frontier
   - https://substrate.dev/frontier-workshop

Custom Pallets are the best choice if you are building a greenfield project that serves a specific purpose and needs to run its own network.

Substrate already provides a lot of Pallets which you can combine together as needed, including your custom Pallet(s) in order to build a lightweight and unique runtime for your application specific blockchain:

• Default Pallets of Substrate (Note: The link points to the v3.0.0 version of Substrate which was released a few days ago. Most tutorials are still based on v2.0.0)
• NFT Pallet (There is also a WIP reference implementation that showcases CryptoKitties on Substrate)

I also discovered a Substrate Marketplace where you are able to find different Pallets (at least the default ones) which you can include into your runtime.

If you aim to write contracts in ink! or port a Solidity based application from Ethereum over to a Substrate-based Parachain you don’t necessarily need to launch your own Parachain as there already exist projects that plan to launch a Parachain and include the required Pallets in their runtime:

• Edgeware (ink! based contracts)
• Moonbeam (Solidity contracts)

Note: To check out how Moonbeam works you can clone the following repository: https://github.com/51nodes/moonbeam-playground

Polkadot compatibility with Cumulus

When you have written (and hopefully tested ;-)) your custom Pallets you need to make sure that you build a Polkadot compatible runtime that exposes an interface for validating its state transition and provides interfaces to send and receive messages of other Parachains.

Cumulus is an extension to Substrate that makes it easy to transform any Substrate-built runtime into a Polkadot-compatible Parachain. It is still in development, but the idea is that it should be simple to take a Substrate chain and add the Parachain code by importing the crates and adding a single line of code. You can also dive deeper into that topic and learn more about the mechanics of Polkadot by reading the Parachain Implementers Guide.

If you have built your own Parachain and want to join the official Rococo Testnet you have to follow this guide. The auction mechanism mentioned above is not active yet and Rococo is currently controlled by the development leads at Parity Technologies. For joining the network you have to run at least 1 Collator for your Parachain and 1 Validator node for Rococo.

Note: Learn how to setup a local Polkadot network based on Rococo by following the Readme of our repository: https://github.com/51nodes/polkadot-local-rococo

Final Thoughts

Substrate has a clean design and makes it very easy to build an application specific blockchain for everyone familiar with Rust. If you decide to run your own Parachain (or Parathread) you should also think about how to incentivise other players to run a Collator node in your network — for example by introducing your own network token.

Although Parachains are not live on Kusama and Polkadot yet we can already see many projects that aim to become a Parachain as soon as possible. It will be interesting to watch the battle of those projects to secure their exclusive Parachain slot on Kusama and Polkadot. Personally I am very interested to see how the auction mechanism works and which projects will secure their Parachain slot via a Crowdloan Campaign.

I am also looking forward to play around with the interoperability features that Polkadot provides when the implementation is mature enough.

51nodes GmbH based in Stuttgart is a provider of Crypto Economy solutions.

51nodes is member of the Substrate Delivery Partners program and supports companies and other organizations in realizing their blockchain projects. 51nodes offers technical consulting and implementation with a focus on smart contracts, decentralized apps (dApps), integration of blockchain with industry applications, and tokenization of assets.

Kusama & Polkadot: Build an application specific Blockchain and launch a Parachain! was originally published in 51nodes on Medium, where people are continuing the conversation by highlighting and responding to this story.

Motivation

Following this informative article provided by my colleague Majd, in which you can learn the basics about Quorum and how to set up a minimal Quorum network using docker containers, we decided to take things a bit further and use our knowledge to provide a dynamic Quorum setup deployable to Kubernetes. In our GitHub repository, we provide a Helm chart and some scripts which aim to make developing and testing on Quorum a lot faster and easier. This article will provide some insights regarding the setup and the functionalities of the project.

Tooling & Prerequisites

For our setup, we need a running Kubernetes cluster and Helm. Helm enables us to deploy a preconfigured network to a running cluster using the concept of charts. This, in combination with some scripts, gives us the ability to dynamically add and remove nodes to and from the network.

Minikube (Kubernetes)

Minikube is one of the multiple tools which you can use to spin up a Kubernetes cluster locally. Other viable options are k3s or kind. Those tools are widely used to develop and test applications on local infrastructure, before deploying them to the target infrastructure.

Helm

Here is a pretty good explanation of what Helm does and how it is connected with Kubernetes:

In simple terms, Helm is a package manager for Kubernetes. Helm is the K8s equivalent of yum or apt. Helm deploys charts, which you can think of as a packaged application. It is a collection of all your versioned, pre-configured application resources which can be deployed as one unit. You can then deploy another version of the chart with a different set of configuration.

Dynamic Nodes

Now that we know the technical requirements we can take a closer look at the actual repository. The intention for this setup was to use Quorum on Kubernetes. As we did not find any solutions for this besides Qubernetes — the officially supported way to deploy Quorum to Kubernetes — we decided to create our own deployments using Helm. This in particular has the benefit of being more flexible than the officially supported way where Kubernetes deployments get generated once and have to be regenerated and redeployed everytime you need another setup. Using our dynamic approach you will be able to keep the network running while adding or removing nodes.

Quorum Configuration

The following code shows the values.yaml file in which most of the action will take place. Those values are used to dynamically fill a set of templates which you can then reuse for the number of nodes you want in your network. The setup also allows you to modify some additional configurations regarding the deployment such as input parameters for Quorum and Geth.

The values file additionally takes input for the nodes which are going to be deployed to the cluster. Some of those values are needed for adding a node to a raft based Quorum cluster and others give some additional functionality like enabling or disabling endpoints.

Under endpoints, RPC, as well as WebSocket endpoints, can be turned on and off. This in particular is used for communication inside the cluster. Additionally — if needed — you can enable the ingress controller to easily access nodes from outside the cluster. If you have Ingress enabled you can access the nodes at http://<cluster-ip>/quorum-node<n>-rpc or http://<cluster-ip>/quorum-node<n>-ws. Nodekey and enode represent the key material for a raft node, they can be generated using the bootnode command provided by Geth. The key value is a Geth Keystore file which holds the account credentials for a Geth account. Only with the combination of bootnode credential and Geth account, the node will be able to function properly. Note that example credentials as provided in the repository should never be reused in any production environment.

Reusing Templates

The above values will then be used to fill in the provided templates. Those templates usually take values for exactly one node/deployment.

By looping through the nodes object of the values file at the top of each template, we can reuse these for every new node added to the values file. Here's an example for a PersistentVolumeClaim:

Deploying and Updating the Chart

Now, let's start with deploying the network to a running Kubernetes infrastructure.

Upgrades, after changing the configuration of your network can be installed by using:

Adding New Nodes

Besides adding and removing nodes by modifying the values file manually, we also implemented a more convenient way to do this. For this, we provide multiple scripts that allow us to add and remove specific as well as multiple nodes.

In the following, I will use the addNodes.sh script to dynamically add new nodes to a running cluster.

The addNodes.sh script now allows me to decide how many nodes I want to add to the cluster. I choose to add 2 additional nodes and the script automatically generates the corresponding credentials, adds them to the values file, and upgrades the deployed Helm chart.

Note the additionally generated value raftId. This value is required for every additional node as it enables the flag “ — raftjoinexisting <raftId>” which is needed to properly add a new node to the initial 3 nodes cluster.

After updating the chart you can see that the new nodes have been added to the cluster. To confirm that the cluster is running and is properly synchronized you can run the following command, which will open a shell to Geth running inside the container. You can then use this open shell to inspect the state of raft (see below) or execute different geth commands. If the resulting nodeActive value is true, the node is properly synced with the cluster and ready to go.

Conclusion

All in all this project should help people to improve testing and development of Quorum networks running on Kubernetes infrastructure. Besides that, it is also a simple and convenient way to gather some experience using Quorum and experiment with various settings, especially if you have problems figuring out the right configuration for your cause. So if you want to take a look at the repository feel free to do so. If you have any suggestions or improvements you might want to create a PR which we will happily review.

51nodes GmbH based in Stuttgart is a provider of crypto-economy solutions.

51nodes supports companies and other organizations in realizing their Blockchain projects. 51nodes offers technical consulting and implementation with a focus on smart contracts, decentralized apps (DApps), integration of Blockchain with industry applications, and tokenization of assets.

Quorum Kubernetes Templates was originally published in 51nodes on Medium, where people are continuing the conversation by highlighting and responding to this story.