Sharding [ /ˈʃɑːdɪŋ/ ] is a method of partitioning a database horizontally across separate servers to improve scalability, performance and data availability.
In distributed ledgers (DLTs) like Radix, sharding is used to allocate both data storage and transaction execution across a decentralized network of nodes to achieve a high transactional capacity.
Background
Traditional blockchains like Bitcoin and Ethereum rely on every node in the network processing and storing every transaction. This provides strong decentralization and security by avoiding reliance on any trusted parties. However, it fundamentally limits the throughput of the system to what a single node can validate, resulting in poor scalability.
Sharding aims to transcend this trilemma by creating a mechanism where nodes only need to store and process a subset of the total transactions, known as a ‘shard.’ By splitting up the workload in this manner, sharding enables distributed ledgers to parallelize the processing of transactions across shards, potentially increasing the throughput of the network quadratically compared to a non-sharded system.
Sharding in Radix
Radix has developed an integrated sharding and consensus architecture specifically designed for hyper-scalability of its decentralized network. In Radix’s case, sharding applies to both data availability and transaction execution as both functions are performed by nodes.
Ledger Pre-Sharding
The current Radix Mainnet (Babylon) is sharded into a fixed number of 2^256 shards. Responsibility for validating shards is undertaken by groups of validators called shard groups, which may grow or shrink dynamically in response to load demand. Currently, the number of shard groups is capped at one but this will be lifted with Radix’s forthcoming Xi’an release.
Pre-sharding is in contrast to the dynamic adaptive state sharding model adopted by Shardeum, MultiversX, and NEAR, where shards are added incrementally as required. While sharding can improve scalability, an ad hoc approach to sharding leads to substantial difficulties as any changes to the shard structure require reorganizing the entire network - a time consuming and expensive process. The larger the sharded ledger grows, the more problematic this becomes. Ad hoc sharding also complicates queries and data lookups within the ledger. By sharding the data randomly, it becomes much harder to locate specific transactions or data points since they could be stored anywhere. This slows down queries as more extensive searches are required.
Deterministic Shard Indexing
Shards on Radix are indexed deterministically by public keys. This means that the shard index for any address can be calculated by taking the modulo of the public key over the shard space.
By deterministically grouping related data into the same shard, Radix avoids the need for expensive data reorganization as the network grows. This creates four major advantages:
- Proximity: All transactions from a particular account are guaranteed to be in the same shard, which makes it trivial to identify attempted double-spends.
- Asynchrony: Transactions from separate accounts will always involve separate shards, enabling asynchronous, parallel processing of unrelated transactions.
- Indexing: Lookup complexity and query time are reduced since shard locations can be easily derived from public keys.
- Load balancing: Hash sharding typically results in a more uniform distribution of data across nodes.
Network Security
A key challenge in sharding distributed ledgers is ensuring sufficient security and node coverage across all shards. If some shards have much fewer nodes than others, it creates vulnerabilities. Radix employs several techniques to maintain security across its sharded network:
- Node Identity Shard Mapping: To secure the network, validator node addresses are mapped to a single ‘root’ shard. Nodes must permanently maintain their root shards, but can support additional shards to earn more transaction fees. Underserved shards offer higher returns, attracting more validators and preventing any shards from being overlooked. This free market approach maintains security even as the network scales.
- Incentives for Multi-Shard Validation: Based on factors like computing resources, validators can choose to support additional shards beyond their root shard. The more shards a node supports, the greater the amount of transaction fees it can earn. This creates an incentive for validators to support as many shards as feasible to maximize profits. In this way, the overall validation workload is distributed across nodes.
- Dynamic Shard Support via Free Market: As the network grows, some shards may end up with fewer nodes supporting them compared to other oversubscribed shards. These underserved shards then inherently offer higher potential returns since there is less competition for fees. The higher relative profits attract more validators to begin supporting the underserved shards. This brings coverage back into equilibrium across shards through a free market approach.
- Scaling Security Through Staking: In proof-of-stake networks like Radix, staking provides additional security. The more tokens a validator stakes, the more shards it can validate. This allows validation load to scale up securely. High stake validators may validate transactions across many shards in parallel for efficiency. However low stake nodes still play a key role in providing decentralized shard coverage.
Together, these mechanisms ensure Radix can securely scale to an exponentially growing shard space without running into coverage gaps or centralization issues. The network organically self-regulates to distribute validation across shards.
Cerberus Consensus
Main article: Cerberus
Radix's Cerberus consensus protocol introduces ‘braided’ sharding to atomically compose transactions across shards. Cerberus shards transaction validation while braiding validation across shards to enforce system-wide transaction ordering and prevent double-spending. This unique braided architecture ensures that Radix can securely scale transaction throughput across a sharded network of effectively unlimited size.
Advantages
The key advantage of sharding is that it enables parallel transaction processing and storage, significantly increasing throughput. By sharding the ledger and workload, each node only needs to maintain a subset of the overall data while still contributing to the security and integrity of the full ledger. However, sharding done through an ad hoc approach leads to difficulties. Later changes to the shard structure would require full network reorganization. Query complexity also increases with ad hoc sharding as locating data becomes more difficult. These challenges make ad hoc sharding unsuitable for distributed ledgers as they grow to Internet scale.
By breaking up large datasets into smaller, more manageable pieces, network nodes are better able to search through and retrieve individual data points as well as handle concurrent requests. This approach is especially beneficial when unsharded systems grow excessively large, resulting in performance degradation. By implementing sharding, data storage and processing tasks are distributed across multiple computers, enhancing the system's scalability.
Sharding is commonly used in distributed databases, where it allows for the efficient storage and retrieval of large amounts of data across multiple nodes. By dividing the data into shards and distributing them across multiple nodes, a sharded database can support more concurrent requests and handle larger volumes of data without slowing down or becoming overloaded.
In addition to improving scalability and performance, sharding can also help to improve the availability and reliability of a distributed system. By storing data on multiple nodes, a sharded system can continue to function even if one or more nodes fail, ensuring that the data remains accessible and that the system can continue to serve requests.
There are several different approaches to sharding, each with its own tradeoffs and benefits. Some common sharding strategies include range-based sharding, which divides data into shards based on a key value or range; hash-based sharding, which uses a hash function to distribute data across shards; and directory-based sharding, which uses a lookup table to determine which shard a piece of data belongs to.
Disadvantages
There are several key tradeoffs and limitations that sharded architectures must contend with:
- Vulnerability to adaptive adversaries that can target specific nodes, and weaker accountability when parts of the system are compromised compared to breaking the entire network.
- Reliance on a sufficient number of continually online nodes performing data availability sampling for each shard. This creates a "few-of-N" trust assumption among nodes rather than full trust minimization.
- Increased data propagation requirements across shards compared to traditional blockchains, raising risks under extreme network conditions.
- Potential for "shard-level" attacks that could overwhelm the relatively fewer nodes participating in each individual shard's peer-to-peer network.
While formidable, these concessions are considered an acceptable tradeoff to achieve the scalability promised by sharding. As sharded throughput increases, these limitations effectively bound the overall scalability gains relative to an idealized, fully parallelized system.
Sharded designs strive to retain the user-level decentralization properties of permissionless blockchains like Ethereum, while enabling orders of magnitude higher transaction throughput. This differentiates sharding from alternative "scaling" paths like centralized high-throughput chains or multi-chain ecosystems with reduced security.
Sharding's complexities within distributed ledgers hinge on ensuring single transactional occurrences. Given the spatially distributed nature of ledger shards, instantaneous consistency remains elusive. The CAP theorem suggests that a distributed data store can't simultaneously guarantee Consistency, Availability, and Partition tolerance.
Comparison with Other Scaling Solutions
Sharding represents one approach to scaling blockchain throughput among several proposed techniques. Each solution makes different tradeoffs between scalability, decentralization, and security:
Traditional Blockchains
Projects like Bitcoin and Ethereum have strong decentralization and security by requiring every node to process every transaction. However, this fundamentally limits scalability to what a single node can validate.
High-TPS Chains
Chains like those using Delegated Proof-of-Stake (DPoS) consensus achieve scalability by having a small set of nodes (often 10-100) maintain consensus. This provides security against Byzantine faults by that set, but requires trusting a majority of those privileged nodes, reducing decentralization.
Multi-Chain Ecosystems
The idea of using multiple parallel chains that communicate ("scaling out") provides scalability and decentralization in the aggregate. However, each individual chain has lower security - attackers need only compromise a small portion of the resources to break a single chain.
Sharded blockchains aim to be scalable like high-TPS and multi-chain designs, while maintaining a high degree of decentralization and security closer to traditional blockchains. This is achieved through secure data partitioning and distributed validation across shards.
Compared to traditional blockchains, sharding trades off simplicity and minimized trust assumptions for increased throughput. Versus high-TPS chains, sharding avoids inherent centralization by avoiding fixed validator sets. Against multi-chain ecosystems, sharding's unified state and security model prevents compromising the full system by attacking a minority partition.
Some proposals suggest combining the throughput of high-TPS production with sharded validation as a middle-ground. However, this fails to replicate sharding's decentralization advantages and makes censorship harder to detect.
Sharding uniquely aims for an integrated solution that is scalable, secure, and maintains blockchain's core property of minimizing trust in privileged third-parties overall.