Leaf States

Each leaf is in one of the following states:

State Description
Unknown In this state, a leaf is not part of the cluster. This is the state before you run ADD LEAF to introduce the leaf into the system. A leaf that is unknown will not show up in SHOW LEAVES . The REMOVE LEAF command will transition a leaf into the unknown state.
Online This is the default, healthy state of a leaf. In the online state, the leaf is an active member of the distributed system and is either currently serving or ready to serve data to the aggregators.
Offline A leaf enters the offline state if the master aggregator cannot reach it. The master aggregator sends a heartbeat to each leaf node and if a leaf fails enough heartbeats it will be moved to the Offline state. If the cluster is in redundancy 2 any partitions on the offline node will be failed over to the leaves pair. The master aggregator continues to ping offline nodes to detect when they should be moved into the attaching state.
Recovering Not online, and not available for read or write queries. A leaf in recovering state has been restarted and is replaying data back into memory.
Detached A leaf transitions from online to detached when DETACH LEAF is run. You can reintroduce the leaf into the online state with the ATTACH LEAF command. This command will automatically try to reuse the data present on the leaf; this is especially important in redundancy-1. DETACH LEAF is the same as REMOVE LEAF except that after rebalancing away a leaf’s partitions, it transitions it to detached instead of unknown.
Attaching A leaf transitions from offline to attaching when it is once again reachable by heartbeats. The master aggregator will automatically run ATTACH LEAF to attach any partitions on the leaf back into the cluster. While the attach is running the leaf will stay in the attaching state. When the attach is finished the leaf will move to the online state.


The following diagram summarizes the leaf states and the transitions between them.