---

Upgrading a Cluster

For a MemSQL cluster, the upgrade process involves individually upgrading each node in the cluster in a very specific sequence. There are two approaches:

• If the cluster is running at Redundancy Level 2, you have the option to perform an incremental cluster upgrade, in which you upgrade individual nodes while the cluster remains running. Follow the instructions in Incremental Cluster Upgrade.
• Otherwise, you must shut down the entire cluster, upgrade each node, and then start up the new version of MemSQL on each node in the cluster. Follow the instructions in Total Cluster Shutdown and Restart.

In either approach, first deploy MemSQL Ops.

Deploying MemSQL Ops

1. If the old MemSQL Ops (from versions 3.2 and earlier) is installed: On the master aggregator, uninstall the old MemSQL Ops and collectd:

$ sudo service memsql-ops stop
$ sudo service collectd stop
$ sudo mv /var/lib/memsql-ops /var/lib/memsql-ops.bak

2. Deploy MemSQL Ops to the master aggregator as the primary agent.

$ VERSION="0.0.25" # Use the latest version number
$ wget http://download.memsql.com/memsql-ops-$VERSION/memsql-ops-$VERSION.tar.gz
$ tar xzf memsql-ops-$VERSION.tar.gz
$ sudo memsql-ops-$VERSION/install.sh --no-cluster

3. Use the primary agent to deploy agents to rest of cluster, either via web UI or, in a new shell, via the command line interface:

$ cp ~/.ssh/id_rsa /var/lib/memsql-ops
$ chmod a+r /var/lib/memsql-ops/id_rsa
$ memsql-ops agent-deploy -h <HOST_IP> -u <USER> -i /var/lib/memsql-ops/id_rsa

4. Add your license key:

$ memsql-ops license-add --license-key <license key>

Total Cluster Shutdown and Restart

Make sure that MemSQL Ops is deployed on all nodes in the cluster.

1. Shut down the master aggregator.

$ sudo service memsql stop

2. Shut down all child aggregator nodes.
3. Shut down all leaf nodes.
4. Upgrade all the nodes in the cluster according to the instructions in Upgrading a Node, in the following order: leaves, then master aggregator, then child aggregator nodes.
5. Verify that all leaves are online with SHOW LEAVES. If a leaf is detached then you can reattach it with ATTACH LEAF. If it is offline, then it is not reachable from the aggregator. See Managing High Availability for further details on leaf states.

Incremental Cluster Upgrade (Redundancy 2 Clusters Only)

For a cluster operating in Redundancy Level 2, you can upgrade individual nodes in the cluster while other nodes in the cluster are running. This setting is configured by doing either of the following:

• Set the redundancy_level system variable (redundancy_level = 2) (see Setting Redundancy Level ).
• In the memsql.cnf file of your master aggregator node, specify redundancy_level = 2 (see memsql.cnf Settings for details).

Step 1: Prepare for Upgrade

Make sure that MemSQL Ops is deployed on all nodes in the cluster.

On the master aggregator:

1. Disable auto-attach

memsql> SET GLOBAL auto_attach = OFF;

2. Disable leaf and aggregator failure detection

memsql> SET GLOBAL aggregator_failure_detection = OFF;
memsql> SET GLOBAL leaf_failure_detection = OFF;

Step 2: Upgrade Leaf Nodes

1. Begin by upgrading leaf nodes in the cluster. Determine in which order you want to upgrade the leaves - to avoid downtime, for each pair of leaves in the cluster you must update one host at a time (you can, however, upgrade all leaves in an availability group in parallel - refer to the example below). For each availability group (1, 2), and for each leaf in the availability group:

Run DETACH LEAF to detach the leaf from the cluster.

memsql> DETACH LEAF 'host':port;

2. Upgrade MemSQL and migrate the data according to the instructions in Upgrading a Node.
3. Run ATTACH LEAF to attach the leaf back to the cluster.

memsql> ATTACH LEAF 'host'[:port];

The instructions in this section use a scenario shown in the following illustration.

In this scenario, there are two availability groups:

• leaves L1 - L4 are in Availability Group 1
• leaves L5 - L8 are in Availability Group 2

There are four pairs of leaf nodes:

• L1 and L5
• L2 and L6
• L3 and L7
• L4 and L8

To upgrade these leaf nodes:

1. Detach leaf nodes L1, L2, L3, and L4.
2. Upgrade and start leaf nodes L1, L2, L3, and L4 according to the instructions in Upgrading a Node.
3. Attach leaf nodes L1, L2, L3, and L4.
4. Detach leaf nodes L5, L6, L7, and L8.
5. Upgrade and start leaf nodes L5, L6, L7, and L8. according to the instructions in Upgrading a Node. Attach leaf nodes L5, L6, L7, and L8 to the cluster.

Step 3: Upgrade the Master Aggregator Node

1. On all child aggregators, pause replicating all databases by running:

memsql> PAUSE REPLICATING <db>;

for each database <db> (except memsql and information_schema), for each child aggregator. You can run the following script from the MemSQL Ops primary agent:

for HOST in `memsql-ops agent-list --memsql-role aggregator -q`
do
  echo "Connecting to host $HOST"
  memsql-ops agent-ssh --agent-ids $HOST --no-prompt '
    for DB in `mysql -u root -h 127.0.0.1 --batch --skip-pager --skip-column-names --execute="SHOW DATABASES" | grep -vE "^(memsql|information_schema)$"`
    do
      echo " + Pausing replication for DB $DB"
      mysql -u root -h 127.0.0.1 --batch --execute "PAUSE REPLICATING \"$DB\""
    done
  '
done

2. Upgrade the master aggregator node according to the instructions in Upgrading a Node.

Step 4: Upgrade Child Aggregator Nodes

Step 5: Re-enable Auto Cluster Operations

On the master aggregator:

1. Enable auto-attach

memsql> SET GLOBAL auto_attach = ON;

2. Enable leaf and aggregator failure detection
3. Run:

memsql> SET GLOBAL aggregator_failure_detection = ON;
memsql> SET GLOBAL leaf_failure_detection = ON;

Upgrading a Node

To upgrade a node:

1. Stop (and backup) the MemSQL instance that will be upgraded

$ sudo service memsql stop

2. Deploy the new version of MemSQL via MemSQL Ops

$ memsql-ops memsql-deploy --role <leaf|master|aggregator>

After this is finished a new MemSQL instance will be installed into the MemSQL Ops installs folder, default /var/lib/memsql-ops/data/installs

3. Stop the new MemSQL instance

$ memsql-ops memsql-stop [<memsql_id>]

4. Move data from the old MemSQL instance into the new MemSQL instance, preserving the newly assigned id

$ cd $(memsql-ops memsql-path)
$ sudo mv ./data/memsql_id ./memsql_id
$ sudo mv ./data ./data.bak
$ sudo mv /var/lib/memsql/data ./
$ sudo mv ./memsql_id data/memsql_id

and if everything goes well, run:

$ sudo rm -rf ./data.bak

5. Copy the old config file

sudo cp /var/lib/memsql/memsql.cnf ./

6. Start the new MemSQL instance

$ memsql-ops memsql-start [<memsql_id>]

7. Verify that the node is running, then delete the old MemSQL binaries

$ ps aux | grep memsql
$ sudo rm -rf /var/lib/memsql