Outdated Version

You are viewing an older version of this section. View current production version.

Upgrading MemSQL from 3.2 and earlier min read


MemSQL Ops has been deprecated

Please follow this guide to learn how to migrate to SingleStore tools.

Alert

This page describes how to upgrade MemSQL from versions 3.2 and earlier. To upgrade from versions 4.0 and later, see Upgrading MemSQL.

Upgrading a Cluster

For a MemSQL cluster, the upgrade process involves individually upgrading each node in the cluster in a very specific sequence. There are two approaches:

  • If the cluster is running at Redundancy Level 2, you have the option to perform an incremental cluster upgrade, in which you upgrade individual nodes while the cluster remains running. Follow the instructions in Incremental Cluster Upgrade.
  • Otherwise, you must shut down the entire cluster, upgrade each node, and then start up the new version of MemSQL on each node in the cluster. Follow the instructions in Total Cluster Shutdown and Restart.

In either approach, first deploy MemSQL Ops.

Deploying MemSQL Ops

Info

These instructions are for upgrading MemSQL 3.2 and earlier, or a cluster that is not already managed by MemSQL Ops.

These instructions assume that the master aggregator will be MemSQL Ops primary agent.

  1. If the old MemSQL Ops (from versions 3.2 and earlier) is installed: On the master aggregator, uninstall the old MemSQL Ops and collectd:
$ sudo service memsql-ops stop
$ sudo service collectd stop
$ sudo mv /var/lib/memsql-ops /var/lib/memsql-ops.bak
  1. Deploy MemSQL Ops to the master aggregator as the primary agent.
$ VERSION="0.0.25" # Use the latest version number
$ wget http://download.memsql.com/memsql-ops-$VERSION/memsql-ops-$VERSION.tar.gz
$ tar xzf memsql-ops-$VERSION.tar.gz
$ sudo memsql-ops-$VERSION/install.sh --no-cluster
Info

Note: during the installation the current user will be added to the memsql group. Launch a new shell after the installation completes.

  1. Use the primary agent to deploy agents to rest of cluster, either via web UI or, in a new shell, via the command line interface:
$ cp ~/.ssh/id_rsa /var/lib/memsql-ops
$ chmod a+r /var/lib/memsql-ops/id_rsa
$ memsql-ops agent-deploy -h <HOST_IP> -u <USER> -i /var/lib/memsql-ops/id_rsa
Info

Note: memsql user needs read access to the SSH credentials file. This file will be copied into /var/lib/memsql-ops/data/keys.

  1. Add your license key:
$ memsql-ops license-add --license-key <license key>

Total Cluster Shutdown and Restart

Info

These instructions are for a MemSQL cluster that is not managed by MemSQL Ops 4 or later.

These instructions assume you have installed MemSQL using a Linux package manager. If you installed MemSQL from a binary, follow these steps but use the command syntax for a direct binary installation which can be found at Installing MemSQL on a Node.

Make sure that MemSQL Ops is deployed on all nodes in the cluster.

  1. Shut down the master aggregator.
$ sudo service memsql stop
  1. Shut down all child aggregator nodes.
  2. Shut down all leaf nodes.
  3. Upgrade all the nodes in the cluster according to the instructions in Upgrading a Node, in the following order: leaves, then master aggregator, then child aggregator nodes.
  4. Verify that all leaves are online with SHOW LEAVES. If a leaf is detached then you can reattach it with ATTACH LEAF. If it is offline, then it is not reachable from the aggregator. See Managing High Availability for further details on leaf states.

Incremental Cluster Upgrade (Redundancy 2 Clusters Only)

For a cluster operating in Redundancy Level 2, you can upgrade individual nodes in the cluster while other nodes in the cluster are running. This setting is configured by doing either of the following:

  • Set the redundancy_level system variable (redundancy_level = 2) (see Setting Redundancy Level ).
  • In the memsql.cnf file of your master aggregator node, specify redundancy_level = 2 (see memsql.cnf Settings for details).
Info

Follow the upgrade sequence exactly as described in this section: leaf nodes first, then master aggregator, and finally all child aggregator nodes.

This sequence is specific to upgrades from MemSQL 3.2 and earlier to MemSQL 4.0 and later.

Step 1: Prepare for Upgrade

Make sure that MemSQL Ops is deployed on all nodes in the cluster.

On the master aggregator:

  1. Disable auto-attach
memsql> SET GLOBAL auto_attach = OFF;
  1. Disable leaf and aggregator failure detection
memsql> SET GLOBAL aggregator_failure_detection = OFF;
memsql> SET GLOBAL leaf_failure_detection = OFF;

Step 2: Upgrade Leaf Nodes

Info

For each pair, you can upgrade only one node in the pair at a time.

  1. Begin by upgrading leaf nodes in the cluster. Determine in which order you want to upgrade the leaves - to avoid downtime, for each pair of leaves in the cluster you must update one host at a time (you can, however, upgrade all leaves in an availability group in parallel - refer to the example below). For each availability group (1, 2), and for each leaf in the availability group:

Run DETACH LEAF to detach the leaf from the cluster.

memsql> DETACH LEAF 'host':port;
  1. Upgrade MemSQL and migrate the data according to the instructions in Upgrading a Node.
  2. Run ATTACH LEAF to attach the leaf back to the cluster.
memsql> ATTACH LEAF 'host'[:port];
Info

It may take some time before the leaf recovered all partitions after upgrade and the leaf won’t be attached until fully recovered. To avoid downtime, all leaves in an availability group must be attached before proceeding with the upgrade of the next availability group.

The instructions in this section use a scenario shown in the following illustration. image

In this scenario, there are two availability groups:

  • leaves L1 - L4 are in Availability Group 1
  • leaves L5 - L8 are in Availability Group 2

There are four pairs of leaf nodes:

  • L1 and L5
  • L2 and L6
  • L3 and L7
  • L4 and L8

To upgrade these leaf nodes:

  1. Detach leaf nodes L1, L2, L3, and L4.
  2. Upgrade and start leaf nodes L1, L2, L3, and L4 according to the instructions in Upgrading a Node.
  3. Attach leaf nodes L1, L2, L3, and L4.
  4. Detach leaf nodes L5, L6, L7, and L8.
  5. Upgrade and start leaf nodes L5, L6, L7, and L8. according to the instructions in Upgrading a Node. Attach leaf nodes L5, L6, L7, and L8 to the cluster.

Step 3: Upgrade the Master Aggregator Node

Info

When upgrading the master aggregator, do not run DDL operations and DML operations against reference tables as well as database-level and cluster-level operations. Furthermore, child aggregators will not see any of these operations take affect until after they are upgraded. In short, avoid altering tables while upgrading your aggregators.

  1. On all child aggregators, pause replicating all databases by running:
memsql> PAUSE REPLICATING <db>;

for each database <db> (except memsql and information_schema), for each child aggregator. You can run the following script from the MemSQL Ops primary agent:

for HOST in `memsql-ops agent-list --memsql-role aggregator -q`
do
  echo "Connecting to host $HOST"
  memsql-ops agent-ssh --agent-ids $HOST --no-prompt '
    for DB in `mysql -u root -h 127.0.0.1 --batch --skip-pager --skip-column-names --execute="SHOW DATABASES" | grep -vE "^(memsql|information_schema)$"`
    do
      echo " + Pausing replication for DB $DB"
      mysql -u root -h 127.0.0.1 --batch --execute "PAUSE REPLICATING \"$DB\""
    done
  '
done
  1. Upgrade the master aggregator node according to the instructions in Upgrading a Node.

Step 4: Upgrade Child Aggregator Nodes

Info

When upgrading a child aggregator, do not run queries against that host. Furthermore, child aggregators will not see any of these operations take affect until after they are upgraded. Basically, don’t try to alter table while upgrading your aggregators

Step 5: Re-enable Auto Cluster Operations

On the master aggregator:

  1. Enable auto-attach
memsql> SET GLOBAL auto_attach = ON;
  1. Enable leaf and aggregator failure detection
  2. Run:
memsql> SET GLOBAL aggregator_failure_detection = ON;
memsql> SET GLOBAL leaf_failure_detection = ON;

Upgrading a Node

To upgrade a node:

  1. Stop (and backup) the MemSQL instance that will be upgraded
$ sudo service memsql stop
  1. Deploy the new version of MemSQL via MemSQL Ops
$ memsql-ops memsql-deploy --role <leaf|master|aggregator>

After this is finished a new MemSQL instance will be installed into the MemSQL Ops installs folder, default /var/lib/memsql-ops/data/installs

  1. Stop the new MemSQL instance
$ memsql-ops memsql-stop [<memsql_id>]
  1. Move data from the old MemSQL instance into the new MemSQL instance, preserving the newly assigned id
$ cd $(memsql-ops memsql-path)
$ sudo mv ./data/memsql_id ./memsql_id
$ sudo mv ./data ./data.bak
$ sudo mv /var/lib/memsql/data ./
$ sudo mv ./memsql_id data/memsql_id

and if everything goes well, run:

$ sudo rm -rf ./data.bak
  1. Copy the old config file
sudo cp /var/lib/memsql/memsql.cnf ./
  1. Start the new MemSQL instance
$ memsql-ops memsql-start [<memsql_id>]
  1. Verify that the node is running, then delete the old MemSQL binaries
$ ps aux | grep memsql
$ sudo rm -rf /var/lib/memsql