Outdated Version

You are viewing an older version of this section. View current production version.

Upgrading MemSQL min read


MemSQL Ops has been deprecated

Please follow this guide to learn how to migrate to SingleStore tools.

This topic describes how to upgrade MemSQL from an earlier version. The upgrade process depends on the version from which you are upgrading.

Warning

To upgrade MemSQL from versions 3.2 and earlier, see Upgrading MemSQL from 3.2 and earlier.

This page describes how to upgrade MemSQL from versions 4.0 and later. This page assumes the MemSQL cluster is managed by MemSQL Ops.

Step 1: Upgrade MemSQL Ops

Execute the following command:

$ memsql-ops agent-upgrade

If necessary, add your license key (see Replacing a MemSQL License).

There are two approaches to upgrading a MemSQL cluster:

  • The simplest upgrade process is to shut down the entire cluster, upgrade each node, and then start up the new version of MemSQL on each node in the cluster. Follow the instructions in Total Cluster Shutdown and Restart.
  • If the cluster is running with High Availability, you also have the option to perform an incremental cluster upgrade, in which you upgrade individual nodes while the cluster remains running. Follow the instructions in Incremental Cluster Upgrade.

Step 2: Upgrade MemSQL

There are two approaches to upgrading a MemSQL cluster:

Option 1: Offline upgrade

Note: This upgrade method is referred to as an “offline” upgrade because your existing MemSQL cluster will be shut down and restarted over the course of the upgrade. You do not need to shut down your cluster prior to starting the upgrade.

Execute the following command to start an offline upgrade:

$ memsql-ops memsql-upgrade [--version <VERSION>]

This command will shut down the entire cluster. For more information, see the full documentation for the command MEMSQL-UPGRADE.

If the cluster is running with High Availability, you also have the option to perform an Incremental Cluster Upgrade.

Option 2: Online upgrade (High Availability clusters only)

For a cluster operating in High Availability (Redundancy Level 2), you can upgrade individual nodes in the cluster while other nodes in the cluster are running.

Ensure your cluster is running with High Availability (Redundancy Level 2). The redundancy_level variable should be 2.

memsql> show variables like 'redundancy_level';
+------------------+-------+
| Variable_name    | Value |
+------------------+-------+
| redundancy_level | 2     |
+------------------+-------+
Info

Follow the upgrade sequence exactly as described in this section: leaf nodes first, then all child aggregator nodes, and finally the master aggregator.

This procedure is NOT safe for upgrades from MemSQL 3.2 and earlier to MemSQL 4.0 and later. For that, see Upgrading MemSQL from 3.2 and earlier.

Part 1: Prepare for Upgrade

  1. (Optional) Prior to upgrading, you may run SNAPSHOT DATABASE on any databases, which compacts the transaction logs, reducing database recovery time on startup.

  2. Enable manual control in MemSQL Ops: In the web interface, go to Settings > Manual Cluster Control and check “Enable Manual Control”. Or with the command line interface:

$ memsql-ops cluster-manual-control --enable

(If you don’t do this, MemSQL Ops will attempt to restart leaves after you bring them down for the upgrade.)

  1. On the master aggregator, disable auto-attach and node failure detection:
memsql> SET GLOBAL auto_attach = OFF;
memsql> SET GLOBAL aggregator_failure_detection = OFF;
memsql> SET GLOBAL leaf_failure_detection = OFF;

Part 2: Upgrade Leaf Nodes

Info

For each pair, you can upgrade only one node in the pair at a time.

Determine which availability groups each of your leaves are in with SHOW LEAVES.

Determine in which order you want to upgrade the leaves. To avoid downtime, for each pair of leaves in the cluster you must update one node at a time. You can, however, upgrade all leaves in an availability group in parallel.

To keep the procedure simple, you can upgrade all leaf nodes in availability group 1, and then all leaf nodes in availability group 2. You may upgrade all the nodes in an availability group in parallel as desired. You can also upgrade leaf nodes in a different order - you do not have to upgrade all nodes in one availability group before the other - as long as you ensure at least one leaf node in each pair is online at all times.

Upgrading a leaf node

To upgrade each leaf node:

  1. This step is only necessary for upgrades from versions 4.1 and earlier; it is not necessary for upgrades from versions 5.0 and later. Check if any master partitions on the leaf are far ahead of their replicas. See SHOW REPLICATION STATUS. If yes, wait for thereplicas to catch up. Note in particular that this means between upgrading the first leaf in a pair (A) and the second (B), you must wait for A, the newly upgraded leaf (which is currently a replica) to catch up replicating from B, its replication master which we are about to upgrade, on any newly written data while it was offline.

  2. Run DETACH LEAF from the Master Aggregator to detach the leaf from the cluster.

memsql> DETACH LEAF 'host'[:port];
  1. Upgrade the MemSQL leaf node according to the instructions in the Upgrading a Node section.

  2. Re-attach the leaf back to the cluster by running ATTACH LEAF from the Master Aggregator. Wait for the attach to complete and the leaf to move to the online status.

memsql> ATTACH LEAF 'host'[:port];
Info

It may take some time before the leaf recovers all partitions after upgrade and the leaf won’t be attached until fully recovered. To avoid downtime, all leaves in an availability group must be attached before proceeding with the upgrade of the next availability group.

Example

For example: assume we have the following cluster configuration, with 8 leaf nodes in redundancy 2:

There are two availability groups:

  • Leaves L1 - L4 are in Availability Group 1
  • Leaves L5 - L8 are in Availability Group 2

There are four pairs of leaf nodes:

  • L1 and L5
  • L2 and L6
  • L3 and L7
  • L4 and L8

image

To upgrade these leaf nodes:

  1. Upgrade Availability Group 1: leaf nodes L1, L2, L3, and L4
  2. Detach leaf nodes L1, L2, L3, and L4
  3. Upgrade and start leaf nodes L1, L2, L3, and L4 according to the instructions in the Upgrading a Node section
  4. Attach leaf nodes L1, L2, L3, and L4
  5. Upgrade Availability Group 2: leaf nodes L5, L6, L7, and L8
  6. Detach leaf nodes L5, L6, L7, and L8
  7. Upgrade and start leaf nodes L5, L6, L7, and L8 according to the instructions in the Upgrading a Node section
  8. Attach leaf nodes L5, L6, L7, and L8 to the cluster.

Part 3: Upgrade Child Aggregator Nodes

Upgrade each child aggregator node according to the instructions in the Upgrading a Node section.

You may upgrade child aggregators in parallel if desired. Keep enough aggregators online at any given time to ensure you are able to continue running client queries smoothly.

Info

When upgrading a child aggregator, you will be unable to run queries against that node. Furthermore, it will not see any operations take affect until it is upgraded, so to minimize risks, avoid running DDL operations such as ALTER TABLE while upgrading any of your aggregators.

Part 4: Upgrade the Master Aggregator Node

Upgrade the master aggregator node according to the instructions in the Upgrading a Node section.

Info

When upgrading the master aggregator, you will be unable to run queries against that node. This means you will be unable to run DDL operations, DML operations against reference tables, as well as database-level and cluster-level operations.

Part 5: Re-enable Auto Cluster Operations

Restore the auto-cluster operations settings to their initial state.

  1. On the master aggregator, re-enable auto-attach and node failure detection:
memsql> SET GLOBAL auto_attach = ON;
memsql> SET GLOBAL aggregator_failure_detection = ON;
memsql> SET GLOBAL leaf_failure_detection = ON;
  1. Disable manual control in MemSQL Ops: In the web interface, go to Settings > Manual Cluster Control and uncheck “Enable Manual Control”. Or with the command line interface:
$ memsql-ops cluster-manual-control --disable

This completes the upgrade procedure.

Upgrading a Node

Follow these instructions to upgrade an individual node within the above procedure.

  1. Stop the MemSQL node:
$ memsql-ops memsql-stop [memsql_id]
  1. Unmonitor the node in MemSQL Ops:
$ memsql-ops memsql-unmonitor [memsql_id]
  1. Move the entire MemSQL installation directory elsewhere. This is usually located at a path like /var/lib/memsql/leaf-3306/. For example:
$ sudo mv $(memsql-ops memsql-path [memsql_id]) somepath/memsql.old

You should also make a backup copy of the data directory.

  1. Deploy the new version of MemSQL with MemSQL Ops. Specify the agent and port appropriately as needed.
$ memsql-ops memsql-deploy --role <leaf|master|aggregator>

After this is finished a new MemSQL instance will be installed, by default located at a path like /var/lib/memsql/leaf-3306/. It will not be started automatically because manual control is on. Leave it stopped.

  1. Move the data/ directory from the old MemSQL installation into the new MemSQL installation, but preserve the new data/memsql_id file. Also copy the files memsql.cnf and lib/license from the old MemSQL installation into the new MemSQL installation. Make sure all files are owned by the memsql user. For example:
$ cd $(memsql-ops memsql-path [memsql_id])
$ sudo mv ./data/memsql_id ./memsql_id
$ sudo mv ./data somepath/data.newemptyinstall.bak
$ sudo mv somepath/memsql.old/data ./
$ sudo mv ./memsql_id data/memsql_id
$ sudo cp somepath/memsql.old/memsql.cnf ./
$ sudo cp somepath/memsql.old/lib/license ./lib/license
  1. Start the new MemSQL instance
$ memsql-ops memsql-start [memsql_id]
  1. Wait for the new MemSQL instance to recover. You can check whether it is recovering by running show databases extended and seeing if any row has State = ‘recovering’.

  2. Verify that the node is online. If everything goes well, clean up the old files

$ sudo rm -rf somepath/memsql.old/
$ sudo rm -rf somepath/data.newemptyinstall.bak