When running MemSQL on machines with multiple Non-Uniform Memory Access (NUMA) nodes, you should configure MemSQL for NUMA with numactl
for optimal performance. If you do not configure MemSQL this way, performance will be greatly degraded due to expensive cross-NUMA-node memory access. Configuring for NUMA should be done as part of the installation process; however, you can reconfigure your deployment later, if necessary.
To simplify the configuration process, MemSQL Ops automatically detects if a host machine has multiple NUMA nodes and can configure MemSQL with numactl
to bind individual MemSQL nodes to NUMA nodes. The following sections show you how to do this through both the MemSQL Ops UI and the command-line interface. Note: The steps in this topic assume you have already downloaded and installed MemSQL Ops onto a host machine. If you have not done so already, install MemSQL Ops before proceeding.
Configuring for NUMA when installing MemSQL through the Ops web interface
-
Log into each host and install the
numactl
package. For example, for Debian-based OSes:$ sudo apt-get install numactl
For Redhat/CentOS, run the following:
$ sudo yum install numactl
-
Check the number of NUMA nodes your machines by running
numactl --hardware
. For example:$ numactl --hardware available: 2 nodes (0-1)
The output shows that there are 2 NUMA nodes on this machine, numbered 0 and 1.
-
Install MemSQL Ops on host machines through the Ops web interface. After the Ops agent is installed on each host machine, you can install MemSQL nodes through the UI. The suggested leaf node count should align to the NUMA node count on each machine. Make sure to install one MemSQL leaf node per NUMA node.
MemSQL Ops will then configure the MemSQL nodes appropriately for NUMA on each machine during deployment.
Configuring for NUMA when installing MemSQL through the Ops command line interface
-
On each host, install the
numactl
package. For example, for Debian-based OSes:$ sudo apt-get install numactl
For Redhat/CentOS, run the following:
$ sudo yum install numactl
-
Check the number of NUMA nodes your machines by running
numactl --hardware
. For example:$ numactl --hardware available: 2 nodes (0-1)
The output shows that there are 2 NUMA nodes on this machine, numbered 0 and 1.
-
On each host that will contain MemSQL leaf nodes, deploy one leaf node per NUMA node by running
memsql-ops memsql-deploy
until you have the same number of leaf nodes as NUMA nodes. You will need to choose a different--port
for each MemSQL node on the same host.For example, on a host machine with two NUMA nodes, you would deploy two leaf nodes.
$ memsql-ops memsql-deploy -r leaf --port 3307 $ memsql-ops memsql-deploy -r leaf --port 3308
-
Run
memsql-ops memsql-optimize
and confirm yes at the prompt.Do you want MemSQL Ops to take these steps? [Y/n]
This will configure
numactl
to bind MemSQL nodes to NUMA nodes, and configure MemSQL memory settings. If you have fewer leaf nodes on a machine than the number of NUMA nodes,memsql-optimize
will display a message like:You should deploy 1 more leaf on this machine in order to maximize performance.
See MEMSQL-OPTIMIZE for more information on this command.
-
Restart the cluster for the configuration changes to take effect:
memsql-ops memsql-restart --all
Reconfiguring an existing MemSQL cluster for NUMA
-
Follow the steps in the previous section to deploy and optimize your leaf nodes.
-
Rebalance all MemSQL databases by running REBALANCE PARTITIONS on each database:
REBALANCE PARTITIONS ON <database>