When running SingleStore DB on machines with multiple Non-Uniform Memory Access (NUMA) nodes, you should configure SingleStore DB for NUMA with numactl for optimal performance. If you do not configure SingleStore DB this way, performance will be greatly degraded due to expensive cross-NUMA-node memory access. Configuring for NUMA should be done as part of the installation process; however, you can reconfigure your deployment later, if necessary.

To simplify the configuration process, MemSQL Ops can automatically detect if a host machine has multiple NUMA nodes and then configure SingleStore DB with numactl to bind individual SingleStore nodes to NUMA nodes.

Configuring a new cluster for NUMA

Before binding NUMA nodes to specific leaf nodes, you must install numactl.

1. On each host, install the numactl package. For example, for Debian-based OSes:

   sudo apt-get install numactl
   

   For Redhat/CentOS, run the following:

   sudo yum install numactl
   

2. Check the number of NUMA nodes your machines by running numactl --hardware. For example:

   numactl --hardware
   available: 2 nodes (0-1)
   

   The output shows that there are 2 NUMA nodes on this machine, numbered 0 and 1.

Configuring for NUMA through the Ops web interface

If you are managing your cluster using MemSQL Ops, install Ops agents on host machines through the Ops web interface. After the Ops agent is installed on each host machine, you can install SingleStore nodes through the UI. The suggested leaf node count should align to the NUMA node count on each machine. Make sure to install one leaf node per NUMA node.

MemSQL Ops will then configure the SingleStore nodes appropriately for NUMA on each machine during deployment.

Configuring for NUMA through the Ops CLI

1. On each host that will contain leaf nodes, create one leaf per NUMA node by running memsql-ops memsql-deploy until you have the same number of leaf nodes as NUMA nodes. You will need to choose a different --port for each SingleStore node on the same host.

   For example, on a host machine with two NUMA nodes, you would deploy two leaf nodes.

   memsql-ops memsql-deploy -r leaf --port 3307
   memsql-ops memsql-deploy -r leaf --port 3308
   

2. Run memsql-ops memsql-optimize and confirm the prompt. This will configure numactl to bind SingleStore nodes to NUMA nodes, and configure SingleStore DB memory settings. If you have fewer leaf nodes on a machine than the number of NUMA nodes, the memsql-optimize command will display a message like:

   You should deploy 1 more leaf on this machine in order to maximize performance.

   See MEMSQL-OPTIMIZE for more information.

3. Restart the cluster for the configuration changes to take effect.

   memsql-ops memsql-restart --all
   

Reconfiguring an existing cluster for NUMA

1. Follow the steps in the previous section to deploy and optimize your leaf nodes.

2. Rebalance all SingleStore DB databases by running REBALANCE PARTITIONS on each database:

   REBALANCE PARTITIONS ON <database>
   

Was this Article Helpful?

Thanks For Your Feedback!