Capacity Limit Error
ERROR 2350: Leaf node could not be added because you are using the SingleStore DB free license which has a limit of 4 license units and after adding the node you would be using XX license units.
ERROR 2349: Code generation for new statements is disabled because the total number of license units of capacity used on all leaf nodes is XX, which is above the limit of 4 for the SingleStore DB free license.
Both errors 2350 and 2349 are caused because the total number of license units in your cluster is larger than the four unit limit imposed by the free license. You will encounter these errors when trying to add a new leaf node to the cluster, which can happen during initial deployment, or when expanding your cluster.
This error can occur when using MemSQL 6.7.13 or later with a unit-based free license. For previous RAM-based free licenses, a total memory limit of 128 GB of RAM for the cluster was used instead. See Memory Errors for more information on this limit.
See the FAQ for a definition of license units and how they are calculated.
There are two potential solutions depending on what your needs are:
If you need a cluster with more than four license units, you will need an Enterprise license. Sign up for a free 30-day trial and create an Enterprise License trial key before deploying a larger cluster.
You can artificially constrain resources by deploying SingleStore DB onto a host machine (physical or virtual) with fewer resources. See the FAQ link above for more details.
WARN: The ready queue has not decreased (currently
elements, pops) for seconds. This workload needs more threads.
Each query that runs on a cluster (including internal queries used by the nodes to communicate with each other) requires exactly one thread on an aggregator. This warning appears when the
max_connection_threads limit is reached on an aggregator and the aggregator stops responding.
The following is a brief description of each warning parameter:
elementsrefers to the number of queued queries (queries which are waiting for execution threads)
popsrefers to the cumulative number of queries that have been scheduled. If the value of
popsremains same, it means that the scheduler is not making progress because queries are still running.
This workload needs more threadsrefers to the kernel threads dedicated to SingleStore DB, specifically to the value of
Increase the value of
max_connection_threadson aggregator nodes (maximum 8192). The default value for aggregators is 192. Increasing the value of
max_connection_threadson aggregator nodes will allow more queries to run at the same time; however, this can increase resource pressure on the cluster.
Note: The value on leaf nodes does not need to be changed from the default of 8192, which is the maximum value.
The warning may also occur due to execution of
BACKUPis executed when there are long-running queries executing in the cluster, the backup operation will have to wait for the long-running queries to complete. During this time new queries executed on the cluster will also have to wait for the long-running query to finish. If this warning appears in close proximity to
BACKUPentries in the master aggregator’s memsql.log, check to make sure backups are not being executed when long-running queries are present in the cluster.
Monitor Thread Utilization
SHOW STATUS EXTENDED command to check the current number of running threads (threads_running) and number of connections in queue currently waiting (ready_queue).
If a workload runs enough long-running concurrent queries that reaches the
max_connection_threads threshold, and the queries are sufficiently long-running so that none have finished for over few seconds, the aggregator’s memsql.log file will have the
WARN: This workload needs more threads. entry.