SingleStore DB Toolbox Release Notes

2022-06-06 version 1.13.8

• Added a new informationSchemaMvReplicationManagement collector to the sdb-report collect command that runs SELECT * for the following tables:
  • MV_REPLICATION_MANAGEMENT
  • MV_REPLICATION_MANAGEMENT_PARTITIONS_STATE
  • MV_REPLICATION_MANAGEMENT_REF_DB_STATE
  • MV_REPLICATION_MANAGEMENT_STATE

2022-05-10 version 1.13.7

• Fixed a bug where the sdb-report calibrate-blob-storage command would hang when all blobs were not successfully evicted
• Added retrying an online upgrade when the hit error preparing for failover error is encountered
• Updated Toolbox to only request sudo credentials when more privileges are required than provided by the default memsql user
• Changed the order of SQL statements in the output from the sdb-admin dump command

2022-03-28 version 1.13.6

• Added the following timestamp flags to the sdb-report collect command that allow logs to be collected starting from the provided timestamp:
  • For audit logs: memsqlAuditlogs.auditLogStartTimestamp
  • For command logs: memsqlCommandLogs.commandLogStartTimestamp
  • For tracelogs: memsqlTracelogs.tracelogStartTimestamp
• Fixed a bug in the installedPermissions checker where the sdb-repot check command would incorrectly report a failure
• Updated the sdb-admin unregister-node and sdb-deploy setup-cluster commands to display more detailed messages should an error occur

2022-03-02 version 1.13.5

• Fixed a validation bug in the sdb-admin register-node command when configuring a node
• Fixed the partition count in the diagnostic output of the sdb-report calibrate command
• Fixed permission issues for the sdb-report calibrate-blob-storage command
• Added the --skip-validation option to the sdb-admin restore-backup command to skip validating backups before restoring
• Updated incremental backup validation for the sdb-admin restore-backup and sdb-admin validate-backup commands
• Added a tracelogHardShutdown checker that checks for hard shutdowns in a node’s tracelog

2022-02-03 version 1.13.4

• Added the sdb-admin enable-http-api command to enable the HTTP API on selected nodes
• Added a new --role option to the sdb-admin update-config command. This option allows updating the configuration on all nodes that have selected role.
• Changed the sdb-toolbox-config describe-host command to prompt interactively for a hostname
• Made the online upgrade more reliable by making multiple attempts to re-attach a leaf node on account of a failure

2022-01-20 version 1.13.3

• Added a new --skip-system-warnings option to the sdb-admin start-node, sdb-admin restart-node, and sdb-deploy upgrade commands. This option allows users to skip displaying the extended version of system configuration limitations and warnings.
• Fixed a data file-permissions issue in the sdb-report calibrate-blob-storage command
• Fixed the TSV files generated by the sdb-report collect commands to escape newlines, tabs, and slashes when saving query results
• By default, the sdb-deploy setup-cluster command now configures the host’s operating system per SingleStore’s recommendations. To disable this auto-configuration, use the new skip-auto-config option in the YAML configuration file. The --skip-auto-config option has also been added to the sdb-deploy cluster-in-a-box command.

2021-12-20 version 1.13.2

• Updated the sdb-deploy setup-cluster and sdb-deploy cluster-in-a-box commands to allow the --preinstalled-path option to be used with Debian and Red Hat package installations
• Expanded the output of the sdb-admin start-node command with warnings about system configurations and limits
• Fixed the output of the sdb-admin update-config command that is returned when the command is run for multiple hosts or for updating global variables. This fix applies to SingleStore DB versions 7.3.22 and later, 7.5.15, 7.1.25 and later, and 7.6.7 and later
• Improved the Toolbox monitoring commands by optimizing purge_blobs and purge_metrics procedures
• Updated the mv_queries.query_text column’s type to LONGTEXT
• Added a new --create-service-only option to the sdb-admin enable-nodes-autostart command that allows the command to be run without prompting for a root password; this option creates a memsql.service file on the specified hosts without enabling the service
• Added a new informationSchemaMvBlockedQueries collector to the sdb-report commands that runs the SELECT * FROM information_schema.MV_BLOCKED_QUERIES query on the Master Aggregator
• Added a new informationSchemaMvBottomless collector to the sdb-report commands that saves data from the following tables:
  • MV_BOTTOMLESS_API_EVENTS
  • MV_BOTTOMLESS_REMOTE_STATS
  • MV_BOTTOMLESS_STATUS
  • MV_BOTTOMLESS_STATUS_EXTENDED
  • MV_BOTTOMLESS_SUMMARY
  • MV_MILESTONES

2021-12-06 version 1.13.1

• Added support for using the rsync command to copy files on a remote host if scp is not found
• Fixed the sdb-admin optimize command to work for SingleStore DB versions 7.5.13, 7.5.14, 7.6.4, and 7.6.5
• Updated the --preinstalled-path option in the sdb-deploy upgrade command to work with Debian and Red Hat package installations
• Updated the pre-installation checks of the sdb-deploy upgrade command to additionally verify that the folder specified by the --preinstalled-path option meets the prerequisites

2021-11-23 version 1.13.0

• Added support for starting each node as a separate systemd unit via systemctl. To perform this action, systemctl uses an sdb-systemd-client script, which can be distributed across the cluster using the sdb-admin enable-systemd-client command
• Updated the --password flag in sdb-admin change-root-password and sdb-admin create-node commands to support the MEMSQL_PASSWORD environment variable
• Fixed the sdb-admin update-config command to work for SingleStore DB versions7.6.4 and 7.5.13, however, without the support for extended output
• Improved the error message caused by running sdb-deploy commands with the --preinstalled-path option in Debian and RPM package installations

2021-11-08 version 1.12.5

• Fixed an error with calling a nil function in the ​sdb-deploy​​ commands
• Updated the ​sdb-deploy install​ command ​not​ to mark the new version of SingleStore DB as active in tarball-based installations. To install a new version and make it the active version, upgrade the cluster by running the ​sdb-deploy upgrade --preinstall-path​​ command. In Debian and RPM package installations, only one version of SingleStore DB can be installed at any given time
• Fixed an issue with creating incremental backups on Network File System (mounted on a remote machine) when the target file does not exist
• Improved the Toolbox monitoring commands by optimizing ​purge_blobs​ and ​purge_metrics​ procedures; updated the ​mv_queries.query_text​ column’s type to ​LONGTEXT​​
• Updated the ​sdb-deploy upgrade​​ command to retry upgrade if the process fails during partition promotion
• Added a new ​sdb-admin show-cluster​ command that outputs the results of both ​sdb-admin show-leaves​ and ​sdb-admin show-aggregators​​ commands. This command can be run on any node in the cluster if the Master Aggregator is not available. This command works with SingleStore DB version 7.6 and above

2021-10-25 version 1.12.4

• Fixed the default value of the pipelines_extractor_get_offsets_timeout_ms variable in the defaultVariables checker of the sdb-report check command
• Updated the sdb-admin update-config command to restrict users from setting specific variables on node startup

2021-10-12 version 1.12.3

• Added SAML and JWT authentication methods to the --auth-method option of the sdb-admin sync-ldap command and a new --saml-user-domain-attribute flag. SAML requires either the --user-principal-name-attribute or the --saml-user-domain-attribute specification for successful authentication. For more information, see sdb-admin sync-ldap command reference.

2021-09-27 version 1.12.2

• Added a new --precheck-only flag to the sdb-admin enable-high-availability command to verify that high availability can be enabled on a cluster
• Added a force_registration: <true | false> option to the node section of a cluster configuration YAML file for use with the sdb-deploy setup-cluster command
• Added a new --temp-dir flag to the sdb-report calibrate command to specify the absolute path on the Master Aggregator in which to extract the dataset
• Improved the sdb-report command to verify that the command is run by a memsql group member. If so, the report collects data that is otherwise restricted. If not, a warning is displayed.
• Updated the sdb-deploy upgrade command to remove the prior memsql-server/singlestorerdb-server version package after a successful upgrade. Use the new --skip-uninstall flag to skip this step.
• Fixed a file-permissions issue in the sdb-deploy setup-cluster command that would sometimes cause the command to fail when run against large, frequently-accessed databases

2021-09-13 version 1.12.1

• By default, the sdb-toolbox-config register-host command now configures the host’s operating system per SingleStore’s recommendations. To disable this option, use the new skip-auto-config flag
• Added support for verifying the length of the path to the runtime directory to prevent SSH issues due to long socket paths
• Added new informationSchemaMvAggregatedReplicationStatus and replicationLag collectors to report when the replication on the secondary cluster is out of sync with the primary cluster
• Improved the Toolbox command outputs to hide user passwords in verbose mode

2021-08-30 version 1.12.0

• Added calibrate-blob-storage to the sdb-report command to measure blob storage performance
• Extended the output of the sdb-admin update-config command to better integrate with automation processes
• Improved the performance of permissions checks for the sdb-deploy setup-cluster command
• Extended the defaultVariables checker to check for all documented variables
• Fixed a bug in the sdb-deploy upgrade command that would not properly set the toolbox_version variable on all nodes if an offline upgrade failed

2021-08-16 version 1.11.11

• Improved sdb-deploy cluster-in-a-box to optimize the cluster with respect to the NUMA configuration of the host
• Added the ability to limit the number of rows collected by the informationSchemaPipelines collector
• Improved sdb-admin restart-node command to not require the --all flag when performing an online restart
• Improved the output of the environment validation checks in the sdb-deploy setup-cluster command
• Expanded the output from the memsqlctl command in the sdb-admin optimize command
• Fixed a bug in the sdb-deploy upgrade command that may have led to issues with an offline upgrade following a failed online upgrade

2021-08-02 version 1.11.10

• Improved the sdb-admin create-backup command to create missing directories specified in the location of file system backups
• Updated the sdb-report collect command to allow optional compression of node log files
• Fixed the sdb-report collect-and-check --include-performance command to run performance-related collectors and checkers in addition to the default ones
• Fixed a bug in the sdb-report check command that would cause invalid decompression of some files in cluster diagnostics reports

2021-07-19 version 1.11.9

• Added a new dataSkewness collector to collect information about data skew in the cluster
• Fixed a bug in the sdb-report send-alert command that would cause some collectors to become unavailable
• Updated the sdb-deploy cluster-in-a-box command to search for free ports
• Improved the sdb-report collect-and-check command to use the values defined by --remote-temp-dir and --temp-dir flags

2021-07-05 version 1.11.8

• Added a new studioVersion collector to collect information about the SingleStore DB Studio version
• Updated the sdb-report collect command to compress node log files before adding them to a report
• Updated the sdb-admin start-monitoring command to set retention period and purge frequency on individual monitored hosts
• Fixed a pre-installation check in the sdb-deploy setup-cluster command that would cause an error in certain incremental cluster setups

2021-06-21 version 1.11.7

• Added pre-installation checks to sdb-admin add-leaf and sdb-admin add-aggregator commands. The checks can be skipped using a new --skip-validate-env flag
• Added support for creating incremental backups to the sdb-admin create-backup command
• Added support for incremental backup restoration to the sdb-admin restore-backup command. Specific versions of incremental backups can be restored via the --incr-backup flag
• Added support for deleting incremental backups to the sdb-admin delete-backup command
• Fixed the sdb-admin delete-backup command to clean up .manifest files
• Improved the JSON output of the sdb-report check command
• Updated the sdb-deploy upgrade command to allow users to retry the upgrade steps upon the occurrence of specific errors

2021-06-07 version 1.11.6

• Changed the output of the sdb-admin list-backups command to include incremental backups
• Updated the sdb-admin validate-backup command to validate incremental backups
• Added the --incr-backup-id flag to the sdb-admin validate-backup command to validate specific versions of an incremental backup
• During an offline upgrade of SingleStore DB 7.0 and later versions via the sdb-deploy upgrade command, the leaf_failure_detection engine variable will no longer be set to OFF
• Improved the description of the sdb-admin optimize command

2021-05-24 version 1.11.5

• Removed the redundant output from the sdb-deploy ui command
• Added support for installing singlestoredb-server packages via the sdb-deploy command
• Updated the --help flag in specific Toolbox commands to display the minimum version of SingleStore DB required to run the commands
• Changed the sdb-admin start-monitoring command to perform query sampling by default. A new disable-sample-queries flag has been added to this command to disable query sampling
• Improved the sdb-report collect-kube command to support in-cluster configurations
• Fixed a bug in the sdb-deploy setup-cluster command that would sometimes cause the command to crash when running pre-installation checks

2021-05-10 version 1.11.4

• Updated the numaConfiguration checker to better handle multiple NUMA nodes per host
• Fixed the transparentHugepage checker to display the correct output when transparent huge pages are enabled and /sys/kernel/mm/transparent_hugepage/defrag is set to madvise
• Added the toolboxEnv collector to gather information about the Toolbox configuration
• Fixed a bug in the sdb-report collect --merge-path command that would result in either an inconsistent directory structure or a failure when merging reports from remote hosts
• Added the sdb-admin update-node-basedir command to move the base directory of a node and update the corresponding engine variables
• Fixed a bug in the sdb-report collect command that would cause the command to crash if a node was offline
• Added the nodeDirectoriesDiskUsage collector to collect disk usage information for each node directory
• Improved the execution time of informationSchemaTableStatistics, informationSchemaIndexStatistics and informationSchemaMvQueries collectors

2021-04-26 version 1.11.3

• Added the sdb-admin disable-nodes-autostart command to disable the use of systemd for restarting all nodes on specified hosts at startup
• Added an ability to create backup using WITH SPLIT PARTITIONS in the sdb-admin create-backup command
• Fixed the installedPermissions collector that would cause a crash if system information is unavailable for some files
• Added a snapshot retry when a database connection is not established
• Added the --rename-to flag to the sdb-report restore-backup command to rename a database while it is restored
• Fixed a bug in the minFreeKbytes checker that would cause the checker to use an incorrect threshold
• Updated the sdb-deploy command to prevent the direct upgrade from SingleStore DB version 6.x to 7.5 and later
• Updated the sdb-deploy setup-cluster command to set the skip_install property to true without specifying the package_type property in the cluster file
• Improved the sdb-report collect-kube command to discover Kube services

2021-04-12 version 1.11.2

• Fixed a bug in the sdb-admin enable-high-availability command that would cause it to fail for clusters running memsql-server versions earlier than 7.1
• Improved the description of the --reset flag in the sdb-admin update-config command
• Improved the execution time of the sdb-admin delete-node and the sdb-admin sync-ldap commands

2021-03-29 version 1.11.1

• Added the sdb-admin dump command to dump cluster objects into a DDL file similar to the usage of mysqldump
• Added OpenSSL FIPS support for Toolbox via the --ssl-fips-mode flag in the sdb-admin create-node and the sdb-admin register-node commands, and via the ssl_fips_mode cluster file property in the sdb-deploy setup-cluster command
• Added the --load-balanced flag and the load_balanced cluster file property to the sdb-deploy setup-cluster command to allow clusters to be set up with high availability in the load_balanced mode
• Replaced the leafPairs checker with a highAvailability checker in the sdb-report command that determines the fidelity of high-availability clusters in both paired and load_balanced modes
• Changed the cpuHyperThreading checker in the sdb-report command to not report stderr when it only contains whitespace characters, which is considered empty
• Fixed a bug in the sdb-admin enable-high-availability command that would cause leaf nodes to be paired incorrectly

2021-03-15 version 1.11.0

• Added the sdb-report calibrate command to calibrate a cluster’s performance against a benchmark using a variety of queries
• Added the whitespacesInObjectName checker to the sdb-report command to check for objects in the schema that have leading or trailing whitespaces in their names
• Added the --temp-dir flag to the sdb-report collect and the sdb-report collect-and-check commands to select a directory for storing temporary files on both remote and local hosts. The --remote-temp-dir flag is now deprecated.
• Added a new option to specify a password while executing the sdb-admin configure-monitoring command; the password can now be specified via the EXPORTER_PASSWORD environmental variable in addition to the --exporter-password flag
• Added a check to verify that a node is registered in Toolbox and runs the same version as the Master Aggregator when the node is added as a leaf or an aggregator.
• Added the --load-balanced flag to the sdb-admin enable-high-availability command to allow high availability to be enabled in the load_balanced mode
• Enabled the --mask-ip flag in the sdb-report collect-kube command
• Removed an unused field from the Toolbox state file
• Fixed a bug in the sdb-report check command that would cause the duplicatePartitionDatabases checker to return incorrect output in some cases

2021-03-01 version 1.10.1

• Added Implementation and LockfreeTimestamp fields to the backup cache
• Improved the sdb-deploy setup-cluster command to skip the root password prompt if each node has a password already set
• Improved the dmesg collector in the sdb-report collect command to also look for the dmesg executable in /usr/bin, /bin, and /usr/local/bin directories
• Changed the informationSchemaProcesslist collector in the sdb-report collect command to mask user credentials in the output
• Changed the leafAverageRoundtripLatency checker in the sdb-report check command to apply stricter criteria while detecting outliers
• Fixed a bug in the sdb-admin list-backups command that would cause a crash
• Fixed a bug in the sdb-admin start-monitoring command that would cause timestamps in the mv_events table to be invalid

2021-02-15 version 1.10.0

• Added the --kerberos-realm flag to the sdb-admin sync-ldap command to override the Kerberos realm value, which is used to create new SQL users with the --auth-method kerberos flag
• Improved the sdb-admin describe-host command to display the memsqlctl path and the memsqlctl config path correctly for installations that are not tarball-based
• Changed the informationSchemaUserPrivileges collector to no longer run during environment validation
• Fixed a bug in the sdb-report collect --mask-ip command that would cause an error when the command is run by a non-member of the memsql group

2021-02-01 version 1.9.7

• Added the sdb-report collect-kube command to build a diagnostics report for a Kubernetes cluster
• Added the cgroupInfo collector and the cgroupMemoryUsage checker to the sdb-report command to check if the memsqld process runs inside the memory cgroup and that memory usage is not too close to the limit
• Added an ability to set availability groups in the sdb-deploy ui command
• Added an ability to set a memsql-server .tar file as an installation target in the sdb-deploy ui command
• Improved the output of the leafAverageRoundtripLatency checker in the sdb-report check command to pinpoint the nodes that have the longest round-trip latency
• Improved the error messages caused by running Toolbox with sudo
• Improved error messages in the sdb-deploy upgrade command
• Changed all Toolbox commands to no longer require a special configuration to run as the root user directly and without using sudo
• Changed the validation performed by the sdb-deploy setup-cluster command to be less strict for the memsql directory on each host
• Changed the row timeout of some queries used in the sdb-deploy upgrade command from 1 minute to 15 minutes
• Fixed the Toolbox auto-complete feature for bash and zsh so that it will work with both memsql-* and sdb-* commands
• Added taking database(s) snapshots when using Toolbox to stop or restart a node to decrease the time it takes for the node to recover

2021-01-19 version 1.9.6

• Added the sdb-admin snapshot-database command to create database snapshots
• Added the --snapshot to the sdb-admin restart-node and the sdb-admin stop-node commands to create snapshots of databases before restarting or stopping a node
• Added the contents of the memsqlctl.hcl and the nodes.hcl files to the output of the memsqlcltInfo collector
• Added the ServerVersion and the MemoryUsage fields to the backup cache
• Improved the description of the --online flag in the sdb-deploy upgrade command
• Improved the request to run as sudo message in the Toolbox commands
• Changed the warning messages in the defaultWorkloadManagement checker to be more consistent
• Fixed a bug in the sdb-report collect --mask-ip command that would cause an error when collecting report information for a stopped node

2021-01-08 version 1.9.5

• Improved the error message of the sdb-deploy setup-cluster command when an environment validation fails due to different users owning data directories of a node

2021-01-04 version 1.9.4

• Added the --mask-ip flag to the sdb-report collect command to mask sensitive data in the report files, such as IP addresses, MAC addresses, usernames, and passwords
• Added the role column to the mv_nodes table for population by the monitoring pipeline
• Added the sdb-admin enable-nodes-autostart command to enable nodes autostart on system reboot
• Changed the chronydStatus and the partitionsStart collectors in the sdb-report command to be excluded by default
• Changed the description of analyzeTable collector in the sdb-report command
• Changed the memory units in the consistent_maximum_memory checker from kilobytes (KB) to megabytes (MB)

2020-12-15 version 1.9.3

• Fixed an issue in the deployment UI that would occur after clicking the Add Another Host button
• Fixed a bug in the sdb-deploy setup-cluster command that would sometimes place the memsqlctl.hcl file in an incorrect folder

2020-12-14 version 1.9.2

• ​As of Toolbox 1.9.2, each ​memsql-​​ command now has an associated ​sdb-​​ analog that is both compatible with, and functionally identical to, the memsql-​​ command. All memsql- commands continue to function as expected. With this enhancement:
  • sdb-admin is equivalent to memsql-admin
  • sdb-deploy is equivalent to memsql-deploy
  • sdb-report is equivalent to memsql-report
  • sdb-toolbox-config is equivalent to memsql-toolbox-config
• Added the sdb-toolbox-config state, the sdb-deploy state, the sdb-admin state, and the sdb-report state commands to display the Toolbox state from the toolbox-state.hcl file
• Added the --preinstalled-path flag to the sdb-deploy install, sdb-deploy cluster-in-a-box, and the sdb-deploy setup-cluster commands to specify the folder where a memsql-server .tar file has been extracted as an install target
• Added the networdBuffersMax checker to the sdb-report command to check that the wmem_max and the rmem_max values are at least 8MB
• Added the --config-file flag to the sdb-admin send-alert command to specify a YAML file containing both the location and the thresholds as an alternative to using the --alert-location and the --thresholds flags
• Added the --monitoring-version flag to the sdb-admin start-monitoring command to specify a memsql-server version to pass to the memsql-exporter
• Added the --batch-interval flag to the sdb-admin start-monitoring command to specify a time interval between data extractions
• Added the --sample-queries and the --high-cardinality-metrics flags to the sdb-admin start-monitoring command to enable the collection of additional monitoring data
• Updated the sdb-admin start-monitoring command to support features introduced in the SingleStore DB 7.3 version of the memsql-exporter
• Changed the mallocActiveMemory checker to fail if malloc_active_memory is greater than 5GB (up from 2GB)
• Changed the diskUsage and the diskInodesUsage checkers in the sdb-report check command to not report high usage on loopback filesystems
• Changed the sdb-report collect command to not output a stack trace when a collection error is encountered
• Fixed a bug in the sdb-deploy upgrade command that would sometimes cause aggregator actions to be displayed in an incorrect order during a dry run

2020-11-30 version 1.9.1

• Added the processFiles collector to the memsql-report command to collect all /proc/<pid>/* files for the memsqld and the memsqld_safe processes
• Added the autoAttach checker to the memsql-report command to warn if the auto_attach engine variable is not set to ON
• Added the --preinstalled-path flag to the memsql-deploy upgrade command to specify the folder as an upgrade target where a memsql-server .tar file has been extracted
• Improved the error message for the memsql-deploy ui command when registered hosts are detected

2020-11-16 version 1.9.0

• Added the cgroupDisabled checker and osRelease collector to the memsql-report command to check that cgroup memory is disabled for Red Hat 7.x or later
• Added the memsql-report send-alert command to automatically run memsql-report collect-and-check and alert on fails or warnings
• Updated the descriptions of multiple checkers in the memsql-report command
• Improved the long message for the memsql-admin change-root-password command
• Improved the error message for the memsql-admin start-monitoring command when the exporter host is inaccessible from other hosts
• Improved the error messages related to environment validation in the memsql-report collect and the memsql-deploy setup-cluster commands
• Improved the orchestratorProcesses checker to include each process in the report only once, even if more than one instance of the process is running on the host
• Changed the memsql-deploy setup-cluster command to check that all of the nodes that will be registered have the correct file permissions and owner
• Changed the output of the schema collector to include views definitions, and improved how functions, procedures, and error messages are displayed
• Changed the memsql-deploy setup-cluster command to set the secure-file-priv engine variable to a default path of <memsqlctl's-default-install-dir>/db_files for each newly-created node
• Fixed a bug in the memsq-admin restart-node command that would sometimes display an incorrect error message
• Fixed a bug in the clusterMemoryUsage checker that would produce incorrect output
• Fixed a bug in the memsql-admin enable-high-availability command that would sometimes cause a kernel panic if Toolbox is configured incorrectly

2020-11-02 version 1.8.1

• Improved the error message in the memsql-admin backup-related commands when the --repository flag contains an invalid value
• Fixed a bug where the memsql-report check --validate-env command would crash when a host has TarInstallDir defined and memsql-server installed
• Added the installedPermissions checker and collector to the memsql-report command to verify that cluster files and directories have the correct permissions
• Fixed a bug where the --skip-validate-env flag was ignored by the memsql-deploy setup-cluster command
• Added the --data-dir flag to the memsql-deploy setup-cluster command for specifying the absolute path to each node’s data directory (DataDir)
• Improved the memsql-deploy ui command to support pre-deployment environmental checks
• Improved the error message in the memsql-admin restart-node --online command to inform the user that relevant nodes must be running to perform the restart
• Fixed a bug in the memsql-deploy upgrade command that incorrectly determined that all columnstore tables could be safely upgraded

2020-10-19 version 1.8.0

• Added the memsql-admin sync-ldap command to synchronize LDAP users and groups with SingleStore DB/MemSQL database users
• Added a dry (trial) run message to the memsql-report collect command that displays the number of collectors that will be executed
• Fixed a bug where the value of the timeout in the memsql-admin query --row-timeout command was not taken into account by the command
• Added a clusterMemoryUsage checker to the memsql-report check command to determine if there is at least 10% of unused memory left for each node
• Added a diskInodesUsage checker to the memsql-report check command to determine if there is at least 15% of unused disk inodes left on each host
• Fix a bug where the dmesg -T command was not executed while collecting reports with the memsql-report collect command
• Added a diskLatencyRead checker to the memsql-report check command to determine if the disk read latency is below the recommended threshold on each host
• Added a diskLatencyWrite checker to the memsql-report check command to determine if the disk write latency is below the recommended threshold on each host

2020-10-05 version 1.7.2

• Added the memsql-admin unregister-node and memsqlctl unregister-node commands to unregister nodes in Toolbox
• Added the majorPageFaults checker and collector to the memsql-report command to verify if the number of major page faults that the system generates per second is lower than a specific threshold
• Added the swapUsage checker and collector to the memsql-report command to verify if the swap usage is lower than a specific threshold
• Added the cpuIdle checker and the cpuUtilization collector to the memsql-report command to verify if the percentage of time the CPUs have been idle is higher than a specific threshold
• Added the memoryCommitted checker and collector to the memsql-report command to verify if the percentage of memory required for a given workload is less than a specific threshold
• Added the swapEnabled checker to the memsql-report check command to verify if the total swap memory is not less than 10% of the physical memory
• Added the vmSwappiness checker to the memsql-report check command that checks if the vm.swappiness kernel setting is greater than 0 on each host
• Added environment validation to the memsql-deploy setup-cluster command
• Added the --tar-install-dir flag to the memsql-deploy cluster-in-a-box command
• Changed the max_memory_settings checker of the memsql-report check command to accept a maximum memory use of 91%, up from 90%
• Updated the memsql-admin query command to not require a password for the SingleStore DB root user
• Updated the memsql-admin change-root-password, memsql-admin create-node, memsql-admin query, and memsql-admin register-node commands to prompt for a password only after all of the included flags have been verified
• Improved the warning messages in the memsql-deploy upgrade --precheck-only command
• Fixed a bug in the memsql-report check command where messages about unavailable checks were displayed
• Fixed a bug where the syncCnfVariables checker in the memsql-report check command would inadvertently return incorrect results
• Fixed a bug in the memsql-admin query command that would ignore the --user flag
• Fixed a bug in the memsqlctl command where some commands that were run with the --timeout flag would not exit after the specified timeout

2020-09-21 version 1.7.1

• Added an --include memsqlAuditlogs flag to the memsql-report collect command to collect audit logs
• Fixed a bug in the memsql-report check command where maxMemorySettings reported a failure when a host has more than 10GB of memory free
• Added a no_start: true option to the node section of a cluster configuration YAML file for use with the memsql-deploy setup-cluster command
• Fixed a bug where host fingerprinting would fail to compute on some hosts
• Added a --no-partition-change flag to the memsql-admin optimize command to skip any changes made to the default_partition_per_leaf SingleStore DB/MemSQL variable
• Fixed a bug in the memsql-admin enable-high-availability command that would co-locate nodes on the same host that were in different availability groups
• Improved the error message for the memsql-deploy setup-cluster command when duplicate hosts are found in the cluster configuration YAML file
• Improved the logging function by saving all of the messages generated when running Toolbox commands with the --json flag
• Improved the memsql-deploy setup-cluster command to display an error message when a node is specified with an incorrect availability group in the cluster configuration YAML file
• Fixed a bug where standard output of memsqlctl commands was sometimes lost
• Moved the message that memsqlctl is an earlier version to verbose output

2020-09-09 version 1.7.0

• Added the memsql-admin configure-monitoring command that updates the monitoring configuration for the cluster
• Added the memsql-admin start-monitoring and memsql-admin stop-monitoring commands to start and stop the collection of monitoring data
• Added the memsql-report collect-and-check command that runs both collect and check reports
• Improved the high availability functionality in the memsql-deploy setup-cluser command by pairing leaves of different hosts
• Fixed a bug in the memsql-admin enable-high-availability command where nodes were assigned non-optimal availability groups
• Improved the memsql-deploy upgrade command to report all precheck failures
• Improved the memsql-deploy setup-cluster command to explain why a clean cluster is required before deploying SingleStore DB/MemSQL
• Fixed a bug where the syncCnfVariables checker would crash in the memsql-report check command
• Fixed a bug in the memsql-admin query command where --password was not correctly processed

2020-08-26 version 1.6.7

• Fixed a bug where an “access denied” error would cause the memsql-deploy upgrade command to fail

2020-08-21 version 1.6.6

• Added bash and zsh tab completion to all Toolbox commands
• Fixed a bug in the memsql-admin query command where --password was not correctly processed
• Improved the memsql-admin bootstrap-aggregator, memsql-admin create-node, memsql-admin set-license, memsql-deploy cluster-in-a-box, and memsql-deploy setup-cluster commands to use the license from the MEMSQL_LICENSE environment variable
• Fixed a bug where memsql-deploy upgrade would fail due to a replicating database
• Fixed a bug where a blob existed but couldn’t be opened in the memsql-admin backup-related commands
• Improved logging to always include output of SQL queries
• Fixed the error message in the memsql-deploy setup-cluster command when a host without memsql-server installed was already part of the cluster
• Added a warning to not edit the memsql.cnf file manually
• Added retries to sudo prompting
• Improved the error message for the memsql-deploy upgrade command when using a tar file without specifying tar-install-dir
• Improved the transparentHugepage checker of the memsql-report check command to not check defrag if THP is disabled

2020-08-10 version 1.6.5

• Added the attachRebalanceDelay checker to the memsql-report check command that verifies that attachRebalanceDelay is not set to 86400
• Fixed a bug where an error wasn’t displayed when attaching a leaf failed while executing the memsql-deploy upgrade command
• Updated the memsql-toolbox-config register-host command to validate the --memsqlctl-path value and added the--skip-validation flag to skip this validation
• Improved the memsql-deploy upgrade command to attempt to attach and detach a leaf under certain error conditions instead of proceeding to an offline upgrade
• Improved the error message when an attempted SSH connection fails
• Added the diskBandwidth checker to the memsql-report check command that checks disk read, write, and sync-read bandwidth
• Improved the informationSchemaMvEvents checker of the memsql-report check command to order events by time
• Added the --disable-spinner flag to all Toolbox commands to disable the progress spinner, which some terminal sessions/environments may have issues with
• Improved the memsql-deploy setup-cluster command to suppress an error if one or more hosts did not have memsqlctl installed
• Changed the validLicense checker of the memsql-report check command to report in units for clusters with a unit-based license
• Added the --base-install-dir flag to the memsql-deploy setup-cluster command which sets each node’s base install directory on all hosts

2020-07-27 version 1.6.4

• Improved the memsql-admin delete-backup command to no longer error if a backup name is listed more than once
• Added a password prompt to all Toolbox commands as an alternative to specifying the password as a command-line argument, which would be captured in the shell’s history
• Fixed a bug where tarball-based installations failed for remote hosts
• Improved the descriptions of the memsql-admin delete-node, memsql-deploy uninstall, and memsql-toolbox-config unregister-host commands
• Added the consistentMaxMemory checker to the memsql-report check command that confirms the maximum_memory setting is consistent across all leaves and aggregators
• Added the memsql-deploy destroy-cluster command that deletes all data, deletes all SingleStore nodes, and unregisters all hosts
• Improved the dry runs of the memsql-admin delete-node, memsql-deploy uninstall, and memsql-toolbox-config unregister-host commands
• Improved the cluster file used by the memsql-deploy setup-cluster command to accept the case-insensitive role of a node
• Added an --all flag to the memsql-admin describe-node command to list the details for all nodes
• Added a --precheck-only flag to the memsql-deploy upgrade command to check if a cluster is stable and an upgrade is possible without performing an upgrade
• Improved the memsql-admin command’s backup/restore functionality to search for a backup in a repository if one isn’t found in a cache
• Improved the memsql-deploy upgrade command to compare the target upgrade version against the major version that is already installed
• Changed the syncCnfVariables checker of the memsql-report check command to confirm that its value is equal to the runtime value in memsql-server

2020-07-14 version 1.6.3

• Fixed a bug where the memsql-deploy ui command fails to run

2020-07-13 version 1.6.2

• Added the --master-port, --leaf-port, and --bind-address flags to the memsql-deploy cluster-in-a-box command
• Updated the memsql-deploy setup-cluster commands to not allow SSH for localhost
• Fixed a bug where the memsql-deploy upgrade command fails when memsql-server 7.1 is already installed on a host
• Added details as to why some Toolbox commands prompt for a sudo password
• Added a check in the memsql-deploy upgrade command that confirms that the active memsql-server package is the one for the target version
• Updated the memsql-deploy setup-cluster command to provide the same output with or without the --config-file flag
• Removed the connections per node limit in the memsql-deploy upgrade command
• Fixed a bug where SSH standard error output was sometimes discarded
• Added a --file-path flag to the memsql-admin set-license command to allow the license to be stored in, and read from, a file
• Updated the memsql-deploy commands to prompt for which package manager to use if both deb and rpm package managers are installed
• Improved the version checks of the memsql-deploy upgrade command
• Added a --skip-version-check flag to the memsql-deploy upgrade command that ignores the currently active memsql version and enables downgrading
• Added the --role, --license, --user, --user-password, --availability-group, and --force-master flags to the memsql-admin create-node command. Refer to the memsql-admin create-node reference page for more information.
• Improved the functionality of the collector-related flags in the memsql-report commands

2020-06-30 version 1.6.1

• Fixed a bug where memsql-deploy setup-cluster fails when run with the --cluster-file flag

2020-06-29 version 1.6.0

• Changed the memsql-deploy cluster-in-a-box command to bind nodes to the address 127.0.0.1 instead of 0.0.0.0
• Changed the memsql-deploy uninstall --all-versions command to return a 0 exit code if there are no versions to uninstall
• Added the memsql-deploy ui command that runs a web UI to install SingleStore DB and deploy a cluster
• Added syncCnfVariables checker to the memsql-report check command that confirms that there are no sync variables values defined in the memsql.cnf file on a host
• Updated the memsql-deploy cluster-in-a-box command to deduce the package format when the --file-path flag is specified
• Added the --force-package-format flag to the memsql-deploy list-versions command that lists the versions installed by the chosen package manager
• Fixed a bug where memsql-deploy upgrade displays an error message when attempting to upgrade to the MemSQL version that already installed and activated on the host
• Changed the memsql-report check command to display results only for checkers that have all needed data, and list those checkers that do not
• Updated the memsql-admin optimize command to display a message when non-uniform memory allocations for leaves are found
• Updated the description of the memsql-admin put-file command

2020-06-15 version 1.5.5

• Added a required --password flag to the memsql-deploy cluster-in-a-box command that sets a password for the SingleStore DB root user
• Fixed a bug where the memsql-report check command crashes when the orphanTables checker does not find specific information in a report
• Fixed a typo in the memsql-report command’s error message when it fails to detect whether memsqlctl is installed on a host
• Fixed a bug where performance collectors were executed by default with the memsql-report collect command
• Added support for file protocol for memsql-admin backup functionality

2020-06-03 version 1.5.4

• Added a --force-package-format flag to the memsql-deploy uninstall command which will allow MemSQL to be uninstalled when there are both dpkg and rpm package managers found on a host
• Fixed a bug where memsql-admin enable-high-availability fails with MemSQL version below 7.0.6
• Added the result of ANALYZE OPTIMIZER_STATE INTO OUTFILE of the memsql-report collect command to the collector’s section of the analyzeTables report
• Fixed a bug where memsql-report check would issue a warning on a valid distributed_keepalive_timeout value
• Updated the memsql-deploy list-versions command to display its output more quickly
• Displays a more specific error message when a file on a host cannot be accessed
• Updated the output of commands that are run with -vvv to be more readable
• Added --remote-temp-dir flag to the memsql-report collect command that allows setting a directory for storing temporary files on a host
• Fixed a bug where a validLicense check reports a failure on a valid license

2020-05-18 version 1.5.3

• Added analyzeTables collector to memsql-report collect that saves column and histograms recorded on each table
• Added the ability to store a path to a license file (rather than the license itself) in the cluster configuration file used by memsql-deploy setup-cluster --cluster-file
• Improved dmesg collector with timestamps that users can recognize and interpret
• Fixed a bug where memsql-deploy upgrade does not show an error if the final upgrade step fails
• Fixed a bug where memsql-deploy upgrade would hang if there were orphan databases
• Added a --max-column-width flag to the memsql-admin query command that allows query output columns to be truncated to the specified number of characters
• Added a --allow-duplicate-host-fingerprints flag to memsql-deploy setup-cluster and memsql-toolbox-config register-host commands that allows hosts with identical SSH host keys to be added to the cluster

2020-05-04 version 1.5.2

• Fixed a bug where memsql-report collect hangs when collecting reports from some hosts
• Fixed a bug where a cpuFreqPolicy check reports a failure on a valid configuration
• Fixed a bug where the chronydStatus collector does not work for a host that does not have pgrep
• Removed maxOpenFiles and filesystemType checks from memsql-report check --validate-env
• Added a warning and the request for user conformation when memsql-toolbox-config register-host is run with --localhost and --host 127.0.0.1
• Fixed a bug where memsql-deploy upgrade incorrectly estimates the disk space required to upgrade from MemSQL 6 to MemSQL 7
• Made memsql-admin delete-backup faster by deleting multiple backups in parallel
• Added cpuMemoryBandwidth collector and checker to memsql-report that measures CPU-memory bandwidth on each host and confirms that both bandwidth and latency are within acceptable limits
• Added --include-performance flag to memsql-report collect and memsql-report check that is required to run performance checks for a cluster
• Displays a detailed error message when the memsql-toolbox-config register-host and memsql-deploy setup-cluster commands are run with multiple localhosts

2020-04-20 version 1.5.1

• Added new collectors and checkers to memsql-report collect and memsql-report check commands:
  • “memsqldProcessLimits” collects limits for all memsqld processes
  • “chronydStatus” collects the status of the chronyd service
  • “chronydDisabled” checks that chronyd is disabled
  • “partitionsStart” collects information about each partition’s start sector number
  • “partitionsConsistency” checks that all partition start sectors are consistent
  • “cpuThreadingInfo” collects information about CPU threading configuration
  • “cpuHyperThreading” checks that all CPUs have hyperthreading enabled
  • “cpuFreqInfo” collects information about CPU frequency configuration
  • “cpuFreqPolicy” checks that all CPUs have power save disabled and turbo mode enabled
• Fixed a bug where the ulimit checker reports FAIL even when set correctly for the memsql user
• Displays a more meaningful error message when region or endpoint_url is missing in a backup command
• Fixed a bug where memsql-report collect fails if some node IDs could not be read
• Fixed a bug where memsql-deploy upgrade --online fails to upgrade from MemSQL version 7.0.15
• Added “region” to the backup functionality help examples

2020-04-06 version 1.5.0

• Added functionality for backup management:
  • memsql-admin create-backup - backup one or more databases to a repository
  • memsql-admin delete-backup - delete a backup
  • memsql-admin list-backups - list all the latest backups
  • memsql-admin restore-backup - restore a database from a backup
  • memsql-admin summarize-backups - print summary information for backups
  • memsql-admin validate-backup - verify that a backup is correct
• Fixed a bug where memsql-deploy setup-cluster fails if the cluster configuration file contains skip_install:true
• Added a check to memsql-deploy setup-cluster that prevents nodes from sharing the same folder on a host when specified in a cluster configuration file

2020-03-23 version 1.4.4

• Added computation of a “fingerprint” for each host to avoid adding the same host multiple times under a different name.
• Added --online flag for memsql-deploy upgrade that runs online upgrade for SingleStore DB.
• Fixed a bug where detectCrashStackTraces check displayed warnings if the memsqlStacks.files.json file was absent.
• Improved error message for memsql-deploy setup-cluster when the path to the source file is incorrect.
• Added toolboxConfig collector to memsql-report collect that saves the Toolbox’s configuration to a report file.
• Added a host collector to memsql-report collect that saves each host’s description to a report file.
• Fixed a bug where memsql-admin enable-high-availability does not request the correct privilege.
• Added a snapshot retry before upgrade if a database is re-provisioning.
• Removed extra escape characters in log outputs to make verbose logs more readable.

2020-03-09 version 1.4.3

• Improved command memsql-deploy upgrade:
  • Added a disk space check and a database snapshot is taken before an upgrade is started.
  • Added cleanup steps in case of an unsuccessful upgrade.
• Added memsql-admin put-file command that copies a file to the same path on all hosts.
• Fixed a bug where memsql-admin stop-node --all --force did not stop any nodes if one node was unavailable.
• Stopped producing messages if no action will be taken by a command.
• Fixed a bug where memsql-deploy upgrade did not wait for the database to restart after an upgrade is finished.
• Fixed a bug in memsql-admin enable-high-availability where the command does not work correctly if two leaf nodes on the same host have different hostnames.
• Added an additional check to memsql-report that ensures all cluster hosts have SSD drives.
• Added --validate-env option to memsql-report that will validate if MemSQL can be installed.

2020-02-11 Version 1.4.2

• Improved the prompt message that is displayed when upgrading databases to the new replication modes in MemSQL 7.0.
• Improved the speed of health checks that are performed on columnstore tables during upgrades.
• Improved the way memsql-deploy upgrade waits for node recovery.
• Fixed a bug where memsql-toolbox-config is not able to resolve a state file but returns incorrect error.
• Fixed a bug where memsql-deploy downloaded the package twice for online upgrade.
• Fixed the behavior of disk checks before upgrade: always fail upgrade if there is not enough space, show success message on success only.
• Fixed a bug where toolbox commands did not close the environment after a command failure.
• Added command-line flag checks to memsqlctl client.
• Added possibility to set MaxSSHSessionsPerHost parameter with command line (--ssh-max-sessions) or config file.
• Fixed bug where the memsql-deploy setup-cluster --cluster-file <cluster-file> command returns nodes with Unknown role.

2019-11-25 Version 1.4.1

• Fixed a bug where all Toolbox commands were not able to pass the --parallelism flag.
• Fixed a bug where running memsql-deploy upgrade without any flags would perform a rolling restart on the cluster.
• Fixed a bug where the user would always be prompted for changes to the replication mode of the databases, instead of only being asked when upgrading to MemSQL 7.0.

2019-11-19 Version 1.4.0

• Updated memsql-deploy to support upgrade to MemSQL 7.0
• Added TSV based output for some memsql-report collectors.
• Fixed a bug where memsql-report check fails to check for blocked queries with time 0.
• Updated memsql-admin update-config --set-global to automatically run on the Master Aggregator.
• Updated the securityLimits collector to include the output of /etc/security/limits.d and /proc/PID/limits
• Updated the numactl checker to check if the sum of numa node memory capacity exceeds machine memory capacity.
• Added new collectors to memsql-report collect:
  • top
  • informationSchemaMvBackupHistory
• Added new checkers to memsql-report check:
  • orphanTables

2019-10-09 Version 1.3.0

• Added support for memsql-deploy setup-cluster to accept a YAML file with cluster configurations.
• Fixed a bug where memsql-report exits with a status of 0 if it fails.
• Fixed a bug that caused the output of memsql-report collect to be not parsable by memsql-report check
• Added new collectors to memsql-report collect:
  • uptime
  • shutdownTime
  • rebootTime
• Added new checkers to memsql-report check:
  • tracelogOOM
  • tracelogOOD
  • explainRebalancePartitionsChecker
  • longRunningQueries
  • detectCrashStackTraces
  • defaultVariables
• Updated the checkers defaultKeepaliveSettings and defaultHeartbeatSettings to be part of defaultVariables checker

2019-09-03 Version 1.2.3

• Fixed a bug where the output of memsql-report collect could be missing information.

2019-08-06 Version 1.2.2

• Added support for rootless memsql-server installs using tar-based packages.
• Added the --role flag to memsql-admin start-node to start the nodes by role.
• Added the --role flag to memsql-admin stop-node to stop the nodes by role.
• Added the Bind Address column to memsql-admin list-nodes output.
• Updated memsql-admin delete-node to remove leaves and child aggregators from the cluster before deleting them.
• Updated the sort order for memsql-admin list-nodes to be consistent with other commands.
• Improved the readability of the License Expiration field in memsql-admin show-license.
• Fixed a bug that caused the schema collector in memsql-report collect to not collect procedures from memsql-server version 6.7.11 and above.
• Added new collectors to memsql-report collect:
  • explainRestoreRedundancy
  • explainRebalancePartitions
  • showClusterStatus
• Added new checkers to memsql-report check:
  • delayedThreadLaunches

2019-04-22 Version 1.1.1

• Added the --stop flag to memsql-admin delete-node to stop the nodes before deleting them.
• Added the --host, --port, --user, and --password flags to memsql-admin query for querying a node directly.
• Added new checkers to memsql-report check:
  • disconnectedReplicationSlaves
  • interpreterMode
  • leafAverageRoundtripLatency
  • runningBackup
  • secondaryDatabases
  • validLicense
• Updated checkers in memsql-report check:
  • blockedQueries also looks for sleeping threads with open transactions
  • offlineLeaves renamed to leavesNotOnline
• Added table-based output for some memsql-report collectors.
• Shortened the filenames inside a cluster report to avoid name length limits in archive files.

2019-03-19 Version 1.1.0

• Fixed a bug that prevented NOPASSWD sudo from being considered passwordless sudo.
• Added the memsql-admin query command.
• Added more properties to memsql-admin list-nodes and memsql-admin describe-node.
• Removed the swappiness checker. The setting it checked is not required by MemSQL 6.7.
• Updated the informationSchemaPipelines collector to include information_schema.PIPELINES_CURSORS and information_schema.PIPELINES_FILES.
• Added new collectors to memsql-report collect:
  • explainOrphanDatabases
  • informationSchemaMvQueries
  • licenseMetadata
• Added new checkers to memsql-report check:
  • defaultHeartbeatSettings
  • defaultKeepaliveSettings
  • defaultWorkloadManagement
  • defunctProcesses
  • duplicatePartitionDatabase
  • failedCodegen
  • offlineAggregators
  • offlineLeaves
  • pendingDatabases
  • queuedQueries
  • userDatabaseRedundancy
  • versionHashes
• Updated the report format to include the source paths of copied files.
• Updated the env commands to include the resolved configFile location.

2019-01-08 Version 1.0.6

• Updated memsql-deploy cluster-in-a-box to set passwords that are required by memsqlctl create-node starting in MemSQL 6.7.8.
• memsql-admin create-node now requires the --password flag.
• Added the --base-install-dir flag to memsql-admin create-node.
• Updated memsql-report collect --only to also include the memsqlctlInfo collector by default.
• Added a safety check to memsql-deploy setup-cluster that prevents enabling High Availability with an odd number of leaves.
• Added the --skip-disk-check flag to memsql-admin enable-high-availability.

2018-12-04 Version 1.0.5

• Added support for advanced usage of memsql-admin register-node.

2018-12-04 Version 1.0.4 (internal only)

• Added the attempted command string to error messages due to failed command execution.
• Added support for running commands as a Linux user that is part of an unnamed group.
• Fixed a bug in memsql-deploy install that improperly failed the installation if it was unable to add the target user to the memsql Linux group.
• Added support for advanced memsqlctl configuration to memsql-report collect-local.
• Reduced spurious errors from memsql-report check when memsqlctl is not installed on the target hosts.
• Added support for changing the SQL root password on all nodes in the cluster with memsql-admin change-root-password.

2018-11-27 Version 1.0.3

• Added advanced memsqlctl configuration flags to memsql-toolbox-config register-host.
• Enabled parallelism for memsql-report collect.
• Added the report file path to the output of memsql-report collect and memsql-report collect-local.
• Fixed a bug in memsql-admin confirmation messages that incorrectly implied the use of --password when it was not used.

2018-11-20 Version 1.0.2

• Fixed bug preventing the use of memsql-admin update-config --all.
• Added “host” field to memsql-admin describe-node output.
• Fixed a bug in memsql-deploy list-versions when there are no hosts registered.
• Changed permissions logic to use memsqlctl configuration to determine the correct user and group for remote commands.

2018-11-09 Version 1.0.1

• Fixed bug causing Toolbox to assume passwordless sudo if there were cached sudo credentials in the current session.

2018-11-06 Version 1.0.0

• Initial release of memsql-toolbox.