Release Highlights

This release improves support for OLTP-style (highly-selective) queries on columnstores, High Availability, and Disaster Recovery.

Support for Operational Applications

• As part of the universal storage initiative, enhanced the columnstore to enable transactional workloads to be processed more efficiently. These enhancements include support for single-column unique hash keys on columnstores and fast joins on columnstores, where there is a highly selective filter on one table in the join.

• Added the INTO option to the SELECT command, which can be used in procedural SQL. SELECT INTO allows you to select the columns in a one-row resultset into a list of variables, making your code more concise.

High Availability and Disaster Recovery

• During replication, if your primary cluster fails, you can reverse replication from your secondary cluster to your primary cluster to quickly failback to the primary cluster.

• In High Availability mode, you can now use the load_balanced setting of the engine variable leaf_failover_fanout to evenly distribute a leaf L’s replica partitions across a set of leaves in the cluster. This is an alternative to the paired setting, where all of leaf L’s replica partitions reside on one paired leaf. Using load_balanced allows you, if leaf L fails, to distribute its failed-over workload to the set of leaves instead of just one, balancing the load.

Additional Features and Improvements in this Release

Storage

• Partition splitting, implemented as an option to the BACKUP command, allows you double the number of partitions in a database easily. You may wish to increase the number of partitions to accommodate more leaves, or to enable more parallelism.

• Added support for full backups to Google Cloud Storage (GCS). Also added support for incremental backups to GCS, S3, and Azure. For more information, see the Backing Up and Restoring Data topic.

• Added global temporary tables, which are similar temporary tables, except that they exist beyond the duration of a client session.

• When creating a table, you can now specify that a DATETIME or DATETIME(6) column should be populated with the current timestamp when inserting or updating a row in the table.

• Highly selective joins on columnstores are now more efficient. A highly selective join has a very selective filter on one table which produces a few rows and these rows are joined with another table.

Query Optimization

• Enhanced cardinality estimation for joins using histograms.

• Updated the behavior of the DROP ALL FROM PLANCACHE command; it now purges all query plans from memory and disk.

• Added more support for queries that use scalar nested subselects.

• Added the NOPARAM() function that disables the parameterization of constants when a query is compiled.

• Improved optimization of hash joins involving expressions.

Usability and Programmability

• You can now access record variable fields inside of a SQL statement in procedural SQL.

• Added the information_schema.MV_BLOCKED_QUERIES view, which provides a summary of activities that are currently waiting.

• Added the TO_NUMBER() function, which converts a CHAR or a VARCHAR value to a DECIMAL value.

• Added the TABLE() function, which converts a procedural SQL array to a set of rows.

• Added the TRUNC(date) and TRUNC(number) functions. TRUNC(date) allows you to truncate a date to a specified granularity. TRUNC(number) allows you to truncate a given number to a specified number of decimal places.

• Added the INET6_ATON() function, which converts an IPv6 or IPv4 internet address from its text representation to its binary representation.

• Added the INET6_NTOA() function, which converts an IPv6 or IPv4 internet address from its binary representation to its text representation.

• Added the REGEXP_SUBSTR() function, which searches for a regular expression pattern and returns the matching substring.

• Stored procedures now support the CREATE VIEW and DROP VIEW commands.

• ECHO SELECT now supports ECHO SELECT <expr> where <expr> is a variable or record field. Previously, an alias was required as in ECHO SELECT <expr> as <alias>.

Data Ingest

• Added support for Google Cloud Storage (GCS) Pipelines.

Security

• Added configurable password complexity rules. Now you can specify criteria for passwords, including criteria such as minimum length or the number of uppercase, lowercase, numeric, and special characters.

• Added a configurable account lockout feature. You can specify the number of times a user can enter an incorrect password before they are locked out of the system. When a user reaches this limit, their account is locked for the specified number of seconds. For more information on the failed login attempt lockout feature, see Securing SingleStore DB.

• OPTIMIZE TABLE now requires the INSERT permission.

Other

• Fixed a race condition where plan eviction during a distributed join would cause the error “SingleStore DB code generation has failed: Cannot compile from background threads”.

• Fixed a bug where decimals lose precision when scientific notation is parsed.

Maintenance Release Changelog

2021-10-12 Version 7.1.23

• Masked blob store CREDENTIALS text in BACKUP, RESTORE, and LINK commands in SHOW PROCESSLIST, information_schema.PROCESSLIST, and in all error messages.

2021-09-13 Version 7.1.22

• Added a new system variable called master_promote_kill_timeout_seconds, which specifies how long an online failover will wait to drain open write transactions before issuing a kill to those transactions. It is limited to 120 seconds. After the write transactions are killed, an application-level retry is required.
• Fixed an issue that can cause defunct processes for pipelines or code generation in a rare race condition.
• Updated memsqlctl start-node to check the validity of NUMA configurations when nodes are restarted. If the system doesn’t have NUMA configurations, the nodes will be restarted. If any of the nodes have invalid NUMA configurations, the NUMA bindings will be recalculated for all the nodes on the host.
• Fixed a crash that could occur when ALTER TABLE COMPRESSION=SPARSE is run in the same ALTER statement with an operation that doesn’t require data movement (i.e., AUTOSTATS_ENABLED = ON).
• Fixed an issue where modifying the case of a column via ALTER TABLE could cause a pipeline using that table to crash.
• Updated memsqlctl to establish an SSL connection to the node if a CA certificate is set explicitly. SingleStore now falls back to an SSL connection via the host’s default certificates if the target user requires SSL.
• Fixed an issue where incorrect results could be returned when using INSERT_ALL to insert into a temporary table.

2021-07-06 Vesion 7.1.21

• Fixed an issue where a malformed INIT_DB command sent by a client could cause an aggregator to crash .
• Fixed an issue where a crash could occur when running UPDATE on a query with a scalar subselect in a SET clause after ALTER TABLE.
• Properly set keep alive and timeouts on the persistent leaf-to-master aggregator connection that is used to communicate replication link status. This fixes an issue with REBALANCE commands that failed with synchronization timeouts.
• Fixed a synchronization bottleneck, improving the ability of the pipeline scheduler to run many pipelines in parallel.
• Improved the performance of pipeline and database DDL actions when a database has many pipelines running.
• Fixed a blocking issue that could occur when several pipelines are being background scheduled simultaneously.

2021-06-07 Version 7.1.20

• Added a new system variable, auditlog_retention_period, which controls the retention period (in days) of audit log files on disk. The default is 0 which means files remain on disk indefinitely.
• ADMIN_ONLY audit log level now audits DDL statements only. Previously it would audit other non-DDL statements, which could cause bloat in the audit logs in ADMIN_ONLY mode.
• Fixed an issue where a crash could occur during concurrent columnstore DELETEs while escalating from row level locking to table level locking.
• Fixed an issue where incorrect results could be returned when using UNION ALL and SELECT ... INTO S3.
• Fixed an issue with DATETIME equality testing when updating columnstore tables.
• Optimized partition placements when rebalancing partitions in load_balanced mode.
• Fixed an issue where CREATE GLOBAL TEMPORARY TABLE LIKE ... would create a normal user table instead of a global temporary table.
• Fixed a crash that could occur when using a table reference like table_name.* inside TO_JSON.
• Fixed a rare race condition that could cause sync replication to get stuck.

2021-04-05 Version 7.1.19

• Fixed an issue that caused auto-rebalance operations occurring when leaf_failover_fanout was set to load_balanced not to fully rebalance the cluster.
• Added a new engine variable, backup_max_threads, which controls the number of concurrent threads used to run backups on leaves. By default, each backup uses one thread per partition. For more information, see List of Engine Variables.

2021-03-22 Version 7.1.18

• Fixed an issue with previously replicated databases showing as offline in SingleStore DB Studio after STOP REPLICATING has been run.
• Fixed an issue with the system variable redundancy_level being incorrectly set on leaf nodes, which previously caused errors when CHECK BLOB CHECKSUM WITH REPAIR was run.
• Monitoring clusters are now supported via SSL.
• Fixed a an issue where an unsupported FULLTEXT could crash the server.
• Increased TasksMax to 128000 for memsql.service in order to prevent errors and/or unexpected node behavior. Previously, this setting was often exceeded, causing systemd to limit the ability of memsqld to fork processes.
• Updated the Websocket Proxy to respect the engine variables, ssl_cipher and tls_version, in configuring TLS for external communication via https.

2021-02-16 Version 7.1.17

• Fixed an issue with automatic histograms on clusters that were upgraded with the variable cardinality_estimation_level set to 6.0.
• Fixed an issue where an UPDATE query on a columnstore table with unique key could wrongly report a duplicate key error in some cases.
• Added metrics about the availability of database partitions to memsql_exporter.
• Allowed memsqlctl to establish a secure connection when a node is configured for SSL using either the engine variable ssl_ca, or the host system’s trusted certificates for verification.
• Added the --insecure-ssl flag, which allows memsqlctl to support SSL connections without verification.
• Fixed an issue that occurred when running ALTER or TRUNCATE against a table that included ON DUPLICATE KEY UPDATE in its CREATE PIPELINE statement.

2021-02-01 Version 7.1.16

• Fixed an issue that caused read actions on columnstore JSON objects to result in an error (Error reading from parquet: Column not found in schema) in cases where trailing whitespaces were present in the JSON key names stored for a given table.
• Allowed memsqlctl to be run as the root user without additional configuration.
• The memsql_exporter process is now properly shut down when the exporter port is changed or the cluster is uninstalled. Previously, it left a defunct process.
• When running memsql_exporter, the password for the monitoring user is no longer exposed in the ps output.
• When inserting into a columnstore table, columns not needed to deal with a duplicate key are no longer decompressed.
• Improved the optimizer’s cost estimates for hash joins when one of the tables in a given join has an index matching the join condition.
• Fixed a crash that occurred when an aggregate function was used in an unsupported context inside a stored procedure.
• Added a new engine variable optimizer_empty_tables_limit, which sets a threshold for the number of empty tables that must be present in a query before the optimizer falls back to being rule based, instead of cost based. For more information, see List of Engine Variables.

2021-01-11 Version 7.1.15

• Improved the speed of server shutdown and of recovery for databases with large numbers of tables. This improvement is evident at 1,000 tables or more, and increases significantly after 10,000 tables.
• Fixed an issue that caused RESTORE commands to fail in cases where a database contained a large number of reference tables.
• The system’s root CA is now used where no ssl.ca.location is specified for a Kafka pipeline connection using SSL.
• Fixed an issue that caused DROP TEMPORARY TABLE IF NOT EXISTS to incorrectly return an error in cases where the table didn’t exist.
• Fixed a crash caused by a check on the maximum number of partitions allowed on a leaf node. Now, an error message is surfaced when this is exceeded.
• Fixed issue that caused an error (Transaction rolled back mid-query) to occur when running cross-database queries in multi-statement transactions.
• Enabled use of gzip encoded files in GCS pipelines.

2020-12-17 Version 7.1.14

• Reduced blocks on UPDATE and DELETE commands via background merger.
• Added a new variable, default_columnstore_table_lock_threshold, which sets a threshold for the number of rows that are locked when updating rows in a columnstore table before a table lock is acquired. For more information, see List of Engine Variables.
• Fixed an issue that occurred when a TO_QUERY statement referred to a table that did not exist.
• When using binary protocol, YEAR type results are now returned as MYSQL_TYPE_SHORT. Previously, these results were incorrectly returned as a length-encoded string.

2020-11-23 Version 7.1.13

• Fixed a crash caused by timestamp values during daylight savings time changes in columnstore tables using a timestamp as a sort key or unique key.
• Fixed a crash that occurred when querying information_schema.COLUMNAR_SEGMENTS, information_schema.MV_COLUMSTORE_FILES, or information_schema.MV_COLUMNAR_SEGMENT_INDEX while ALTER TABLE … DROP INDEX ... is being run concurrently in order to drop a secondary columnstore index.
• APPROX_COUNT_DISTINCT now supports aggregates with a grouping set. For example, SELECT APPROX_COUNT_DISTINCT(column_name) FROM table_name GROUP BY ROLLUP(column_name).
• In the output of PROFILE, indicated cases where a corrupted secondary key is encountered while running a given query.

2020-11-09 Version 7.1.12

• Added Avro schema evolution capabilities to Pipelines.
• Fixed an issue that caused crashes when using GROUPING() on expression inputs.
• information_schema.load_data_errors now shows errors on tables a user has access to via roles and groups. Previously, only direct grants allowed access to the view.
• Fixed an issue that caused crashes when using reference tables with unique keys on columnstores.
• Fixed an issue that caused index corruption with columnstore secondary keys in cases where a single partition has more than 2 billion unique values.
• Added support for audit logging levels other than ALL-RESULTS mode when clients connect using server side prepared statements. Previously, enabling audit logging disabled use of prepared statements.
• Fixed a crash that occurred when ANY_VALUE(NULL) is included in a query.
• Fixed a crash that occurred where a query-typed variable accesses a table via a 3-part name (for example, dbname.tablename.columnname) and the database referred to does not exist.
• When a leaf node moves from the attaching state to the online state and an error is encountered attaching its partitions, we now more aggressively retry to ensure all partitions are attached before moving the leaf into the online state.
• Snapshots of the memsql system database are now taken more frequently. Previously, the memsql database was slower to recover on higher latency disk than user databases; more frequent snapshots speed up restart times.
• Blocked the creation of pipelines which refer to computed columns or columns set via a default value in the WHERE clause.
• Allowed memsqlctl to register multiple nodes with the same auditlogsdir.
• Fixed a wrong result issue that caused only the first of multiple values with the same hash in a segment to be output when using an IN list (for example, IN (a, b, c)) with a columnstore secondary hash index.

2020-10-12 Version 7.1.11

• Added a new engine variable disable_subquery_merge_with_straight_joins that disables merging sub-queries if they contain STRAIGHT_JOIN in order to avoid unwanted query plans. Here, merging sub-queries refers to removing unnecessary nested layers from joins including sub-queries. For example, the query select * from (select * from table_a straight_join table_b) table_c straight_join table_d would be rewritten as select * from table_a straight_join table_b straight_join table_c if the engine variable disable_subquery_merge_with_straight_joins is set to OFF. If the variable is set to ON, the query would not be rewritten. The default setting in 7.1 is AUTO, which is equivalent to OFF.
• Added the command memsqlctl unregister-node to unregister nodes in memsqlctl.

2020-10-05 Version 7.1.10

• Fixed an issue where restoring a backup from a local or network drive caused the partitions to be unevenly distributed across the leaves. The issue occurred when leaf_failover_fanout was set to load_balanced.
• Fixed an issue where some memsqlctl commands that were run with the --timeout flag would not exit after the specified timeout.

2020-09-28 Version 7.1.9

• Fixed an issue that could prevent SingleStore DB/MemSQL from cleaning up stale data after a node goes offline due to an out of disk condition.
• Fixed a crash caused by stored procedures which include an INTO clause (that modifies variables), followed by an IN clause.
• BACKUP and RESTORE commands can now be governed by resource pools. See the BACKUP and RESTORE topics for more information.
• Fixed an issue causing RESTORE to fail after upgrading to SingleStore DB 7.1 and executing BACKUP … SPLIT PARTITIONS.
• Query hints are now supported inside query type variables in user defined functions and stored procedures.
• The CHECK BLOB CHECKSUM command now supports a WITH REPAIR option that can repair file system corruptions as long as one copy of a corrupt columnstore blob file exists on the cluster. See more information in the CHECK BLOB CHECKSUM topic.
• The query hint OPTION (MATERIALIZE_CTES = AUTO) was added to enable materialized common table expressions per query, as an alternative to the existing session variable MATERIALIZE CTES.
• Fixed a wrong results bug that occurred when joining a query type value (QTV) with a user table, and accessing a column that doesn’t exist in the QTV definition, but does exist in the user table schema. Previously, even if the column was specified as coming from the QTV, it was being read from the table. Now, specifying the QTV works correctly.
• BACKUP … TO AZURE now waits four minutes before timing out slow operations against the Blob Store.

2020-08-31 Version 7.1.8

• Fixed an issue that occurred when using server side prepared statements via the newer versions of the MariaDB client. The MySQL client is unaffected.
• Fixed an issue cleaning up global temp tables on leaves during rebalancing operations.
• Fixed an issue where two window functions with identical ORDER BY and PARTITION BY clauses, but differing ROWS BETWEEN clauses, incorrectly output the same values.
• Fixed a performance issue with JSON_TO_ARRAY().
• Fixed an issue where incremental autostats may enter an invalid state and be unable to recover automatically due to out of memory on a leaf node.
• Improved error message in cases where SingleStore DB has issues accessing the memsql.cnf file.
• Fixed an issue where REPLICATE DATABASE would fail in cases where no replicated database existed yet and the node containing the master aggregator was previously replaced with a new node.
• Added a multipart_chunk_size_mb setting to SELECT .. INTO S3 that controls the size of the file uploaded to S3 to support uploading larger result sets to S3.
• Fixed an issue where auto-attach would fail to attach a partition but succeed in bringing the leaf online, resulting in offline partitions.
• Ensured that any re-used existing replica partition is still forced in sync as part of the copy operation run as a step of a REBALANCE operation. Previously, this was a cause of failure in cases where REBALANCE failed once prior.
• Fixed a failure when running a ROLLUP query over a constant expression.
• Audit logging now supports auditing logins and login attempts.
• Audit logging file headers now include information about what level the file is logging at as well as if the level was changed.
• Added a tracelog (memsql.log) message to indicate if audit logging has been disabled from previously having been enabled.
• Added support for a SELECT containing a semicolon in the CREATE TABLE ... AS SELECT statement.
• Modified the infrastructure of the native monitoring solution by removing the Kafka and memsql-pusher components and automating the monitoring setup with SingleStore DB Toolbox commands.

2020-08-10 Version 7.1.7

• Fixed a compatibility issue that occurred when running newer versions of SAP Business Object Data Services (BODS) against SingleStore DB. The issue was that the command SET sql_mode = <expression> was not supported by SingleStore DB.
• Fixed an issue that could occur when using columnstore unique keys in an update query with a join.
• Avoid taking unneeded locks for queries writing into columnstore tables with unique key columns. Taking these locks could cause transactions containing multiple statements to deadlock.
• Improved the error message returned when REMOVE AGGREGATOR is run on the master aggregator.
• The behavior of the max_compilation_time_s engine variable no longer applies to DDL queries. Eliminating compilation timeouts for these queries, especially for ALTER TABLE, will prevent the queries from potentially running forever.
• Reduced the CPU usage required for many threads to allocate memory for variable length strings.
• Added a new engine variable convert_nonunique_hash_to_skiplist which, when set to ON, creates a skiplist index in place of any non-unique hash index. By default, the variable is set to OFF.
• Pipelines no longer stop executing if free disk space drops lower than 5 * <the value of the minimal_disk_space engine variable>. They now stop executing if free disk space drops lower than 400 mb + <the value of the minimal_disk_space engine variable>.

2020-07-27 Version 7.1.5

• Now, SHOW PROCESSLIST returns the IP address of the client (in the Host column) if the host name cannot be resolved.
• Fixed an issue that occurred when the engine variable interpreter_mode was set to interpret_first, where the first run of the query remained in interpreted mode even after compilation had finished.
• Added support for ALTER PIPELINE ... SET RESOURCE POOL.
• Improved the performance of rebalance partition operations that copy partitions with many small columnstore BLOB files.
• Improved predicate transitivity rewrites to avoid adding redundant predicates. This fixes compile timeouts for some queries with large numbers of predicates.
• Fixed an issue where the engine variable explicit_defaults_for_timestamp could be set to a different value on new nodes added to the cluster, as compared to the existing value already set on existing nodes. This would cause the error “partition’s table metadata are out of sync” when queries were run.
• Improved heartbeat failure detection when leaves were unresponsive to queries.
• Fixed an issue where setting the engine variable sql_mode to ONLY_FULL_GROUP_BY would , for queries using INTERSECT and EXCEPT, incorrectly generate an error indicating a field is unaggregated and not in the GROUP BY clause.
• Fixed an issue where setting the engine variable materialized_ctes to ALL would prevent certain queries using INTERSECT and EXCEPT from running.
• Re-enabled support for certain nested subselects that had been added previously and then removed erroneously.
• Improved the performance of batch deletes on columnstores. The increase in performance depends on the encoding of the columns.

2020-07-06 Version 7.1.4

• Now, license checks are Linux control group aware. For containerized deployments, CPU and memory limits are checked against each container.
• ANALYZE TABLE now requires the SELECT and (INSERT or ALTER) permissions to execute.
• Added the TO_JSON() function, which converts a table column, an entire table, a scalar value, or a single row to a JSON object.
• Fixed a syntax error that occurred when using the LINES STARTING BY or LINES TERMINATED BY options with SELECT ... INTO S3 or SELECT ... INTO KAFKA.
• Improved the accuracy of the PROFILE commands that have ColumnStoreScan components.
• When preserve_original_colstore_json is set to ON, null values and empty arrays are preserved in JSON columns in columnstores.
• SSL ciphers that use elliptic curves are now supported.
• Added an optimizer setting that allows you to adjust the cost of performing a cross join.
• Fixed an issue where user-defined tables having the same name as system tables, such as USERS, were treated as case-sensitive when table_name_case_sensitivity was set to OFF.

2020-06-15 Version 7.1.3

• Correctly generate an error when running ALTER VIEW on a schema-bound view v1, where a schema-bound view v2 refers to v1.
• Fixed an auto-attach failure in specific failure conditions where a node quickly got marked offline and then online by the aggregator, but the node itself did not restart.
• Fixed an issue where setting one of the three engine variables collation_server, collation_database, or collation_connection would not persist the value for all three variables on node restart. (Setting any of these variables will change the other two variables to the same value).
• EXPLAIN <query>, PROFILE <query>, and SHOW PROFILE commands are now allowed inside of stored procedures.
• Improved predicate transitivity rewrites to avoid adding redundant predicates. This fixes compile timeouts for some queries with large numbers of predicates.
• The STATE column of the information_schema.PIPELINES view now indicates if a Pipeline has failed due to insufficient disk space.

2020-05-28 Version 7.1.2

• Initial GA release of SingleStore DB 7.1

Was this Article Helpful?

Thanks For Your Feedback!