MemSQL 4.1 features the new MemSQL Streamliner and columnstore enhancements that simplifies the creation and management of real-time data pipelines running in Apache Spark.

Streamliner

The MemSQL Streamliner deploys Apache Spark and runs user-defined, real-time data pipelines in Spark. A data pipeline is a workflow that extracts information from a real-time data source such as Kafka, performs a transformation on the data, and loads it into MemSQL. All Streamliner configuration can be set up within MemSQL Ops.

Streamliner opens the door for a whole new set of data streaming use cases into MemSQL, and integrates Apache Spark more tightly with MemSQL. Using Streamliner, users can deploy a Spark cluster - master, workers and the MemSQL Spark Interface - with one click, then immediately create data pipelines with configurable extract, transform and load phases, all processed within Apache Spark. Data is loaded as micro-batches into MemSQL, and Streamliner gives visibility into the full data pipeline.

For more information about the MemSQL Streamliner, see the Spark Management section.

Enhanced Columnstore Functionality

Faster streaming into columnstore tables

MemSQL now puts a memory-optimized data structure in front of each columnstore table which groups small inserts into larger batches for increased insert efficiency. Rows are available for queries immediately after the commit and do not wait for the write to disk. As with the MemSQL rowstore, inserts are fully transactional and durable.

Alter table for columnstore

MemSQL now supports the ALTER TABLE command for columnstore tables.

Increased columnstore data compression

JSON columns in a columnstore table are now represented in a more compressed format on disk.

Other Changes

Support for non-recursive CTEs

MemSQL now supports non-recursive common table expressions (CTEs). CTEs begin with a WITH clause prior to a SELECT, INSERT, UPDATE or DELETE operation. Non-recursive CTEs do not reference themselves within the CTE.

skip_name_resolve behavior

The skip_name_resolve variable now has 3 options - auto, true, and false. The default is auto, which tries to do a reverse DNS lookup. If the lookup succeeds, it uses rules defined for ‘%', and . If the lookup fails and there are host-based users created on the system, login is disallowed.

Maintenance Release Changelog

2016-02-04 Version 4.1.3

• Fixed issue when writing columnar segments that end with a JSON number value

2015-12-01 Version 4.1.2

• Fixed issue where low memory conditions could cause the columnstore memory-to-disk flusher to stop flushing rows
• Fixed issue where the check_system utility always complains that CRC32 is unavailable

2015-11-04 Version 4.1.1

• Fixed an issue with CREATE TABLE... LIKE for columnstore tables
• Fixed a performance issue with Parquet compression for certain JSON schemas

Was this Article Helpful?

Thanks For Your Feedback!