Outdated Version

You are viewing an older version of this section. View current production version.

START PIPELINE

The START PIPELINE statement starts an existing pipeline in a MemSQL database. An error will occur if the pipeline is already running.

Syntax

START PIPELINE [IF NOT RUNNING] pipeline_name [FOREGROUND [LIMIT count BATCHES] ];
START ALL PIPELINES;

Arguments

pipeline_name

The name of the pipeline.

count

An integer that represents the maximum number of batches to ingest.

IF NOT RUNNING

If included, this statement will start the pipeline if it is not running and do nothing otherwise.

START ALL PIPELINES

This command starts all pipelines in the current database in the background.
It is equivalent to running START PIPELINE IF NOT RUNNING pipeline_name for every pipeline in the database.

Remarks

Starting a Pipeline in the Background

Start a pipeline in the background by running the START PIPELINE statement without the FOREGROUND clause.

After starting the pipeline, the client will not report the number of affected rows or any errors that it encounters. You can find this information in the Pipelines information schema tables.

The engine variable pipelines_stop_on_error specifies whether a pipeline started in the background will stop if it encounters an error.

Use a background pipeline when you want to ingest data from a data source at regular intervals. You can also use a foreground pipeline for the same purpose, but the foreground pipeline will stop under certain conditions, as explained in the next section.

Starting a Pipeline in the Foreground

When a pipeline is started in the foreground, the client will report the number of affected rows or any errors it encounters.

To start a pipeline in the foreground without a batch limit, run START PIPELINE mypipeline FOREGROUND;.

To start a pipeline in the foreground with a batch limit, use the LIMIT clause. For example, to specify a batch limit of ten, run START PIPELINE mypipeline FOREGROUND LIMIT 10 BATCHES;.

A foreground pipeline will stop when any of the following occur:

  • The batch limit is reached (if it is specified).
  • An error occurs.
  • All data from the data source has been loaded.

A foreground pipeline is intended for the following purposes:

  • Testing and debugging.
  • Running a one-time bulk data load.
  • Writing your own scheduler to load data at specific times.

Examples

Example 1: Starting a Pipeline in Background

The following example demonstrates how to start a pipeline with no additional clauses specified:

START PIPELINE mypipeline;
****
Query OK, 4 rows affected (0.04 sec)

To see if the pipeline is running, run SHOW PIPELINES:

SHOW PIPELINES;
****
+-------------------+---------+
| Pipelines_in_mydb | State   |
+-------------------+---------+
| mypipeline        | Running |
+-------------------+---------+

Example 2: Starting a Pipeline in the Foreground

The following example demonstrates how to start a pipeline in the foreground, which displays the number of rows written to the destination table:

START PIPELINE mypipeline FOREGROUND;
****
Query OK, 4 rows affected (1.39 sec)

To see if the pipeline is running, run SHOW PIPELINES:

SHOW PIPELINES;
****
+-------------------+---------+
| Pipelines_in_mydb | State   |
+-------------------+---------+
| mypipeline        | Running |
+-------------------+---------+

If your pipeline is not in the running state, see the section Starting a Pipeline in the Foreground for further details.

Example 3: Starting a Pipeline in the Foreground with a Batch Limit

The following example demonstrates how to start a pipeline in the foreground with a specified limit, which displays the number of rows written to the destination table:

START PIPELINE mypipeline FOREGROUND LIMIT 5 BATCHES;
****
Query OK, 37 rows affected (0.71 sec)

To see if the pipeline is running, run SHOW PIPELINES:

SHOW PIPELINES;
****
+-------------------+---------+
| Pipelines_in_mydb | State   |
+-------------------+---------+
| mypipeline        | Running |
+-------------------+---------+

If your pipeline is not in the running state, see the section Starting a Pipeline in the Foreground for further details.