You are viewing an older version of this section. View current production version.
START PIPELINE
The START PIPELINE statement starts an existing pipeline in a MemSQL database. An error will occur if the pipeline is already running.
Syntax
START PIPELINE [IF NOT RUNNING] pipeline_name [FOREGROUND [LIMIT count BATCHES] ];
START ALL PIPELINES;
Arguments
pipeline_name
The name of the pipeline.
count
An integer that represents the maximum number of batches to ingest.
IF NOT RUNNING
If included, this statement will start the pipeline if it is not running and do nothing otherwise.
START ALL PIPELINES
This command starts all pipelines in the current database in the background.
It is equivalent to running START PIPELINE IF NOT RUNNING pipeline_name for every pipeline in the database.
Remarks
Starting a Pipeline in the Background
Start a pipeline in the background by running the START PIPELINE statement without the FOREGROUND clause.
After starting the pipeline, the client will not report the number of affected rows or any errors that it encounters. You can find this information in the Pipelines information schema tables.
The engine variable pipelines_stop_on_error specifies whether a pipeline started in the background will stop if it encounters an error.
Use a background pipeline when you want to ingest data from a data source at regular intervals. You can also use a foreground pipeline for the same purpose, but the foreground pipeline will stop under certain conditions, as explained in the next section.
Starting a Pipeline in the Foreground
When a pipeline is started in the foreground, the client will report the number of affected rows or any errors it encounters.
To start a pipeline in the foreground without a batch limit, run START PIPELINE mypipeline FOREGROUND;.
To start a pipeline in the foreground with a batch limit, use the LIMIT clause. For example, to specify a batch limit of ten, run START PIPELINE mypipeline FOREGROUND LIMIT 10 BATCHES;.
A foreground pipeline will stop when any of the following occur:
- The batch limit is reached (if it is specified).
- An error occurs.
- All data from the data source has been loaded.
A foreground pipeline is intended for the following purposes:
- Testing and debugging.
- Running a one-time bulk data load.
- Writing your own scheduler to load data at specific times.
Permissions Needed to Start a Pipeline
START PIPELINE requires permission only to start the pipeline, not to use the resource pool of the pipeline. For more information on resource pools, see Setting Resource Limits.
Examples
Example 1: Starting a Pipeline in Background
The following example demonstrates how to start a pipeline with no additional clauses specified:
START PIPELINE mypipeline;
****
Query OK, 4 rows affected (0.04 sec)
To see if the pipeline is running, run SHOW PIPELINES:
SHOW PIPELINES;
****
+-------------------+---------+
| Pipelines_in_mydb | State |
+-------------------+---------+
| mypipeline | Running |
+-------------------+---------+
Example 2: Starting a Pipeline in the Foreground
The following example demonstrates how to start a pipeline in the foreground, which displays the number of rows written to the destination table:
START PIPELINE mypipeline FOREGROUND;
****
Query OK, 4 rows affected (1.39 sec)
To see if the pipeline is running, run SHOW PIPELINES:
SHOW PIPELINES;
****
+-------------------+---------+
| Pipelines_in_mydb | State |
+-------------------+---------+
| mypipeline | Running |
+-------------------+---------+
If your pipeline is not in the running state, see the section Starting a Pipeline in the Foreground for further details.
Example 3: Starting a Pipeline in the Foreground with a Batch Limit
The following example demonstrates how to start a pipeline in the foreground with a specified limit, which displays the number of rows written to the destination table:
START PIPELINE mypipeline FOREGROUND LIMIT 5 BATCHES;
****
Query OK, 37 rows affected (0.71 sec)
To see if the pipeline is running, run SHOW PIPELINES:
SHOW PIPELINES;
****
+-------------------+---------+
| Pipelines_in_mydb | State |
+-------------------+---------+
| mypipeline | Running |
+-------------------+---------+
If your pipeline is not in the running state, see the section Starting a Pipeline in the Foreground for further details.
