Outdated Version

You are viewing an older version of this section. View current production version.

calibrate

Run ‘calibrate’ to measure cluster performance.

Usage

Run 'calibrate' to measure cluster performance.

A 'calibrate' database is created with rowstore and columnstore tables that are populated with data. A number of queries are run against this data to measure cluster performance. 

The 'calibrate' tool requires a dataset, which will be loaded into tables. By default, the dataset is downloaded from the web, unpacked, and deleted at the end of the run. However, if your cluster cannot access the internet or already has the dataset, use the '--data-path' flag, which is an absolute path to the dataset on the Master Aggregator node. 
You can download the dataset here: https://s3.amazonaws.com/calibrate.singlestore.com/calibrate-dataset.tar.gz

The dataset download uses 1.5 Gb of disk space, and the unpacking of data takes up another 12 Gb.

The '--partition-ratio' flag is used to test clusters under different concurrency conditions, such as, four CPU cores per partition. This flag specifies a proportion of CPU cores to Partitions and defaults to 2:1. 

Workload consists of the following steps:
 - Download the dataset
 - Create the 'calibrate' database and set up the required variables
 - Load data into tables
 - Run calibration queries
 - Retrieve results of the run and save them in a file on the Master Aggregator node

Workload duration divided into three main categories:
 - FAST: under 12 minutes
 - INTERMEDIATE: between 12 and 16 minutes
 - SLOW: over 16 minutes

In case the cluster gets 'SLOW' or 'INTERMEDIATE' result, the tool will print data about the run, in addition to the file version, into the terminal.

Examples:

# A simple example that runs 'calibrate' workload with the root user of the cluster and downloads dataset from the internet
sdb-report calibrate

# Runs 'calibrate' workload with the user 'user' and password 'pass' with a dataset on path '/home/admin/dataset.tar.gz' on the Master Aggregator node
sdb-report calibrate -u usr -p pass --data-path /home/admin/dataset.tar.gz

# Run calibrate to test non-default partition ratio and retain calibrate database after the run
sdb-report calibrate --partition-ratio 4:1 --retain-database

Execution time of the following queries will be measured:

Load data queries:

+--------------+------------------------------------------------------------------------------------------------------+
|     NAME     |                                                 SQL                                                  |
+--------------+------------------------------------------------------------------------------------------------------+
| Load Queries | LOAD /*! calibrate_load_rs_1 */ DATA INFILE '{{CALIBRATE_LOAD_RS_1}}' INTO TABLE disttable2 FIELDS   |
|              | TERMINATED BY ',' OPTIONALLY ENCLOSED BY '\"' LINES TERMINATED BY '\n';                              |
|              |                                                                                                      |
|              | LOAD /*! calibrate_load_cs_2 */ DATA INFILE '{{CALIBRATE_LOAD_CS_2}}' INTO TABLE disttable2_cs       |
|              | FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '\"' LINES TERMINATED BY '\n';                       |
|              |                                                                                                      |
|              | LOAD /*! calibrate_load_rs_3 */ DATA INFILE '{{CALIBRATE_LOAD_RS_3}}' INTO TABLE foreignstring440k   |
|              | FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '\"' LINES TERMINATED BY '\n';                       |
|              |                                                                                                      |
|              | LOAD /*! calibrate_load_cs_4 */ DATA INFILE '{{CALIBRATE_LOAD_CS_4}}' INTO TABLE                     |
|              | foreignstring440k_cs FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '\"' LINES TERMINATED BY '\n';  |
|              |                                                                                                      |
|              | LOAD /*! calibrate_load_rs_5 */ DATA INFILE '{{CALIBRATE_LOAD_RS_5}}' INTO TABLE primarystring2      |
|              | FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '\"' LINES TERMINATED BY '\n';                       |
|              |                                                                                                      |
|              | LOAD /*! calibrate_load_cs_6 */ DATA INFILE '{{CALIBRATE_LOAD_CS_6}}' INTO TABLE primarystring2_cs   |
|              | FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '\"' LINES TERMINATED BY '\n';                       |
+--------------+------------------------------------------------------------------------------------------------------+


Calibration queries:

+--------------------------------+------------------------------------------------------------------------------------------------------+
|              NAME              |                                                 SQL                                                  |
+--------------------------------+------------------------------------------------------------------------------------------------------+
| Reshuffle Joins                | SELECT /*! calibrate_reshuff_rs_q1    */ COUNT(1) FROM ( SELECT b.* FROM  disttable2 a STRAIGHT_JOIN |
|                                | primarystring2 b ON a.a=b.a) A;                                                                      |
|                                |                                                                                                      |
|                                | SELECT /*! calibrate_reshuff_rs_cs_q2 */ COUNT(1) FROM ( SELECT b.* FROM  disttable2 a STRAIGHT_JOIN |
|                                |  primarystring2_cs b ON a.a=b.a) A;                                                                  |
|                                |                                                                                                      |
|                                | SELECT /*! calibrate_reshuff_rs_cs_q3 */ COUNT(1) FROM ( SELECT b.* FROM  disttable2_cs a            |
|                                | STRAIGHT_JOIN  primarystring2 b ON a.a=b.a) A;                                                       |
|                                |                                                                                                      |
|                                | SELECT /*! calibrate_reshuff_cs_q4    */ COUNT(1) FROM ( SELECT b.* FROM  disttable2_cs a            |
|                                | STRAIGHT_JOIN  primarystring2_cs b ON a.a=b.a) A;                                                    |
|                                |                                                                                                      |
| Broadcast Joins                | SELECT /*! calibrate_bcast_rs_q1     */ COUNT(1) FROM ( SELECT b.* FROM disttable2 a  STRAIGHT_JOIN  |
|                                | WITH(broadcast_left=true) primarystring2 b ON a.a=b.a) A;                                            |
|                                |                                                                                                      |
|                                | SELECT /*! calibrate_bcast_rs_cs_q2  */ COUNT(1) FROM ( SELECT b.* FROM disttable2 a  STRAIGHT_JOIN  |
|                                | WITH(broadcast_left=true) primarystring2_cs b ON a.a=b.a) A;                                         |
|                                |                                                                                                      |
|                                | SELECT /*! calibrate_bcast_rs_cs_q3  */ COUNT(1) FROM ( SELECT b.* FROM disttable2_cs a              |
|                                | STRAIGHT_JOIN WITH(broadcast_left=true) primarystring2 b ON a.a=b.a) A;                              |
|                                |                                                                                                      |
|                                | SELECT /*! calibrate_bcast_cs_q4     */ COUNT(1) FROM ( SELECT b.* FROM disttable2_cs a              |
|                                | STRAIGHT_JOIN WITH(broadcast_left=true) primarystring2_cs b ON a.a=b.a) A;                           |
|                                |                                                                                                      |
| Local Group By                 | SELECT /*! calibrate_l_gby_rs_q1     */ WITH(leaf_pushdown=true) COUNT(1) FROM (SELECT               |
|                                | COUNT(foreignstring440k.a),primarystring2.a FROM foreignstring440k STRAIGHT_JOIN primarystring2      |
|                                | WITH(table_convert_subSELECT=true) ON primarystring2.a=foreignstring440k.a GROUP BY                  |
|                                | primarystring2.a) A;                                                                                 |
|                                |                                                                                                      |
|                                | SELECT /*! calibrate_l_gby_rs_cs_q2  */ WITH(leaf_pushdown=true) COUNT(1) FROM (SELECT               |
|                                | COUNT(foreignstring440k_cs.a),primarystring2.a FROM foreignstring440k_cs STRAIGHT_JOIN               |
|                                | primarystring2 WITH(table_convert_subSELECT=true) ON primarystring2.a=foreignstring440k_cs.a GROUP   |
|                                | BY primarystring2.a) A;                                                                              |
|                                |                                                                                                      |
|                                | SELECT /*! calibrate_l_gby_rs_cs_q3  */ WITH(leaf_pushdown=true) COUNT(1) FROM (SELECT               |
|                                | COUNT(foreignstring440k.a),primarystring2_cs.a FROM foreignstring440k STRAIGHT_JOIN                  |
|                                | primarystring2_cs WITH(table_convert_subSELECT=true) ON primarystring2_cs.a=foreignstring440k.a      |
|                                | GROUP BY primarystring2_cs.a) A;                                                                     |
|                                |                                                                                                      |
|                                | SELECT /*! calibrate_l_gby_cs_q4     */ WITH(leaf_pushdown=true) COUNT(1) FROM (SELECT               |
|                                | COUNT(foreignstring440k_cs.a),primarystring2_cs.a FROM foreignstring440k_cs STRAIGHT_JOIN            |
|                                | primarystring2_cs WITH(table_convert_subSELECT=true) ON primarystring2_cs.a=foreignstring440k_cs.a   |
|                                | GROUP BY primarystring2_cs.a) A;                                                                     |
|                                |                                                                                                      |
| Local Joins                    | SELECT /*! calibrate_lj_rs_q1       */ COUNT(1) FROM foreignstring440k STRAIGHT_JOIN primarystring2  |
|                                | WITH(table_convert_subSELECT=true) ON primarystring2.a=foreignstring440k.a;                          |
|                                |                                                                                                      |
|                                | SELECT /*! calibrate_lj_rs_cs_q2    */ COUNT(1) FROM foreignstring440k_cs STRAIGHT_JOIN              |
|                                | primarystring2 WITH(table_convert_subSELECT=true) ON primarystring2.a=foreignstring440k_cs.a;        |
|                                |                                                                                                      |
|                                | SELECT /*! calibrate_lj_rs_cs_q3    */ COUNT(1) FROM foreignstring440k STRAIGHT_JOIN                 |
|                                | primarystring2_cs WITH(table_convert_subSELECT=true) ON primarystring2_cs.a=foreignstring440k.a;     |
|                                |                                                                                                      |
|                                | SELECT /*! calibrate_lj_cs_q4       */ COUNT(1) FROM foreignstring440k_cs STRAIGHT_JOIN              |
|                                | primarystring2_cs WITH(table_convert_subSELECT=true) ON primarystring2_cs.a=foreignstring440k_cs.a;  |
|                                |                                                                                                      |
| Single Part Join and Filter    | SELECT /*! calibrate_spj_rs_q1     */ COUNT(1) FROM foreignstring440k STRAIGHT_JOIN primarystring2   |
|                                | ON primarystring2.a=foreignstring440k.a AND primarystring2.a = 'A1000027';                           |
|                                |                                                                                                      |
|                                | SELECT /*! calibrate_spj_rs_cs_q2  */ COUNT(1) FROM foreignstring440k_cs STRAIGHT_JOIN               |
|                                | primarystring2  ON primarystring2.a=foreignstring440k_cs.a AND primarystring2.a = 'A1000027';        |
|                                |                                                                                                      |
|                                | SELECT /*! calibrate_spj_rs_cs_q3  */ COUNT(1) FROM foreignstring440k STRAIGHT_JOIN                  |
|                                | primarystring2_cs  ON primarystring2_cs.a=foreignstring440k.a AND primarystring2_cs.a = 'A1000027';  |
|                                |                                                                                                      |
|                                | SELECT /*! calibrate_spj_cs_q4     */ COUNT(1) FROM foreignstring440k_cs STRAIGHT_JOIN               |
|                                | primarystring2_cs  ON primarystring2_cs.a=foreignstring440k_cs.a AND primarystring2_cs.a =           |
|                                | 'A1000027';                                                                                          |
|                                |                                                                                                      |
| Reshuffle and Distributed      | SELECT /*! calibrate_reshuff_dgby_rs_q1    */ WITH(leaf_pushdown=true) COUNT(1) FROM (SELECT         |
| Group By                       | COUNT(b.a) FROM disttable2 a STRAIGHT_JOIN  primarystring2 b ON a.a=b.a GROUP BY a.b) A;             |
|                                |                                                                                                      |
|                                | SELECT /*! calibrate_reshuff_dgby_rs_cs_q2 */ WITH(leaf_pushdown=true) COUNT(1) FROM (SELECT         |
|                                | COUNT(b.a) FROM disttable2_cs a STRAIGHT_JOIN  primarystring2 b ON a.a=b.a GROUP BY a.b) A;          |
|                                |                                                                                                      |
|                                | SELECT /*! calibrate_reshuff_dgby_rs_cs_q3 */ WITH(leaf_pushdown=true) COUNT(1) FROM (SELECT         |
|                                | COUNT(b.a) FROM disttable2 a STRAIGHT_JOIN  primarystring2_cs b ON a.a=b.a GROUP BY a.b) A;          |
|                                |                                                                                                      |
|                                | SELECT /*! calibrate_reshuff_dgby_cs_q4    */ WITH(leaf_pushdown=true) COUNT(1) FROM (SELECT         |
|                                | COUNT(b.a) FROM disttable2_cs a STRAIGHT_JOIN  primarystring2_cs b ON a.a=b.a GROUP BY a.b) A;       |
|                                |                                                                                                      |
| Local Join and Distributed     | SELECT /*! calibrate_lj_dgby_rs_q1     */ WITH(leaf_pushdown=true) COUNT(1) FROM (SELECT             |
| Group By                       | COUNT(foreignstring440k.a),primarystring2.id FROM foreignstring440k STRAIGHT_JOIN primarystring2     |
|                                | WITH(table_convert_subSELECT=true) ON primarystring2.a=foreignstring440k.a GROUP BY                  |
|                                | primarystring2.id) A;                                                                                |
|                                |                                                                                                      |
|                                | SELECT /*! calibrate_lj_dgby_rs_cs_q2  */ WITH(leaf_pushdown=true) COUNT(1) FROM (SELECT             |
|                                |  COUNT(foreignstring440k_cs.a),primarystring2.id FROM foreignstring440k_cs STRAIGHT_JOIN             |
|                                | primarystring2 WITH(table_convert_subSELECT=true) ON primarystring2.a=foreignstring440k_cs.a GROUP   |
|                                | BY primarystring2.id) A;                                                                             |
|                                |                                                                                                      |
|                                | SELECT /*! calibrate_lj_dgby_rs_cs_q3  */ WITH(leaf_pushdown=true) COUNT(1) FROM (SELECT             |
|                                |  COUNT(foreignstring440k.a),primarystring2_cs.id FROM foreignstring440k STRAIGHT_JOIN                |
|                                | primarystring2_cs WITH(table_convert_subSELECT=true) ON primarystring2_cs.a=foreignstring440k.a      |
|                                | GROUP BY primarystring2_cs.id) A;                                                                    |
|                                |                                                                                                      |
|                                | SELECT /*! calibrate_lj_dgby_cs_q4     */ WITH(leaf_pushdown=true) COUNT(1) FROM (SELECT             |
|                                | COUNT(foreignstring440k_cs.a),primarystring2_cs.id FROM foreignstring440k_cs STRAIGHT_JOIN           |
|                                | primarystring2_cs WITH(table_convert_subSELECT=true) ON primarystring2_cs.a=foreignstring440k_cs.a   |
|                                | GROUP BY primarystring2_cs.id) A;                                                                    |
+--------------------------------+------------------------------------------------------------------------------------------------------+

Usage:
  sdb-report calibrate [flags]

Flags:
      --data-path string         The absolute path to the folder on the Master Aggregator that contains the calibration datasets
  -h, --help                     Help for calibrate
      --partition-ratio RATIO    The ratio of CPU cores to database partitions (CPU cores:database partitions) where both values must be less than or equal to 16 (default 2:1)
  -p, --password STRING          The database user's password for connecting to SingleStoreDB. If a password is specified on the command line, it must not contain an unescaped '$' character as it will be replaced by the shell
      --retain-database          Retain the ‘calibrate’ database after the calibration process completes
      --temp-dir ABSOLUTE_PATH   The directory on the Master Aggregator in which to unpack the dataset (ADVANCED)
  -u, --user string              The database user for connecting to SingleStoreDB (default "root")

Global Flags:
      --backup-cache FILE_PATH                File path for the backup cache
      --cache-file FILE_PATH                  File path for the Toolbox node cache
  -c, --config FILE_PATH                      File path for the Toolbox configuration
      --disable-colors                        Disable color output in console, which some terminal sessions/environments may have difficulty with
      --disable-spinner                       Disable the progress spinner, which some terminal sessions/environments may have issues with
  -j, --json                                  Enable JSON output
      --parallelism POSITIVE_INTEGER          Maximum number of operations to run in parallel
      --runtime-dir DIRECTORY_PATH            Where to store Toolbox runtime data
      --ssh-control-persist SECONDS           Enable SSH ControlPersist and set it to the specified duration in seconds
      --ssh-max-sessions POSITIVE_INTEGER     Maximum number of SSH sessions to open per host, must be at least 3
      --ssh-strict-host-key-checking          Enable strict host key checking for SSH connections
      --ssh-user-known-hosts-file FILE_PATH   Path to the user known_hosts file for SSH connections. If not set, /dev/null will be used
      --state-file FILE_PATH                  Toolbox state file path
  -v, --verbosity count                       Increase logging verbosity: valid values are 1, 2, 3. Usage -v=count or --verbosity=count
  -y, --yes                                   Enable non-interactive mode and assume the user would like to move forward with the proposed actions by default

Remarks

This command is interactive unless you use either the --yes or --json flags to override interactive behavior.