Kafka

1. If a Kafka cluster is not already available, install Kafka on the collecting Master Aggregator host.

2. Create a new “<org-name>_metrics” Kafka topic, replacing <org-name> with your organization’s name, such as myorg_metrics.
   
   Replace <zookeeper-host>:<zookeeper-port> with the values of your Kafka configuration.

   ~/kafka/bin/kafka-topics.sh --create --zookeeper <zookeeper-host>:<zookeeper-port> --replication-factor 1 --partitions 1 --topic <org-name>_metrics
   ****
   WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both.
   Created topic "<org-name>_metrics".
   

3. Create a new “<org-name>_blobs” Kafka topic, replacing <org-name> with your organization’s name, such as myorg_blobs.
   
   Replace <zookeeper-host>:<zookeeper-port> with the values of your Kafka configuration.

   ~/kafka/bin/kafka-topics.sh --create --zookeeper <zookeeper-host>:<zookeeper-port> --replication-factor 1 --partitions 1 --topic <org-name>_blobs
   ****
   WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both.
   Created topic "<org-name>_blobs".
   

4. Optional: Enable security for both of these topics using SASL for username and password encryption and SSL for encryption over the wire.

5. Confirm that all Kafka brokers are accessible from all hosts in the monitored MemSQL cluster, and that the Kafka broker can be accessed from the collecting Master Aggregator host.
   
   Potential sources of issues can include improperly defined AWS security groups, iptables, and/or firewall settings.
   
   For each host in the cluster, attempt to telnet to the Kafka broker. Press Ctrl-C (^C) to close the connection.

   telnet <kafka-broker-host> <kafka-broker-port>
   ****
   Connected to xxx.xxx.xxx.xxx.
   Escape character is '^]'.
   ^CConnection closed by foreign host
   

Grafana

1. If a Grafana instance is not already available, install Grafana on the collecting Master Aggregator host.

2. Add the Grafana pie chart panel.

   sudo grafana-cli plugins install grafana-piechart-panel
   

3. Restart the Grafana server.

   sudo systemctl restart grafana-server
   

4. Add the Grafana multi-bar graph panel.

   sudo grafana-cli --pluginUrl https://github.com/CorpGlory/grafana-multibar-graph-panel/archive/0.2.5.zip plugins install multibar-graph-panel
   

5. Restart the Grafana server.

   sudo systemctl restart grafana-server
   

memsql_exporter

The memsql_exporter is a satellite process that collects data about a running cluster and funnels it to either memsql_pusher or to a Prometheus process which pulls from the memsql_exporter and sends it to a Kafka queue.

The memsql_exporter binary typically resides in memsql-server-<version>/memsql_exporter/.

Note: Perform the following steps on the monitored Master Aggregator.

1. Modify the /home/<user>/memsql/memsql-server-<version>/memsql_exporter/conf/memsql_exporter.cnf file by removing the last two [client] sections and adding the MemSQL password. This file describes the address and the user with which memsql_exporter will login and query the cluster.

   [client]
   host=localhost
   port=3306
   user=root
   password=<password, blank if none>
   database=information_schema
   

2. Start memsql_exporter. The memsql_exporter_ma.args file contains parameters for memsql_exporter that can be used to enable or disable specific functions.

   /home/<user>/memsql/memsql-server-<version>/memsql_exporter/memsql_exporter @/home/<user>/memsql/memsql-server-<version>/memsql_exporter/conf/memsql_exporter_ma.args --config.my-cnf=/home/<user>/memsql/memsql-server-<version>/memsql_exporter/conf/memsql_exporter.cnf
   

3. Confirm that memsql_exporter is running.

   ps -aux | grep memsql_exporter
   ****
   <user>   44384  0.0  0.0  18964 14120 pts/1    Sl+  07:47   0:02 /home/<user>/memsql/memsql-server-<version>/memsql_exporter/memsql_exporter @/home/<user>/memsql/memsql-server-<version>/memsql_exporter/conf/memsql_exporter_ma.args --config.my-cnf=/home/<user>/memsql/memsql-server-<version>/memsql_exporter/conf/memsql_exporter.cnf
   

memsql_pusher

The memsql_pusher is a satellite process that takes input from one or more exporters (such as memsql_exporter or other Prometheus ecosystem tools), processes it, and sends the collected data to the <org-name>_metrics and <org-name>_blobs Kafka topics created earlier. The <org-name> prefix is your organization’s name.

Note: Perform the following steps on the monitored Master Aggregator.

1. Merge the memsql_pusher.args file with the entries below, substituting the values in angle brackets with your MemSQL configuration.

   For MemSQL v7.0.10 and later: /home/<user>/memsql/memsql-server-<version>/memsql_exporter/conf/memsql_pusher.args

   If memsql_pusher was installed separately for MemSQL v7.0.9 and earlier: /home/<user>/memsql/memsql_pusher/conf/memsql_pusher.args

   --cluster.name=<analytic-cluster-name>
   --org=<org-name>
   --monitor.hosts=localhost
   --kafka.bootstrap=<kafka-broker-host>:<kafka-port>
   --kafka.protocol=plaintext
   

   Note: The --org=<org-name> parameter must match the topic prefix created earlier.

2. Start memsql_pusher.

   For MemSQL v7.0.10 and later:

   /home/<user>/memsql/memsql-server-<version>/memsql_exporter/memsql_pusher \
   @/home/<user>/memsql/memsql-server-<version>/memsql_exporter/conf/memsql_pusher.args
   

   If memsql_pusher was installed separately for MemSQL v7.0.9 and earlier:

   /home/<user>/memsql/memsql_pusher/memsql_pusher \
   @/home/<user>/memsql/memsql_pusher/conf/memsql_pusher.args
   

3. Confirm that memsql_pusher is running.

   ps -aux | grep memsql_pusher
   ****
   <user>   44461  1.0  0.1 1165516 26412 pts/0   Sl+  13:41   0:35 /home/<user>/memsql/memsql_pusher/memsql_pusher @/home/<user>/memsql/memsql_pusher/conf/memsql_pusher.args
   

Was this Article Helpful?

Thanks For Your Feedback!