Enabling SSL and Kerberos on Kafka Pipelines

In MemSQL 6.5 and later, you can connect a pipeline to a Kafka cluster through SSL and optionally authenticate with Kerberos through SASL. This topic assumes you have already have set up, configured, and enabled SSL and/or Kerberos on your Kafka brokers. For information on how to enable this functionality, see the SSL and SASL sections in the Kafka documentation.

Like pipelines such as S3 or Azure, credentials are passed in through JSON in the CREATE PIPELINE statement. Any credentials used to encrypt or authenticate must be present on each node in the MemSQL cluster.

The security.protocol credential specifies the encryption and authentication mechanisms used to connect to Kafka brokers. This property can be one of four values: plaintext, ssl, sasl_plaintext, or sasl_ssl. Depending on this value, you may have to supply additional credentials in your JSON.

Connecting over SSL

To enable SSL encryption between Kafka and MemSQL, perform the following steps:

1. Securely copy the CA certificate, SSL certificate, and SSL key used for connections between the MemSQL cluster and Kafka brokers from the Kafka cluster to every MemSQL node. You should use a secure file transfer method, such as scp, to copy the files to your MemSQL nodes. The file locations on your MemSQL nodes should be consistent across the cluster.

2. In your CREDENTIALS JSON, if you want to enable SSL encryption only, set "security.protocol": "ssl". If you want to enable Kerberos with SSL, set "security.protocol": "sasl_ssl" and set the Kerberos credentials after you’ve completed step three.

3. Set the remaining SSL credentials:

   • ssl.ca.location: Path to the CA certificate on the MemSQL node.
   • ssl.certificate.location: Path to the SSL certificate on the MemSQL node.
   • ssl.key.location: Path to the SSL certificate key on the MemSQL node.
   • ssl.key.password: Password for SSL certificate key.

Authenticating with Kerberos

To configure a Kafka pipeline to authenticate with Kerberos, you must configure all of your nodes in your MemSQL cluster as clients for Kerberos authentication and then set the credentials in your CREATE PIPELINE statement. To do this, perform the following steps:

1. Securely copy the keytab file containing the MemSQL service principal (e.g. memsql/host.domain.com@REALM.NAME) from the Kerberos server to every node in your MemSQL cluster. You should use a secure file transfer method, such as scp, to copy the keytab file to your MemSQL nodes. The file location on your MemSQL nodes should be consistent across the cluster.

2. Make sure your MemSQL nodes can connect to the KDC server using the fully-qualified domain name (FQDN) of the KDC server. This might require configuring network settings or updating /etc/hosts on your nodes.

3. Also ensure that the memsql service account on the node can access the copied keytab file. This can be accomplished by changing file ownership or permissions. If the memsql account cannot access the keytab file, you will not be able to complete the next step because your master aggregator will not be able to restart after applying configuration updates.

4. When authenticating with Kerberos, MemSQL needs to authenticate as a client, which means you must also install a Kerberos client onto each node in your cluster. The following installs the krb5-user package for Debian-based Linux distributions.

   $ sudo apt-get update && apt-get install krb5-user
   

   When setting up your kerberos configuration settings, set your default realm, Kerberos admin server, and other options to those defined by your KDC server. In the examples used in this topic, the default realm is EXAMPLE.COM, and the Kerberos server settings are set to the FQDN of the KDC server host.example.com.

5. In your CREDENTIALS JSON, set "sasl.mechanism": "GSSAPI".

6. Set "security.protocol": "sasl_ssl" for Kerberos and SSL connections, or "security.protocol": "sasl_plaintext" if you want to authenticate with Kerberos without SSL encryption. If you want to use Kerberos with SSL, make sure SSL is configured and enabled on the Kafka brokers and then add the SSL credential properties defined in the previous section.

7. Set the remaining Kerberos credentials:

   • sasl.kerberos.service.name: The Kerberos principal name that Kafka runs as. For example, "kafka".
   • sasl.kerberos.keytab: The local file path on the MemSQL node to the authenticating keytab.
   • sasl.kerberos.principal: The service principal name for the MemSQL cluster. For example, "memsql/host.example.com@EXAMPLE.COM".

Examples

The following examples make the following assumptions:

• Port 9092 is a plaintext endpoint
• Port 9093 is an SSL endpoint
• Port 9094 is a plaintext Kerberos endpoint
• Port 9095 is an SSL Kerberos endpoint

Plaintext

The following CREATE PIPELINE statements are equivalent:

CREATE PIPELINE `kafka_plaintext`
AS LOAD DATA KAFKA 'host.example.com:9092/test'
CREDENTIALS '{"security.protocol": "plaintext"}'
INTO table t;

CREATE PIPELINE `kafka_no_creds`
AS LOAD DATA KAFKA 'host.example.com:9092/test'
INTO table t;

SSL

CREATE PIPELINE `kafka_ssl`
AS LOAD DATA KAFKA 'host.example.com:9093/test'
CREDENTIALS '{"security.protocol": "ssl",
"ssl.key.password": "abcdefgh",
"ssl.certificate.location": "/var/private/ssl/client_memsql_client.pem",
"ssl.key.location": "/var/private/ssl/client_memsql_client.key",
"ssl.ca.location": "/var/private/ssl/ca-cert.pem"}'
INTO table t;

Kerberos with no SSL

CREATE PIPELINE `kafka_kerberos_no_ssl`
AS LOAD DATA KAFKA 'host.example.com:9094/test'
CREDENTIALS '{""security.protocol": "sasl_plaintext",
"sasl.mechanism": "GSSAPI",
"sasl.kerberos.service.name": "kafka",  
"sasl.kerberos.principal": "memsql/host.example.com@EXAMPLE.COM",
"sasl.kerberos.keytab": "/etc/krb5.keytab"}'
INTO table t;

Kerberos with SSL

CREATE PIPELINE `kafka_kerberos_ssl`
AS LOAD DATA KAFKA 'host.example.com:9095/test'
CREDENTIALS '{"security.protocol": "sasl_ssl",
"sasl.mechanism": "GSSAPI",
"ssl.key.password": "abcdefgh",
"ssl.certificate.location": "/var/private/ssl/client_memsql_client.pem",
"ssl.key.location": "/var/private/ssl/client_memsql_client.key",
"ssl.ca.location": "/var/private/ssl/ca-cert.pem",
"sasl.kerberos.service.name": "kafka",
"sasl.kerberos.principal": "memsql/host.example.com@EXAMPLE.COM",
"sasl.kerberos.keytab": "/etc/krb5.keytab"}'
INTO table t;

Was this Article Helpful?

Thanks For Your Feedback!