You are viewing an older version of this section. View current production version.

Configure SingleStore DB Core Dumps

A core dump can be generated for a single pod, or for all pods in the SingleStore DB cluster.

Enable Core Dumps

Core dumps can be enabled on individual pods, or across all pods.

Individual Pods

  1. List all pods in the cluster.

    kubectl get pods -l app.kubernetes.io/name=memsql-cluster,app.kubernetes.io/instance=<cluster-name>
    
  2. Confirm that a core dump will be produced upon crash.

    kubectl exec <pod-name> -c node -- memsqlctl query --sql "SELECT @@core_file"
    ****
    +-------------+
    | @@core_file |
    +-------------+
    | 1           |
    +-------------+
    

    The default value for @@core_file is 1, which indicates that creating a core dump on crash is enabled. If this value is 0, creating a core dump on crash is disabled. To enable it, run the following command.

    kubectl exec <pod-name> -c node -- memsqlctl -y update-config --key core_file --value ON
    

    Restart the node for this change to take effect.

    kubectl exec <pod-name> -c node -- memsqlctl -y restart-node
    
  3. Configure the core dump to be either partial or full. By default, a partial core dump is created. Refer to the core_file_mode engine variable for more information on partial and full core dumps.

    Run the following command to check the current value.

    kubectl exec node-memsql-cluster-master-0 -c node -- memsqlctl query --sql "SELECT @@core_file_mode"
    ****
    +------------------+
    | @@core_file_mode |
    +------------------+
    | PARTIAL          |
    +------------------+
    

    Run the following command to change this value.

    For a partial core dump:

    kubectl exec <pod-namne> -c node -- memsqlctl -y update-config --key core_file_mode --value PARTIAL
    

    For a full core dump:

    kubectl exec <pod-name> -c node -- memsqlctl -y update-config --key core_file_mode --value FULL
    

    Restart the node for this change to take effect.

    kubectl exec <pod-name> -c node -- memsqlctl -y restart-node
    

Across all Pods

  1. To make the above core dump changes across all pods, add the following subsection to the memsql-cluster.yaml file’s spec section. Refer to the core_file_mode engine variable for more information on partial and full core dumps.

    globalVariables:
      core_file: "ON"
      core_file_mode: "FULL"    # or "PARTIAL", depending on your goal
    
  2. Apply these values to enable core dumps on all pods.

    kubectl apply -f memsql-cluster.yaml
    

Generate Core Dumps

  1. Display the pods and the nodes they are on. Note that the following output may vary.

    kubectl get pods -l app.kubernetes.io/name=memsql-cluster,app.kubernetes.io/instance=<cluster-name> -o wide
    ****
    NAME                               READY   STATUS    RESTARTS   AGE     IP          NODE                                   NOMINATED NODE   READINESS GATES
    node-memsql-cluster-aggregator-0   2/2     Running   0          4h13m   10.4.0.20   gke-foo-1-default-pool-ed7b330b-2q1r   <none>           <none>
    node-memsql-cluster-leaf-ag1-0     2/2     Running   0          4h13m   10.4.3.10   gke-foo-1-default-pool-ed7b330b-w12w   <none>           <none>
    node-memsql-cluster-leaf-ag2-0     2/2     Running   0          4h13m   10.4.1.8    gke-foo-1-default-pool-ed7b330b-64r5   <none>           <none>
    node-memsql-cluster-master-0       2/2     Running   0          4h13m   10.4.4.24   gke-foo-1-default-pool-ed7b330b-klnz   <none>           <none>
    

Using an SSH Connection

Note: These steps affect all core dump generation for processes on the nodes, not just SingleStore DB containers on the nodes.

  1. For each node that contains the pods you want to generate a core dump for, SSH into the node and run the following command.

    echo "core.%d" > /proc/sys/kernel/core_pattern
    

Using a DaemonSet

  1. Create a memsql-coredump.yaml file using the following template.

    apiVersion: apps/v1
    kind: DaemonSet
    metadata:
      name: memsql-sysreq
      labels:
        "app.kubernetes.io/name": "memsql-sysreq"
    spec:
      selector:
        matchLabels:
          "app.kubernetes.io/name": "memsql-sysreq"
      updateStrategy:
        type: RollingUpdate
      template:
        metadata:
          labels:
            "app.kubernetes.io/name": "memsql-sysreq"
        spec:
          hostPID: true
          volumes:
          - name: sys
            hostPath:
              path: /sys
          affinity:
            podAffinity:
              requiredDuringSchedulingIgnoredDuringExecution:
              - labelSelector:
                  matchLabels:
                    app.kubernetes.io/instance: CLUSTER_NAME
                    app.kubernetes.io/name: memsql-cluster
                       
                    #Add this line if you only want to generate core for Master Aggregator
                    #app.kubernetes.io/component: master
                    #change master to aggregator or leaf if you only want to generate core
                    #for child aggregator or for leaf
       
                    #Add this line if you only want to generate core for one particular pod, 
                    #substitute POD_NAME with the name of the pod
                    #statefulset.kubernetes.io/pod-name: POD_NAME
                topologyKey: kubernetes.io/hostname
       
          initContainers:
          - name: sysreq
            image: busybox
            securityContext:
              privileged: true
            volumeMounts:
            - name: sys
              mountPath: /rootfs/sys
            command:
            - "/bin/sh"
            - "-c"
            - |
              echo “core.%d” > /proc/sys/kernel/core_pattern
       
          containers:
          - name: pause
            image: gcr.io/google_containers/pause
    
  2. Edit the following placeholders in this file:

    • Change CLUSTER_NAME to the SingleStore DB cluster name.
    • To generate the core dump in a specific pod, change POD_NAME to the desired pod.
  3. Enable the core dump(s) at the Kubernetes host level. Note that you must have cluster admin privileges to run this command.

    kubectl apply -f memsql-coredump.yaml
    

Core Dump Location

Given the above configuration, and should the SingleStore DB server crash, a core dump will be written to /var/lib/memsql/instance/data/core.memsqld in the restarted pod.