github.com/apache/rocketmq-operator @0.3.0

RocketMQ Operator

Table of Contents

• Overview
• Quick Start
• Deploy RocketMQ Operator
• Prepare Volume Persistence
  • EmptyDir
  • HostPath
  • StorageClass (Use NFS for Example)
• Define Your RocketMQ Cluster
• Create RocketMQ Cluster
• Verify the Data Storage
  • Verify HostPath Storage
  • Verify NFS storage
• Horizontal Scale
• Name Server Cluster Scale
• Broker Cluster Scale
  • Up-scale Broker in Out-of-order Message Scenario
• Topic Transfer
• Clean the Environment
• Development
• Prerequisites
• Build
  • Operator
  • Broker and Name Server Images
  • Dashboard

Overview

RocketMQ Operator is to manage RocketMQ service instances deployed on the Kubernetes cluster. It is built using the Operator SDK, which is part of the Operator Framework.

Quick Start

Deploy RocketMQ Operator

1. Clone the project on your Kubernetes cluster master node:

$ git clone https://github.com/apache/rocketmq-operator.git
$ cd rocketmq-operator

1. To deploy the RocketMQ Operator on your Kubernetes cluster, please run the following command:

$ make deploy

1. Use command kubectl get pods to check the RocketMQ Operator deploy status like:

$ kubectl get pods
NAME                                      READY   STATUS    RESTARTS   AGE
rocketmq-operator-564b5d75d-jllzk         1/1     Running   0          108s

If you find that pod image is not found, run the following command to build a new one locally, the image tag is specified by the IMG parameter.

$ make docker-build IMG=apache/rocketmq-operator:0.3.0

Now you can use the CRDs provided by RocketMQ Operator to deploy your RocketMQ cluster.

Prepare Volume Persistence

Before RocketMQ deployment, you may need to do some preparation steps for RocketMQ data persistence.

Currently we provide several options for your RocketMQ data persistence: EmptyDir, HostPath and StorageClass, which can be configured in CR files, for example in rocketmq_v1alpha1_nameservice_cr.yaml:

...
 # storageMode can be EmptyDir, HostPath, StorageClass
  storageMode: HostPath
...

EmptyDir

If you choose EmptyDir, you don't need to do extra preparation steps for data persistence. But the data storage life is the same with the pod's life, if the pod is deleted you may lost the data.

If you choose other storage modes, please refer to the following instructions to prepare the data persistence.

HostPath

This storage mode means the RocketMQ data (including all the logs and store files) is stored in each host where the pod lies on. You need to create a directory on the host where you want the RocketMQ data to be stored. For example:

$ mkdir /data/rocketmq/broker

You can configure the host path in the CRD yaml file like hostPath: /data/rocketmq/broker in the example/rocketmq_v1alpha1_rocketmq_cluster.yaml file.

StorageClass (Use NFS for Example)

If you choose StorageClass as the storage mode, you need to prepare the storage class related provisioner and other dependencies. Using the NFS storage class as an example, the first step is to prepare a storage class based on NFS provider to create PV and PVC where the RocketMQ data will be stored.

1. Deploy NFS server and clients on your Kubernetes cluster. You can refer to NFS deployment document for more details. Please make sure they are functional before you go to the next step. Here is a instruction on how to verify NFS service.

   1) On your NFS client node, check if NFS shared dir exists. $ showmount -e 192.168.130.32 Export list for 192.168.130.32: /data/k8s * 2) On your NFS client node, create a test dir and mount it to the NFS shared dir (you may need sudo permission). $ mkdir -p ~/test-nfc $ mount -t nfs 192.168.130.32:/data/k8s ~/test-nfc 3) On your NFS client node, create a test file on the mounted test dir. $ touch ~/test-nfc/test.txt 4) On your NFS server node, check the shared dir. If there exists the test file we created on the client node, it proves the NFS service is functional. $ ls -ls /data/k8s/ total 4 4 -rw-r--r--. 1 root root 4 Jul 10 21:50 test.txt

2. Modify the following configurations of the deploy/storage/nfs-client.yaml file:

...
            - name: NFS_SERVER
              value: 192.168.130.32
            - name: NFS_PATH
              value: /data/k8s
      volumes:
        - name: nfs-client-root
          nfs:
            server: 192.168.130.32
            path: /data/k8s
...

Replace 192.168.130.32 and /data/k8s with your true NFS server IP address and NFS server data volume path.

1. Create a NFS storage class for RocketMQ, run

$ cd deploy/storage
$ ./deploy-storage-class.sh

1. If the storage class is successfully deployed, you can get the pod status like:

$ kubectl get pods
NAME                                      READY   STATUS    RESTARTS   AGE
nfs-client-provisioner-7cf858f754-7vxmm   1/1     Running   0          136m
rocketmq-operator-564b5d75d-jllzk         1/1     Running   0          108s

Define Your RocketMQ Cluster

RocketMQ Operator provides several CRDs to allow users define their RocketMQ service component cluster, which includes the Name Server, Broker cluster, Console, etc.

1. Check the file rocketmq_v1alpha1_rocketmq_cluster.yaml in the example directory which we put these CR together:

apiVersion: v1
kind: ConfigMap
metadata:
  name: broker-config
  namespace: default
data:
  # BROKER_MEM sets the broker JVM, if set to "" then Xms = Xmx = max(min(1/2 ram, 1024MB), min(1/4 ram, 8GB))
  BROKER_MEM: " -Xms2g -Xmx2g -Xmn1g "
  broker-common.conf: |
    # brokerClusterName, brokerName, brokerId are automatically generated by the operator and do not set it manually!!!
    deleteWhen=04
    fileReservedTime=48
    flushDiskType=ASYNC_FLUSH
    # set brokerRole to ASYNC_MASTER or SYNC_MASTER. DO NOT set to SLAVE because the replica instance will automatically be set!!!
    brokerRole=ASYNC_MASTER

---
apiVersion: rocketmq.apache.org/v1alpha1
kind: Broker
metadata:
  # name of broker cluster
  name: broker
  namespace: default
spec:
  # size is the number of the broker cluster, each broker cluster contains a master broker and [replicaPerGroup] replica brokers.
  size: 1
  # nameServers is the [ip:port] list of name service
  nameServers: ""
  # replicaPerGroup is the number of each broker cluster
  replicaPerGroup: 1
  # brokerImage is the customized docker image repo of the RocketMQ broker
  brokerImage: apacherocketmq/rocketmq-broker:4.5.0-alpine-operator-0.3.0
  # imagePullPolicy is the image pull policy
  imagePullPolicy: Always
  # resources describes the compute resource requirements and limits
  resources:
    requests:
      memory: "2048Mi"
      cpu: "250m"
    limits:
      memory: "12288Mi"
      cpu: "500m"
  # allowRestart defines whether allow pod restart
  allowRestart: true
  # storageMode can be EmptyDir, HostPath, StorageClass
  storageMode: EmptyDir
  # hostPath is the local path to store data
  hostPath: /data/rocketmq/broker
  # scalePodName is [Broker name]-[broker group number]-master-0
  scalePodName: broker-0-master-0
  # env defines custom env, e.g. BROKER_MEM
  env:
    - name: BROKER_MEM
      valueFrom:
        configMapKeyRef:
          name: broker-config
          key: BROKER_MEM
  # volumes defines the broker.conf
  volumes:
    - name: broker-config
      configMap:
        name: broker-config
        items:
          - key: broker-common.conf
            path: broker-common.conf
  # volumeClaimTemplates defines the storageClass
  volumeClaimTemplates:
    - metadata:
        name: broker-storage
      spec:
        accessModes:
          - ReadWriteOnce
        storageClassName: rocketmq-storage
        resources:
          requests:
            storage: 8Gi
---
apiVersion: rocketmq.apache.org/v1alpha1
kind: NameService
metadata:
  name: name-service
  namespace: default
spec:
  # size is the the name service instance number of the name service cluster
  size: 1
  # nameServiceImage is the customized docker image repo of the RocketMQ name service
  nameServiceImage: apacherocketmq/rocketmq-nameserver:4.5.0-alpine-operator-0.3.0
  # imagePullPolicy is the image pull policy
  imagePullPolicy: Always
  # hostNetwork can be true or false
  hostNetwork: true
  #  Set DNS policy for the pod.
  #  Defaults to "ClusterFirst".
  #  Valid values are 'ClusterFirstWithHostNet', 'ClusterFirst', 'Default' or 'None'.
  #  DNS parameters given in DNSConfig will be merged with the policy selected with DNSPolicy.
  #  To have DNS options set along with hostNetwork, you have to specify DNS policy
  #  explicitly to 'ClusterFirstWithHostNet'.
  dnsPolicy: ClusterFirstWithHostNet
  # resources describes the compute resource requirements and limits
  resources:
    requests:
      memory: "512Mi"
      cpu: "250m"
    limits:
      memory: "1024Mi"
      cpu: "500m"
  # storageMode can be EmptyDir, HostPath, StorageClass
  storageMode: EmptyDir
  # hostPath is the local path to store data
  hostPath: /data/rocketmq/nameserver
  # volumeClaimTemplates defines the storageClass
  volumeClaimTemplates:
    - metadata:
        name: namesrv-storage
      spec:
        accessModes:
          - ReadWriteOnce
        storageClassName: rocketmq-storage
        resources:
          requests:
            storage: 1Gi

---
apiVersion: rocketmq.apache.org/v1alpha1
kind: Console
metadata:
  name: console
  namespace: default
spec:
  # nameServers is the [ip:port] list of name service
  nameServers: ""
  # consoleDeployment define the console deployment
  consoleDeployment:
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      labels:
        app: rocketmq-console
    spec:
      replicas: 1
      selector:
        matchLabels:
          app: rocketmq-console
      template:
        metadata:
          labels:
            app: rocketmq-console
        spec:
          containers:
            - name: console
              image: apacherocketmq/rocketmq-console:2.0.0
              ports:
                - containerPort: 8080

The yaml defines the RocketMQ name server and broker cluster scale, the [ip:port] list of name service and so on. By default, the nameServers is an empty string which means it is automatically obtained by the operator.

Notice: Currently the broker image use the formula max(min(1/2 ram, 1024MB), min(1/4 ram, 8GB)) to calculate JVM Xmx size in which ram is the host memory size. If the memory resource limit is lower than the container requirement, it may occur the OOMkilled error.

Create RocketMQ Cluster

1. Deploy the RocketMQ name service cluster by running:

$ kubectl apply -f example/rocketmq_v1alpha1_rocketmq_cluster.yaml
broker.rocketmq.apache.org/broker created
nameservice.rocketmq.apache.org/name-service created
console.rocketmq.apache.org/console created

The name server cluster will be created first, after all name server cluster is in running state, the operator will create the broker cluster.

Check the status:

``` $ kubectl get pods -owide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES broker-0-master-0 1/1 Running 0 71s 10.1.5.91 docker-desk