Raja's Exocortex

Zookeeper Cluster using docker-compose

Architecture Notes

  1. Setup 3 node ZK cluster with hostnames: zk1.multipass, zk2.multipass, zk3.multipass.
  2. ZK requires FQDN for all hosts to ensure cluster members are able to resolve each other and self-signed SSL certs are trusted.
  3. Example below uses multipass to setup 3 VMs on a single server. In production, replace multipass with the FQDN of the ZK hosts.
  4. To ensure cluster members (inside docker) are able to resolve each other, the docker host and zk docker container have the same hostname.

docker-compose.yml

# docker-compose for zookeeper
# Notes:
#  - Ensure docker host and zk container have the same hostname, eg. zk1.multipass
#  -
# Testing:
#  - Run following command on all ZK cluster members and ensure it returns "leader", or "follower"
#    wget -O - http://zk1.multipass:8080/commands/stat | grep server_state
#  - If any server is not a leader or follower, the cluster is not init correctly.
#    Restart the ZK servers one by one and check again, wait 10 seconds between restarts.

version: '2.7'

name: zookeeper-localhost

services:
  zookeeper:
    image: zookeeper
    restart: always

	# ensure docker host is also the same
	hostname: zk1.multipass

    # Publish client port, 2 x cluster communication ports and AdminSerer web interface port
    ports:
      - 2181:2181
      - 2888:2888
      - 3888:3888
      - 8080:8080

	environment:
	  # Match this with server.# from zoo.cfg, ignored if /data/myid is already present
      ZOO_MY_ID: 1

      # ignored if mounting a /conf/zoo.cfg as done below
      ZOO_SERVERS: server.1=zk1.multipass:2888:3888;2181 server.2=zk2.multipass:2888:3888;2181 server.3=zk3.multipass:2888:3888;2181

	  # Java heap size in MB (default is 1000MB)
      ZK_SERVER_HEAP: 4096

    volumes:
      - ./data/data:/data
      - ./data/datalog:/datalog
      - ./data/logs:/logs
      - ./conf/zoo.cfg:/conf/zoo.cfg:ro
      - ./ssl:/ssl:ro

/conf/zoo.cfg

dataDir=/data
dataLogDir=/datalog
tickTime=2000
initLimit=5
syncLimit=2
autopurge.snapRetainCount=3
autopurge.purgeInterval=0
maxClientCnxns=60
standaloneEnabled=true
admin.enableServer=true

# hostnames must correspond with docker-compose ZOO_MY_ID
server.1=zk1.multipass:2888:3888;2181
server.2=zk2.multipass:2888:3888;2181
server.3=zk3.multipass:2888:3888;2181

# Use SSL quorum
sslQuorum=true
serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory
ssl.quorum.keyStore.location=/ssl/keystore.jks
ssl.quorum.keyStore.password=secret_password
ssl.quorum.trustStore.location=/ssl/truststore.jks
ssl.quorum.trustStore.password=secret_password

Securing Quorum Communication using Self-signed Certs

Steps

  1. Inside a ZK docker container use the keytool app to create SSL certs and add them to the keystore and truststore files.
  2. keystore is unique to each ZK node.
  3. truststore is common for all ZK nodes in the cluster, it contains the certs from all the ZK nodes.
# Inside zk1.multipass docker container, run the following

cd /ssl
# Generate keypairs and store them in the keystore
keytool -genkeypair -alias zk1.multipass -keyalg RSA -keysize 2048 -dname "cn=zk1.multipass" -keypass secret_password -keystore keystore-zk1.jks -storepass secret_password
keytool -genkeypair -alias zk2.multipass -keyalg RSA -keysize 2048 -dname "cn=zk2.multipass" -keypass secret_password -keystore keystore-zk2.jks -storepass secret_password
keytool -genkeypair -alias zk3.multipass -keyalg RSA -keysize 2048 -dname "cn=zk3.multipass" -keypass secret_password -keystore keystore-zk3.jks -storepass secret_password

# Export the individial certs from each keystore
keytool -exportcert -alias zk1.multipass -keystore keystore-zk1.jks -file zk1.cer -rfc
keytool -exportcert -alias zk2.multipass -keystore keystore-zk2.jks -file zk2.cer -rfc
keytool -exportcert -alias zk3.multipass -keystore keystore-zk3.jks -file zk3.cer -rfc

# Import all the certs into the truststore
keytool -importcert -alias zk1.multipass -file zk1.cer -keystore truststore.jks -storepass secret_password
keytool -importcert -alias zk2.multipass -file zk2.cer -keystore truststore.jks -storepass secret_password
keytool -importcert -alias zk3.multipass -file zk3.cer -keystore truststore.jks -storepass secret_password

# In each ZK container, copy /ssl/truststore.jks
#    copy /ssl/truststore.jks
#    copy /ssl/keystore-zk1.jks /ssl/keystore.jks

References

  1. Official ZooKeeper docker image: https://hub.docker.com/_/zookeeper
  2. TLS Quorum: https://zookeeper.apache.org/doc/r3.8.0/zookeeperAdmin.html#Quorum+TLS

TODO

  1. Logging, to file and log rotation
  2. Configure autopurge intervals, default is no purging!