Docker + Redis Cluster 實戰

Redis 說明

架構圖

Cluster 模式

Redis Cluster 默認分配了16384個 hash slot。

讀寫資料時，對每個 key 計算 CRC16 值，再對 16384 取模，取得 key 對應的 hash slot ，實現負載均衡的目的。

注意的是，最少必須要有三個主節點三個從節點，否則創建集群時會失敗。

hash slot = CRC16(key) mod 16384

 

Sentinel 模式

Redis Sentinel 是官方推薦的高可用性解決方案，他能監控多個主從集群，當發現主節點當機了，能夠自動運行主重切換。

Redis 5 在建立 redis cluster 時也將 sentinel 機制加入，省去了自己建立 sentinel 時間跟精力。

 

Docker + Redis Cluster 實戰

基本環境說明

1. CentOS 7 

2. Docker + Docker-compose

3. Redis

 

一、Docker 建置

1.安裝 Docker

安裝所需的軟體包

$ sudo yum install -y yum-utils \ device-mapper-persistent-data \ lvm2

設置穩定的存儲庫

$ sudo yum-config-manager \ --add-repo \ https://download.docker.com/linux/centos/docker-ce.repo

安裝 Docker

$ sudo yum install docker-ce

啟動 Docker

$ systemctl start docker

 

2.獲取 Redis 鏡像

在 Docker 庫獲取 Redis 鏡像

$ sudo docker pull redis

查看到已經安裝的鏡像

$ sudo docker images

 

3.準備 Redis 配置文件

創建redis配置文件（redis-cluster.tmpl）

$ sudo cat > /home/jerry/docker/redis-cluster/redis-cluster.tmpl 
port ${PORT} 
protected-mode no 
cluster-enabled yes 
cluster-config-file nodes.conf 
cluster-node-timeout 5000 
cluster-announce-ip 192.168.177.128 
cluster-announce-port ${PORT} 
cluster-announce-bus-port 1${PORT} 
appendonly yes

在/home/jerry/docker/redis-cluster下產生6個redis所需要的資料(conf和data)

$ for port in {7001..7006}; do mkdir -p ./${port}/conf && PORT=${port} envsubst < ./redis-cluster.tmpl > ./${port}/conf/redis.conf && mkdir -p ./${port}/data; done

 

4.運行 Redis

建立6個redis容器-這邊注意--net host 是綁在host  這樣後續才能正常使用

$ for port in {7001..7006}; do sudo docker run -d -ti -p ${port}:${port} -p 1${port}:1${port} -v /home/jerry/docker/redis-cluster/${port}/conf/redis.conf:/usr/local/etc/redis/redis.conf -v /home/jerry/docker/redis-cluster/${port}/data:/data --restart always --name redis-${port} --net host --sysctl net.core.somaxconn=1024 redis redis-server /usr/local/etc/redis/redis.conf; done

查看 Docker 容器資料

$ sudo docker ps

 

5.啟動 Redis Cluster

查看本機IP-192.168.177.128

$ ifconfig

隨便進一個容器啟動 Redis Cluster就可以了

$ sudo docker exec -it redis-7001 redis-cli -h 192.168.177.128 -p 7001 --cluster create 192.168.177.128:7001 192.168.177.128:7002 192.168.177.128:7003 192.168.177.128:7004 192.168.177.128:7005 192.168.177.128:7006 --cluster-replicas 1

查看 Redis Cluster所有節點

7001,7002,7003->master,0-5460,5461-10922,10923-16383

7004,7005,7006->slave

$ sudo docker exec -it redis-7001 redis-cli -c -p 7001 cluster nodes

 

二、Docker-compose 建置

1.安裝 Docker-compose

$ cd /usr/bin
$ sudo wget https://github.com/docker/compose/releases/download/1.25.5/docker-compose-Linux-x86_64
$ mv docker-compose-Linux-x86_64 docker-compose
$ chmod 755 docker-compose

 

2.撰寫 yml檔

在/home/jerry/docker/redis-cluster下新增docker-compose-redis.yml

version: '3.4'
services:
  redis-7001:
    image: redis
    container_name: redis-7001
    restart: always
    network_mode: host
    ports:
      - 7001:7001
      - 17001:17001
    volumes:
      - /home/jerry/docker/redis-cluster/7001/data:/data
      - /home/jerry/docker/redis-cluster/7001/conf/redis.conf:/usr/local/etc/redis/redis.conf 
    command:
      redis-server /usr/local/etc/redis/redis.conf

  redis-7002:
    image: redis
    container_name: redis-7002
    restart: always
    network_mode: host
    ports:
      - 7002:7002
      - 17002:17002
    volumes:
      - /home/jerry/docker/redis-cluster/7002/data:/data
      - /home/jerry/docker/redis-cluster/7002/conf/redis.conf:/usr/local/etc/redis/redis.conf 
    command:
      redis-server /usr/local/etc/redis/redis.conf

  redis-7003:
    image: redis
    container_name: redis-7003
    restart: always
    network_mode: host
    ports:
      - 7003:7003
      - 17003:17003
    volumes:
      - /home/jerry/docker/redis-cluster/7003/data:/data
      - /home/jerry/docker/redis-cluster/7003/conf/redis.conf:/usr/local/etc/redis/redis.conf 
    command:
      redis-server /usr/local/etc/redis/redis.conf

  redis-7004:
    image: redis
    container_name: redis-7004
    restart: always
    network_mode: host
    ports:
      - 7004:7004
      - 17004:17004
    volumes:
      - /home/jerry/docker/redis-cluster/7004/data:/data
      - /home/jerry/docker/redis-cluster/7004/conf/redis.conf:/usr/local/etc/redis/redis.conf 
    command:
      redis-server /usr/local/etc/redis/redis.conf

  redis-7005:
    image: redis
    container_name: redis-7005
    restart: always
    network_mode: host
    ports:
      - 7005:7005
      - 17005:17005
    volumes:
      - /home/jerry/docker/redis-cluster/7005/data:/data
      - /home/jerry/docker/redis-cluster/7005/conf/redis.conf:/usr/local/etc/redis/redis.conf 
    command:
      redis-server /usr/local/etc/redis/redis.conf

  redis-7006:
    image: redis
    container_name: redis-7006
    restart: always
    network_mode: host
    ports:
      - 7006:7006
      - 17006:17006
    volumes:
      - /home/jerry/docker/redis-cluster/7006/data:/data
      - /home/jerry/docker/redis-cluster/7006/conf/redis.conf:/usr/local/etc/redis/redis.conf 
    command:
      redis-server /usr/local/etc/redis/redis.conf

 

3.啟動 Docker Container

$ sudo docker-compose -f docker-compose-redis.yml up -d

 

4.啟動 Redis Cluster

隨便進一個容器啟動 Redis Cluster就可以了

$ sudo docker exec -it redis-7001 redis-cli -h 192.168.177.128 -p 7001 --cluster create 192.168.177.128:7001 192.168.177.128:7002 192.168.177.128:7003 192.168.177.128:7004 192.168.177.128:7005 192.168.177.128:7006 --cluster-replicas 1

查看 Redis Cluster所有節點

7001,7002,7003->master,0-5460,5461-10922,10923-16383

7004,7005,7006->slave

$ sudo docker exec -it redis-7001 redis-cli -c -p 7001 cluster nodes

 

三、Redis 測試

測試資料

每次5k byte資料，總共20萬次請求

跑到124792次的時候停止了

這邊可以發現

CPU使用範圍約莫是10%~30%

記憶體使用範圍約莫是80MB至100MB

查看redis cluster所有節點

這邊可以發現master改變了

7001,7002,7005->master

7003,7004,7006->slave

$ sudo docker exec -it redis-7001 redis-cli -c -p 7001 cluster nodes

 

四、其他指令

暫停容器並刪除容器

$ for port in {7001..7006}; do sudo docker stop redis-${port}; sudo docker rm redis-${port}; done

清除redis內容

$ sudo docker exec -it redis-7001 redis-cli -h 192.168.177.128 -p 7001 flushall ; sudo docker exec -it redis-7002 redis-cli -h 192.168.177.128 -p 7002 flushall ; sudo docker exec -it redis-7003 redis-cli -h 192.168.177.128 -p 7003 flushall ; sudo docker exec -it redis-7004 redis-cli -h 192.168.177.128 -p 7004 flushall ; sudo docker exec -it redis-7005 redis-cli -h 192.168.177.128 -p 7005 flushall ; sudo docker exec -it redis-7006 redis-cli -h 192.168.177.128 -p 7006 flushall ;
$ sudo docker exec -it redis-7001 redis-cli -h 192.168.177.128 -p 7001 flushdb ; sudo docker exec -it redis-7002 redis-cli -h 192.168.177.128 -p 7002 flushdb ; $ sudo docker exec -it redis-7003 redis-cli -h 192.168.177.128 -p 7003 flushdb ; sudo docker exec -it redis-7004 redis-cli -h 192.168.177.128 -p 7004 flushdb ; sudo docker exec -it redis-7005 redis-cli -h 192.168.177.128 -p 7005 flushdb ; sudo docker exec -it redis-7006 redis-cli -h 192.168.177.128 -p 7006 flushdb ; 
$ sudo docker exec -it redis-7001 redis-cli -h 192.168.177.128 -p 7001 cluster reset ; sudo docker exec -it redis-7002 redis-cli -h 192.168.177.128 -p 7002 cluster reset ; sudo docker exec -it redis-7003 redis-cli -h 192.168.177.128 -p 7003 cluster reset ; sudo docker exec -it redis-7004 redis-cli -h 192.168.177.128 -p 7004 cluster reset ; sudo docker exec -it redis-7005 redis-cli -h 192.168.177.128 -p 7005 cluster reset ; sudo docker exec -it redis-7006 redis-cli -h 192.168.177.128 -p 7006 cluster reset ; 

刪除容器應設的配置檔案

$ for port in {7001..7006}; do rm -rf /home/jerry/docker/redis-cluster/${port}/data/; rm -rf /home/jerry/docker/redis-cluster/${port}/conf/; done

修改檔案

$ vim /home/jerry/docker/redis-cluster/700*/conf/redis.conf

VM主機對外開放ip

$ for port in {7001..7006}; do firewall-cmd --zone=public --add-port=${port}/tcp --permanent; done
$ firewall-cmd --reload

查看docker容器資料

$ sudo docker ps

查看docker網卡清單

$ sudo docker network ls

查看host網卡資訊

$ sudo docker network inspect host

查看docker容器使用狀態

$ sudo docker stats redis-7001 redis-7002 redis-7003 redis-7004 redis-7005 redis-7006