10.2.1 redis集群搭建 · linux-devops-suixiaofeng

[TOC] # 1.安装redis ``` [root@linux-node01 local]# wget http://download.redis.io/releases/redis-4.0.10.tar.gz [root@linux-node01 local]# tar -xf redis-4.0.10.tar.gz [root@linux-node01 local]# cd redis-4.0.10 [root@linux-node01 redis-4.0.10]# make &&make install #复制redis 集群管理命令 cp /usr/local/redis-4.0.10/src/redis-trib.rb /usr/local/bin/ ``` # 2.配置节点 ``` [root@linux-node01 redis-4.0.10]# mkdir -p /data/redis-cluster/{7001..7006} [root@linux-node01 redis-4.0.10]# cp /usr/local/redis-4.0.10/redis.conf /data/redis-cluster/7001/redis-7001.conf [root@linux-node01 redis-4.0.10]# cp /usr/local/redis-4.0.10/redis.conf /data/redis-cluster/7002/redis-7002.conf [root@linux-node01 redis-4.0.10]# cp /usr/local/redis-4.0.10/redis.conf /data/redis-cluster/7003/redis-7003.conf [root@linux-node01 redis-4.0.10]# cp /usr/local/redis-4.0.10/redis.conf /data/redis-cluster/7004/redis-7004.conf [root@linux-node01 redis-4.0.10]# cp /usr/local/redis-4.0.10/redis.conf /data/redis-cluster/7005/redis-7005.conf [root@linux-node01 redis-4.0.10]# cp /usr/local/redis-4.0.10/redis.conf /data/redis-cluster/7006/redis-7006.conf 需修改配置文件 #vim /data/redis-cluster/7001/redis-7001.conf bind 0.0.0.0 port 7001 daemonize yes pidfile /var/run/redis_7001.pid dir /data/redis-cluster/7001 logfile "/var/log/redis/7001.log" cluster-enabled yes cluster-node-timeout 2000 appendonly yes cluster-config-file nodes-7001.conf [root@linux-node01 redis-4.0.10]# mkdir /var/log/redis ``` # 3.启动各redis 实例 ``` [root@linux-node01 redis-4.0.10]# redis-server /data/redis-cluster/700X/redis-700X.conf ``` # 4.配置集群redis-trib.rb工具环境（ruby语言编写） ``` 创建集群 [root@linux-node01 7001]# yum install centos-release-scl-rh yum install rh-ruby23 -y scl enable rh-ruby23 bash ruby -v 加入到环境变量： echo "source /opt/rh/rh-ruby23/enable" >> /etc/profile source /etc/profile gem install redis #安装redis插件，如下载较慢可如下切换到国内gem源 ``` # 5.初始化redis集群 ``` root 6595 6571 0 20:12 pts/0 00:00:00 grep --color=auto redis [root@linux-node01 src]# ./redis-trib.rb create --replicas 1 127.0.0.1:7001 127.0.0.1:7002 127.0.0.1:7003 127.0.0.1:7004 127.0.0.1:7005 127.0.0.1:7006 命令的意义如下：给定 redis-trib.rb 程序的命令是 create ，这表示我们希望创建一个新的集群。选项 --replicas 1 表示我们希望为集群中的每个主节点创建一个从节点。之后跟着的其他参数则是实例的地址列表，我们希望程序使用这些地址所指示的实例来创建新集群。简单来说，以上命令的意思就是让 redis-trib 程序创建一个包含三个主节点和三个从节点的集群。接着， redis-trib 会打印出一份预想中的配置给你看，如果你觉得没问题的话，就可以输入 yes ， redis-trib 就会将这份配置应用到集群当中： >>> Creating cluster >>> Performing hash slots allocation on 6 nodes... Using 3 masters: 127.0.0.1:7001 127.0.0.1:7002 127.0.0.1:7003 Adding replica 127.0.0.1:7005 to 127.0.0.1:7001 Adding replica 127.0.0.1:7006 to 127.0.0.1:7002 Adding replica 127.0.0.1:7004 to 127.0.0.1:7003 >>> Trying to optimize slaves allocation for anti-affinity [WARNING] Some slaves are in the same host as their master M: 520c862acd30ac5937a3d863b7592602622bb2aa 127.0.0.1:7001 slots:0-5460 (5461 slots) master M: b8bc230f66f3f717b14227789db95418dad7ec7a 127.0.0.1:7002 slots:5461-10922 (5462 slots) master M: bd65c1c629bf30018504fa3b6793bc78758cf1a5 127.0.0.1:7003 slots:10923-16383 (5461 slots) master S: 967caf167a6b999b680e5c80bc73c2bc3e47080e 127.0.0.1:7004 replicates bd65c1c629bf30018504fa3b6793bc78758cf1a5 S: ad754dec072b452b57d6b33103556825d5cf7e3f 127.0.0.1:7005 replicates 520c862acd30ac5937a3d863b7592602622bb2aa S: 72c25c7c131835e8d2c5ad2cc6f854a2e3bb6345 127.0.0.1:7006 replicates b8bc230f66f3f717b14227789db95418dad7ec7a Can I set the above configuration? (type 'yes' to accept): yes >>> Nodes configuration updated >>> Assign a different config epoch to each node >>> Sending CLUSTER MEET messages to join the cluster Waiting for the cluster to join.. >>> Performing Cluster Check (using node 127.0.0.1:7001) M: 520c862acd30ac5937a3d863b7592602622bb2aa 127.0.0.1:7001 slots:0-5460 (5461 slots) master 1 additional replica(s) S: 72c25c7c131835e8d2c5ad2cc6f854a2e3bb6345 127.0.0.1:7006 slots: (0 slots) slave replicates b8bc230f66f3f717b14227789db95418dad7ec7a M: bd65c1c629bf30018504fa3b6793bc78758cf1a5 127.0.0.1:7003 slots:10923-16383 (5461 slots) master 1 additional replica(s) S: ad754dec072b452b57d6b33103556825d5cf7e3f 127.0.0.1:7005 slots: (0 slots) slave replicates 520c862acd30ac5937a3d863b7592602622bb2aa M: b8bc230f66f3f717b14227789db95418dad7ec7a 127.0.0.1:7002 slots:5461-10922 (5462 slots) master 1 additional replica(s) S: 967caf167a6b999b680e5c80bc73c2bc3e47080e 127.0.0.1:7004 slots: (0 slots) slave replicates bd65c1c629bf30018504fa3b6793bc78758cf1a5 [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... #这表示集群中的 16384 个槽都有至少一个主节点在处理，集群运作正常。 [OK] All 16384 slots covered. 方法二先在三台主机上分别启动一个Master ,在启动slave交叉加入各主机Master，0,1表示slave节点的数量 #设计端口单数为master 双数为slave redis-trib.rb create --replicas 0 192.168.23.128:7001 192.168.23.129:7002 192.168.23.131:7003 redis-trib.rb check 192.168.23.128:7001 #检查任意主机节点，并记录master-id ###启动slave加入各Master redis-trib.rb add-node --slave --master-id e3933a2276962abb7180ebe7767a544bdd539f27 192.168.23.128:7004（新节点） 192.168.23.128:7001（任意旧节点） redis-trib.rb add-node --slave --master-id cee9b75e480a3d707e7079631a280e732e0264b7 192.168.23.129:7005 192.168.23.128:7001 redis-trib.rb add-node --slave --master-id c61feb0e6f33d467758e8278436c03ce265c44b1 192.168.23.131:7006 192.168.23.128:7001 查看集群状态 redis-cli -c -p 7001 cluster nodes 72c25c7c131835e8d2c5ad2cc6f854a2e3bb6345 127.0.0.1:7006@17006 slave b8bc230f66f3f717b14227789db95418dad7ec7a 0 1544617429145 6 connected bd65c1c629bf30018504fa3b6793bc78758cf1a5 127.0.0.1:7003@17003 master - 0 1544617429145 3 connected 10923-16383 ad754dec072b452b57d6b33103556825d5cf7e3f 127.0.0.1:7005@17005 slave 520c862acd30ac5937a3d863b7592602622bb2aa 0 1544617429348 5 connected 520c862acd30ac5937a3d863b7592602622bb2aa 127.0.0.1:7001@17001 myself,master - 0 1544617429000 1 connected 0-5460 b8bc230f66f3f717b14227789db95418dad7ec7a 127.0.0.1:7002@17002 master - 0 1544617429045 2 connected 5461-10922 967caf167a6b999b680e5c80bc73c2bc3e47080e 127.0.0.1:7004@17004 slave bd65c1c629bf30018504fa3b6793bc78758cf1a5 0 1544617429348 4 connected 功能验证 127.0.0.1:7002> set hehe hahha -> Redirected to slot [4618] located at 127.0.0.1:7001 OK 127.0.0.1:7001> get hehe "hahha" 127.0.0.1:7001> [root@linux-node01 src]# redis-cli -c -p 7006 127.0.0.1:7006> get hehe -> Redirected to slot [4618] located at 127.0.0.1:7001 "hahha" ``` # 6.redis集群的一些操作 ## 6.1 对集群进行重新分片 ``` 你只需要指定集群中其中一个节点的地址， redis-trib 就会自动找到集群中的其他节点。 ./redis-trib.rb reshard 127.0.0.1:7001 加入新节点 [root@linux-node01 src]# ./redis-trib.rb add-node 127.0.0.1:7007 127.0.0.1:7001 [root@linux-node01 src]# redis-cli -c -p 7001 cluster nodes 将一个新节点转变为某个主节点的复制品（也即是从节点）的方法 [root@linux-node01 src]# redis-cli -c -p 7007 127.0.0.1:7007> cluster replicate bd65c1c629bf30018504fa3b6793bc78758cf1a5 OK 127.0.0.1:7007> [root@linux-node01 src]# redis-cli -c -p 7001 cluster nodes 72c25c7c131835e8d2c5ad2cc6f854a2e3bb6345 127.0.0.1:7006@17006 slave b8bc230f66f3f717b14227789db95418dad7ec7a 0 1544679501061 6 connected 3e09a2bc380840f626419f69cf1c783e848a0fde 127.0.0.1:7007@17007 slave bd65c1c629bf30018504fa3b6793bc78758cf1a5 0 1544679501061 3 connected bd65c1c629bf30018504fa3b6793bc78758cf1a5 127.0.0.1:7003@17003 master - 0 1544679501261 3 connected 10923-16383 ad754dec072b452b57d6b33103556825d5cf7e3f 127.0.0.1:7005@17005 slave 520c862acd30ac5937a3d863b7592602622bb2aa 0 1544679501000 5 connected 520c862acd30ac5937a3d863b7592602622bb2aa 127.0.0.1:7001@17001 myself,master - 0 1544679501000 1 connected 0-5460 b8bc230f66f3f717b14227789db95418dad7ec7a 127.0.0.1:7002@17002 master - 0 1544679501061 2 connected 5461-10922 967caf167a6b999b680e5c80bc73c2bc3e47080e 127.0.0.1:7004@17004 slave bd65c1c629bf30018504fa3b6793bc78758cf1a5 0 1544679500559 4 connected Removing a node: 移除一个从节点只需 redis-trib.rb del-node 127.0.0.1:7007 3e09a2bc380840f626419f69cf1c783e848a0fde [root@linux-node01 src]# redis-cli -c -p 7001 cluster nodes 72c25c7c131835e8d2c5ad2cc6f854a2e3bb6345 127.0.0.1:7006@17006 slave b8bc230f66f3f717b14227789db95418dad7ec7a 0 1544680022097 6 connected bd65c1c629bf30018504fa3b6793bc78758cf1a5 127.0.0.1:7003@17003 master - 0 1544680022397 3 connected 10923-16383 ad754dec072b452b57d6b33103556825d5cf7e3f 127.0.0.1:7005@17005 slave 520c862acd30ac5937a3d863b7592602622bb2aa 0 1544680022196 5 connected 520c862acd30ac5937a3d863b7592602622bb2aa 127.0.0.1:7001@17001 myself,master - 0 1544680022000 1 connected 0-5460 b8bc230f66f3f717b14227789db95418dad7ec7a 127.0.0.1:7002@17002 master - 0 1544680022096 2 connected 5461-10922 967caf167a6b999b680e5c80bc73c2bc3e47080e 127.0.0.1:7004@17004 slave bd65c1c629bf30018504fa3b6793bc78758cf1a5 0 1544680022096 4 connected 对集群进行重新分片； [root@linux-node01 src]# ./redis-trib.rb reshard 127.0.0.1:7001 How many slots do you want to move (from 1 to 16384)? 1000 What is the receiving node ID? 移动到哪个节点。 How many slots do you want to move (from 1 to 16384)? 1000 What is the receiving node ID? bc57518112b7fc1d33e0cd90b804b6f93df536f2 Please enter all the source node IDs. Type 'all' to use all the nodes as source nodes for the hash slots. Type 'done' once you entered all the source nodes IDs. Source node #1: 从哪个节点移动 all 从所有的节点移动 done 选择好源节点结束符。结果： [root@linux-node01 src]# redis-cli -c -p 7001 cluster nodes bc57518112b7fc1d33e0cd90b804b6f93df536f2 127.0.0.1:7007@17007 master - 0 1544682437163 8 connected 10923-11922 72c25c7c131835e8d2c5ad2cc6f854a2e3bb6345 127.0.0.1:7006@17006 slave b8bc230f66f3f717b14227789db95418dad7ec7a 0 1544682437062 6 connected bd65c1c629bf30018504fa3b6793bc78758cf1a5 127.0.0.1:7003@17003 master - 0 1544682436660 3 connected 11923-16383 ad754dec072b452b57d6b33103556825d5cf7e3f 127.0.0.1:7005@17005 slave 520c862acd30ac5937a3d863b7592602622bb2aa 0 1544682436660 5 connected 520c862acd30ac5937a3d863b7592602622bb2aa 127.0.0.1:7001@17001 myself,master - 0 1544682437000 1 connected 0-5460 b8bc230f66f3f717b14227789db95418dad7ec7a 127.0.0.1:7002@17002 master - 0 1544682437163 2 connected 5461-10922 967caf167a6b999b680e5c80bc73c2bc3e47080e 127.0.0.1:7004@17004 slave bd65c1c629bf30018504fa3b6793bc78758cf1a5 0 1544682436660 4 connected [root@linux-node01 src]# 移除主节点。首先要把节点中的哈希槽转移到其他节点中，执行下面的命令 ./redis-trib.rb reshard 127.0.0.1:7001 系统会提示我们要移动多少哈希槽，这里移动1000个，因为127.0.0.1:7006节点有1000个哈希槽然后系统提示我们输入要接收这些哈希槽的节点的ID，这里使用127.0.0.1:7001的节点ID 然后要我们选择从那些节点中转出哈希槽，这里一定要输入127.0.0.1:7006这个节点的ID，最后输入 done 表示输入完毕 [root@linux-node01 src]# redis-trib.rb del-node 127.0.0.1:7001 bc57518112b7fc1d33e0cd90b804b6f93df536f2 给指定的redis master节点添加从节点… 默认add-node是添加主master节点….. redis-trib.rb add-node --slave --master-id 'b8bc230f66f3f717b14227789db95418dad7ec7a' 127.0.0.1:7007 127.0.0.1:7001 ``` # 7.其他相关优化 ``` 内存优化相关 #修改系统参数 echo never > /sys/kernel/mm/transparent_hugepage/enabled echo 511 > /proc/sys/net/core/somaxconn 并加到 /etc/rc.local vim /etc/sysctl.conf vm.overcommit_memory=1 net.core.somaxconn = 511 kernel.pid_max=1000000 sysctl -p vim /etc/security/limits.conf * soft nproc 65535 * hard nproc 65535 * soft nofile 65535 * hard nofile 65535 ```