pxc群集搭建

1.环境

Percona-XtraDB 5.7.22-22-29.26-log

percona-xtrabackup-24-2.4.12

192.168.99.210:3101(第一节点)

192.168.99.211:3101

192.168.99.212:3101

2.安装

• 安装包获取(根据操作系统类型获取不同安装包):
  wget https://www.percona.com/downloads/Percona-XtraDB-Cluster-LATEST/Percona-XtraDB-Cluster-5.7.22-29.26/binary/tarball/Percona-XtraDB-Cluster-5.7.22-rel22-29.26.1.Linux.x86_64.ssl102.tar.gz
  wget https://www.percona.com/downloads/XtraBackup/Percona-XtraBackup-2.4.12/binary/redhat/7/x86_64/percona-xtrabackup-24-2.4.12-1.el7.x86_64.rpm
• 所有节点上做以下操作:

\\安装依赖包yum install -y git scons gcc gcc-c++ openssl check cmake bison boost-devel asio-devel libaio-devel ncurses-devel readline-devel pam-devel socat\\安装xtrabackupyum localinstall  percona-xtrabackup-24-2.4.12-1.el7.x86_64.rpmtar -xzvf tarball/Percona-XtraDB-Cluster-5.7.22-rel22-29.26.1.Linux.x86_64.ssl102.tar.gz /opt/ln -s /opt/Percona-XtraDB-Cluster-5.7.22-rel22-29.26.1.Linux.x86_64.ssl102/ /usr/local/pxc

3.初始化第一节点

mkdir /data/pxc/mysql3101/{data,logs,tmp}chown -R mysql:mysql /data/pxc/*ln -s /data/pxc/mysql3101/ /3101

• 修改my_pxc.cnf
  如需要全配置文件可在下载

loose_wsrep_cluster_name=pxc_wenyzloose_wsrep_cluster_address=gcomm://192.168.99.210,192.168.99.211,192.168.99.212loose_wsrep_node_address=192.168.99.210#//建议将原SST使用的4444端口改为其他端口,因为4444是msblast蠕虫病毒使用端口,在有云安全产品下可能会被拦截.wsrep_sst_receive_address =192.168.99.210:4588   loose_wsrep_provider=/usr/local/pxc/lib/libgalera_smm.so#wsrep_sst_method=reyncloose_wsrep_sst_method=xtrabackup-v2loose_wsrep_sst_auth=sst:zstxxx

• 初始化实例和启动

初始化:/usr/local/pxc/bin/mysqld --defaults-file=/3101/my_pxc.cnf --initialize-insecure启动:/usr/local/pxc/bin/mysqld --defaults-file=/3101/my_pxc.cnf --wsrep-new-cluster&

• 创建sst账号:

\\登录mysqlGRANT RELOAD, PROCESS, SUPER, REPLICATION CLIENT ON *.* TO 'sst'@'%'  identified by 'zstxxx';

4.其他节点

其他节点不用初始化,直接启动就可以.但启动过程中会遇到很多报错,需要逐一排除.

/usr/local/pxc/bin/mysqld --defaults-file=/3101/my_pxc.cnf&

• 成功加入的过程(第二节点)

刚启动时监听端口状态: #netstat -nltpActive Internet connections (only servers)Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    tcp        0      0 0.0.0.0:4588(SST)          0.0.0.0:*               LISTEN      22451/socat         tcp        0      0 0.0.0.0:3508            0.0.0.0:*               LISTEN      16289/mysqld        tcp        0      0 0.0.0.0:4567(PXC通信)            0.0.0.0:*               LISTEN      21978/mysqld               [root@db212_08:58:42 /3101]  \\节点初始化完成后的端监听状态:#netstat -nltpActive Internet connections (only servers)Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    tcp        0      0 0.0.0.0:4567            0.0.0.0:*               LISTEN      22431/mysqld        tcp6       0      0 :::3101 (mysql)                :::*                    LISTEN      22431/mysqld

• 登录节点查看状态:

root@localhost [(none)]>show global status like 'wsrep_%';+----------------------------------+------------------------------------------------------------------+| Variable_name                    | Value                                                            |+----------------------------------+------------------------------------------------------------------+| wsrep_local_state_uuid           | 8c3106bd-9a40-11e8-8088-822c237073c1                             || wsrep_protocol_version           | 8                                                                || wsrep_last_applied               | 6                                                                || wsrep_last_committed             | 6                                                                || wsrep_replicated                 | 1                                                                || wsrep_replicated_bytes           | 400                                                              || wsrep_repl_keys                  | 2                                                                || wsrep_repl_keys_bytes            | 40                                                               || wsrep_repl_data_bytes            | 295                                                              || wsrep_repl_other_bytes           | 0                                                                || wsrep_received                   | 40                                                               || wsrep_received_bytes             | 5939                                                             || wsrep_local_commits              | 0                                                                || wsrep_local_cert_failures        | 0                                                                || wsrep_local_replays              | 0                                                                || wsrep_local_send_queue           | 0                                                                || wsrep_local_send_queue_max       | 2                                                                || wsrep_local_send_queue_min       | 0                                                                || wsrep_local_send_queue_avg       | 0.250000                                                         || wsrep_local_recv_queue           | 0                                                                || wsrep_local_recv_queue_max       | 1                                                                || wsrep_local_recv_queue_min       | 0                                                                || wsrep_local_recv_queue_avg       | 0.000000                                                         || wsrep_local_cached_downto        | 5                                                                || wsrep_flow_control_paused_ns     | 0                                                                || wsrep_flow_control_paused        | 0.000000                                                         || wsrep_flow_control_sent          | 0                                                                || wsrep_flow_control_recv          | 0                                                                || wsrep_flow_control_interval      | [ 173, 173 ]                                                     || wsrep_flow_control_interval_low  | 173                                                              || wsrep_flow_control_interval_high | 173                                                              || wsrep_flow_control_status        | OFF                                                              || wsrep_cert_deps_distance         | 1.000000                                                         || wsrep_apply_oooe                 | 0.000000                                                         || wsrep_apply_oool                 | 0.000000                                                         || wsrep_apply_window               | 1.000000                                                         || wsrep_commit_oooe                | 0.000000                                                         || wsrep_commit_oool                | 0.000000                                                         || wsrep_commit_window              | 1.000000                                                         || wsrep_local_state                | 4                                                                || wsrep_local_state_comment        | Synced                                                           || wsrep_cert_index_size            | 2                                                                || wsrep_cert_bucket_count          | 22                                                               || wsrep_gcache_pool_size           | 336152                                                           || wsrep_causal_reads               | 0                                                                || wsrep_cert_interval              | 0.000000                                                         || wsrep_ist_receive_status         |                                                                  || wsrep_ist_receive_seqno_start    | 0                                                                || wsrep_ist_receive_seqno_current  | 0                                                                || wsrep_ist_receive_seqno_end      | 0                                                                || wsrep_incoming_addresses         | 192.168.99.212:3101,192.168.99.211:3101,192.168.99.210:3101      || wsrep_desync_count               | 0                                                                || wsrep_evs_delayed                | 2c3c416a-9aa6-11e8-a518-77bd362692b0:tcp://192.168.99.212:4567:1 || wsrep_evs_evict_list             |                                                                  || wsrep_evs_repl_latency           | 0/0/0/0/0                                                        || wsrep_evs_state                  | OPERATIONAL                                                      || wsrep_gcomm_uuid                 | 8d6121e1-9a42-11e8-98e7-6b000d25448a                             || wsrep_cluster_conf_id            | 29                                                               || wsrep_cluster_size               | 3                                                                || wsrep_cluster_state_uuid         | 8c3106bd-9a40-11e8-8088-822c237073c1                             || wsrep_cluster_status             | Primary                                                          || wsrep_connected                  | ON                                                               || wsrep_local_bf_aborts            | 0                                                                || wsrep_local_index                | 1                                                                || wsrep_provider_name              | Galera                                                           || wsrep_provider_vendor            | Codership Oy 
    
                                     || wsrep_provider_version           | 3.26(r)                                                          || wsrep_ready                      | ON                                                               |+----------------------------------+------------------------------------------------------------------+68 rows in set (0.00 sec)root@localhost [(none)]>
    

5.遇到过的报错

• [ERROR] socat not found
• 解决办法:yum install -y socat

29.26.1.Linux.x86_64.ssl102/bin:/usr/local/mysql/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin)        2018-08-07T04:25:23.504726Z WSREP_SST: [ERROR] ******************* FATAL ERROR **********************        2018-08-07T04:25:23.505847Z WSREP_SST: [ERROR] socat not found in path: /usr/sbin:/sbin:/opt/Percona-XtraDB-Cluster-5.7.22-rel22-29.26.1.Linux.x86_64.ssl102/bin:/usr/local/mysql/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin        2018-08-07T04:25:23.506926Z WSREP_SST: [ERROR] ******************************************************2018-08-07T04:25:23.507286Z 0 [ERROR] WSREP: Failed to read 'ready 
    
     ' from: wsrep_sst_xtrabackup-v2 --role 'joiner' --address '192.168.99.210' --datadir '/3101/data/' --defaults-file '/3101/my_pxc.cnf' --defaults-group-suffix '' --parent '20904'  ''        Read: '(null)'2018-08-07T04:25:23.507316Z 0 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'joiner' --address '192.168.99.210' --datadir '/3101/data/' --defaults-file '/3101/my_pxc.cnf' --defaults-group-suffix '' --parent '20904'  '' : 2 (No such file or directory)2018-08-07T04:25:23.507360Z 1 [ERROR] WSREP: Failed to prepare for 'xtrabackup-v2' SST. Unrecoverable.2018-08-07T04:25:23.507371Z 1 [ERROR] Aborting
    

• Possible timeout in receving first data from donor in gtid/keyring stage
  在donor的error.log里查看发现无法连接192.168.99.211:4444 将4444改为其他端口即可解决

2018-08-07T04:50:52.153284Z 0 [Note] WSREP: gcomm: closing backend        2018-08-07T04:50:52.758009Z WSREP_SST: [ERROR] ******************* FATAL ERROR **********************        2018-08-07T04:50:52.759154Z WSREP_SST: [ERROR] Possible timeout in receving first data from donor in gtid/keyring stage        2018-08-07T04:50:52.760273Z WSREP_SST: [ERROR] ******************************************************        2018-08-07T04:50:52.761583Z WSREP_SST: [ERROR] Cleanup after exit with status:322018-08-07T04:50:54.579876Z 0 [Note] WSREP: (756ee09e, 'tcp://0.0.0.0:4567') turning message relay requesting off2018-08-07T04:50:55.579830Z 0 [Note] WSREP: (756ee09e, 'tcp://0.0.0.0:4567') connection to peer b83607a8 with addr tcp://192.168.99.210:4567 timed out, no messages seen in PT3S (gmcast.peer_timeout)2018-08-07T04:50:55.579922Z 0 [Note] WSREP: (756ee09e, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://192.168.99.210:45672018-08-07T04:50:57.079838Z 0 [Note] WSREP: (756ee09e, 'tcp://0.0.0.0:4567') reconnecting to b83607a8 (tcp://192.168.99.210:4567), attempt 02018-08-07T04:50:57.653421Z 0 [Note] WSREP: declaring node with index 1 suspected, timeout PT5S (evs.suspect_timeout)2018-08-07T04:50:57.653448Z 0 [Note] WSREP: evs::proto(756ee09e, LEAVING, view_id(REG,756ee09e,20)) suspecting node: b83607a82018-08-07T04:50:57.653457Z 0 [Note] WSREP: evs::proto(756ee09e, LEAVING, view_id(REG,756ee09e,20)) suspected node without join message, declaring inactive2018-08-07T04:50:57.653484Z 0 [Note] WSREP: Current view of cluster as seen by this nodeview (view_id(NON_PRIM,756ee09e,20)

• Possible timeout in receving first data from donor in gtid/keyring
  解决办法 :yum install -y git scons gcc gcc-c++ openssl check cmake bison boost-devel asio-devel libaio-devel ncurses-devel readline-devel pam-devel socat

018-08-07T12:05:49.545630Z 0 [Warning] WSREP: Member 2.0 (db212) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable        2018-08-07T12:05:50.313182Z WSREP_SST: [ERROR] ******************* FATAL ERROR **********************        2018-08-07T12:05:50.314371Z WSREP_SST: [ERROR] Possible timeout in receving first data from donor in gtid/keyring stage        2018-08-07T12:05:50.315523Z WSREP_SST: [ERROR] ******************************************************        2018-08-07T12:05:50.316914Z WSREP_SST: [ERROR] Cleanup after exit with status:322018-08-07T12:05:50.326100Z 0 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'joiner' --address '192.168.99.212:4588' --datadir '/3101/data/' --defaults-file '/3101/my_pxc.cnf' --defaults-group-suffix '' --parent '17660'  '' : 32 (Broken pipe)2018-08-07T12:05:50.326134Z 0 [ERROR] WSREP: Failed to read uuid:seqno from joiner script.2018-08-07T12:05:50.326146Z 0 [ERROR] WSREP: SST script aborted with error 32 (Broken pipe)2018-08-07T12:05:50.326182Z 0 [ERROR] WSREP: SST failed: 32 (Broken pipe)2018-08-07T12:05:50.326202Z 0 [ERROR] Aborting2018-08-07T12:05:50.326207Z 0 [Note] WSREP: Signalling cancellation of the SST request.2018-08-07T12:05:50.326239Z 0 [Note] WSREP: SST request was cancelled2018-08-07T12:05:50.546015Z 0 [Warning] WSREP: Member 2.0 (db212) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable2018-08-07T12:05:51.546468Z 0 [Warning] WSREP: Member 2.0 (db212) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable2018-08-07T12:05:52.546924Z 0 [Warning] WSREP: Member 2.0 (db212) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable2018-08-07T12:05:53.547325Z 0 [Warning] WSREP: Member 2.0 (db212) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable2018-08-07T12:05:54.547722Z 0 [Warning] WSREP: Member 2.0 (db212) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable2018-08-07T12:05:55.548121Z 0 [Warning] WSREP: Member 2.0 (db212) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable2018-08-07T12:05:56.548496Z 0 [Warning] WSREP: Member 2.0 (db212) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable2018-08-07T12:05:57.548894Z 0 [Warning] WSREP: Member 2.0 (db212) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable2018-08-07T12:05:58.549273Z 0 [Warning] WSREP: Member 2.0 (db212) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable2018-08-07T12:05:59.549743Z 0 [Warning] WSREP: Member 2.0 (db212) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable2018-08-07T12:06:00.550210Z 0 [Warning] WSREP: Member 2.0 (db212) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable