1.1 安装gluterfs
1.1.1 环境准备
1.1.1.1 系统环境准备
# 系统信息 [[email protected] ~]# cat /etc/redhat-release CentOS Linux release 7.2.1511 (Core) [[email protected] ~]# uname -r 3.10.0-327.el7.x86_64 # 网络信息 [[email protected] ~]# hostname -I 10.0.0.211 172.16.1.211 [[email protected] ~]# hostname -I 10.0.0.212 172.16.1.212
提示：先在虚拟机中挂载一块新的硬盘作为非系统盘，实际生产中视情况是否使用独立的存储设备。
1.1.1.2 分区并格式化硬盘
# 重新分区 [[email protected] ~]# fdisk /dev/sdb Command (m for help): n Select (default p): p Partition number (1-4, default 1): First sector (2048-20971519, default 2048): Last sector, +sectors or +size{K,M,G} (2048-20971519, default 20971519): Command (m for help): w # 格式化硬盘 [[email protected] ~]# mkfs.xfs /dev/sdb1
1.1.1.3 挂载并创建共享目录
# 挂载硬盘 [[email protected] ~]# mkdir -p /data/ [[email protected] ~]# mount /dev/sdb1 /data/ # 创建共享目录 [[email protected] ~]# mkdir -p /data/{distributed01,replicated01,striped01} [[email protected] ~]# mkdir -p /data/distributed-replicated0{1,2} [[email protected] ~]# mkdir -p /data/{distributed02,replicated02,striped02} [[email protected] ~]# mkdir -p /data/distributed-replicated0{3,4}
1.1.1.4 制作本地yum源文件
[[email protected] ~]# cat > /etc/yum.repos.d/CentOS-Gluster.repo << EOF [centos-gluster] name=centos-gluster baseurl=http://10.0.0.1/centos/7/storage/x86_64/gluster-3.10/ enabled=1 gpgcheck=0 EOF
1.1.2 安装部署软件
1.1.2.1 安装软件
[[email protected] ~]# yum install -y centos-release-gluster # 如果没有本地源时安装此源 [[email protected] ~]# yum install -y glusterfs-server
1.1.2.2 启动服务
[[email protected] ~]# systemctl enable glusterd [[email protected] ~]# systemctl start glusterd
1.2 配置信任池（即所有节点地址池）
[[email protected] ~]# gluster peer probe 10.0.0.212 [[email protected] ~]# gluster peer probe 10.0.0.211
1.3 创建卷
提示：glusterfs的基础卷分为分布式卷、复制卷、条带卷，并且可以由这三种基础卷组合为其他扩展卷。
1.3.1 创建分布式卷
1.3.1.1 语法及示例
# gluster volume create [transport tcp | rdma | tcp,rdma] # gluster volume create test-volume server1:/exp1 server2:/exp2 server3:/exp3 server4:/exp4
1.3.1.2 命令实例
[[email protected] ~]# gluster volume create distributed-volume 10.0.0.211:/data/distributed01 10.0.0.212:/data/distributed02
1.3.2 创建复制卷（RAID 1）
1.3.2.1 语法及示例
# gluster volume create [replica ] [transport tcp | rdma | tcp,rdma] # gluster volume create test-volume replica 2 transport tcp server1:/exp1 server2:/exp2
1.3.2.2 命令实例
[[email protected] ~]# gluster volume create replicated-volume replica 2 10.0.0.211:/data/replicated01 10.0.0.212:/data/replicated02
1.3.3 创建条带卷（RAID 0）
1.3.3.1 语法及示例
# gluster volume create [stripe ] [transport tcp | rdma | tcp,rdma] # gluster volume create test-volume stripe 2 transport tcp server1:/exp1 server2:/exp2
1.3.3.2 命令实例
[[email protected] ~]# gluster volume create striped-volume stripe 2 10.0.0.211:/data/striped01 10.0.0.212:/data/striped02
1.3.4 创建分布式条带卷
1.3.4.1 语法及示例（必须偶数个节点）
# gluster volume create [stripe ] [transport tcp | rdma | tcp,rdma] # gluster volume create test-volume stripe 4 transport tcp server1:/exp1 server2:/exp2 server3:/exp3 server4:/exp4 server5:/exp5 server6:/exp6 server7:/exp7 server8:/exp8
1.3.5 创建分布式复制卷
1.3.5.1 语法及示例（必须偶数个节点，生产中常用方式）
# gluster volume create [replica ] [transport tcp | rdma | tcp,rdma] # gluster volume create test-volume replica 2 transport tcp server1:/exp1 server2:/exp2 server3:/exp3 server4:/exp4 server5:/exp5 server6:/exp6
1.3.5.2 命令实例（此处使用两个节点模拟四个节点时的情况）
[[email protected] ~]# gluster volume create distributed-replicated-volume replica 2 transport tcp 10.0.0.211:/data/distributed-replicated01 10.0.0.212:/data/distributed-replicated03 10.0.0.211:/data/distributed-replicated02 10.0.0.212:/data/distributed-replicated04 force
提示：复制的顺序和创建的顺序有关：server1:/exp1 server2:/exp2互为复制关系，server1:/exp1 server2:/exp2互为复制关系，以此类推。
1.3.5.3 问题解决
【问题现象】：
[[email protected] ~]# gluster volume create distributed-replicated-volume replica 2 transport tcp 10.0.0.211:/data/distributed-replicated01 10.0.0.211:/data/distributed-replicated02 10.0.0.212:/data/distributed-replicated03 10.0.0.212:/data/distributed-replicated04 volume create: distributed-replicated-volume: failed: Multiple bricks of a replicate volume are present on the same server. This setup is not optimal. Use 'force' at the end of the command if you want to override this behavior.
【解决办法】：
gluster volume create distributed-replicated-volume replica 2 transport tcp 10.0.0.211:/data/distributed-replicated01 10.0.0.212:/data/distributed-replicated03 10.0.0.211:/data/distributed-replicated02 10.0.0.212:/data/distributed-replicated04 force
提示：如果执行命令时提示“Use 'force' at the end of the command if you want to override this behavior.”时，在命令最后添加force参数执行即可。
1.3.6 创建分布式条带复制卷
1.3.6.1 语法及示例
# gluster volume create [stripe ] [replica ] [transport tcp | rdma | tcp,rdma] # gluster volume create test-volume stripe 2 replica 2 transport tcp server1:/exp1 server2:/exp2 server3:/exp3 server4:/exp4 server5:/exp5 server6:/exp6 server7:/exp7 server8:/exp8
1.4 启动卷并查看状态
1.4.1 启动卷
[[email protected] ~]# gluster volume start distributed-volume [[email protected] ~]# gluster volume start replicated-volume [[email protected] ~]# gluster volume start striped-volume [[email protected] ~]# gluster volume start distributed-replicated-volume
1.4.2 查看卷状态
[[email protected] ~]# gluster volume info Volume Name: distributed-replicated-volume Type: Distributed-Replicate Volume ID: 60b17fbc-309e-4893-82ab-208f946b7b74 Status: Started Snapshot Count: 0 Number of Bricks: 2 x 2 = 4 Transport-type: tcp Bricks: Brick1: 10.0.0.211:/data/distributed-replicated01 Brick2: 10.0.0.212:/data/distributed-replicated03 Brick3: 10.0.0.211:/data/distributed-replicated02 Brick4: 10.0.0.212:/data/distributed-replicated04 Options Reconfigured: transport.address-family: inet nfs.disable: on Volume Name: distributed-volume Type: Distribute Volume ID: 4bb1530e-99a1-44fd-9fd1-b90e387262dc Status: Started Snapshot Count: 0 Number of Bricks: 2 Transport-type: tcp Bricks: Brick1: 10.0.0.211:/data/distributed01 Brick2: 10.0.0.212:/data/distributed02 Options Reconfigured: transport.address-family: inet nfs.disable: on Volume Name: replicated-volume Type: Replicate Volume ID: 10517a05-7fdd-47d5-8018-d1788c0482b5 Status: Started Snapshot Count: 0 Number of Bricks: 1 x 2 = 2 Transport-type: tcp Bricks: Brick1: 10.0.0.211:/data/replicated01 Brick2: 10.0.0.212:/data/replicated02 Options Reconfigured: transport.address-family: inet nfs.disable: on Volume Name: striped-volume Type: Stripe Volume ID: 50be9ee6-f8c5-424b-ba2d-9812d3cd6270 Status: Started Snapshot Count: 0 Number of Bricks: 1 x 2 = 2 Transport-type: tcp Bricks: Brick1: 10.0.0.211:/data/striped01 Brick2: 10.0.0.212:/data/striped02 Options Reconfigured: transport.address-family: inet nfs.disable: on
1.5 使用gluterfs
1.5.1 创建共享目录
[[email protected] ~]# mkdir -p /mnt/{distributed,replicated,striped,distributed-replicated}
1.5.2 挂载共享卷
[[email protected] ~]# mount.glusterfs 10.0.0.211:/distributed-volume /mnt/distributed/ [[email protected] ~]# mount.glusterfs 10.0.0.211:/replicated-volume /mnt/replicated/ [[email protected] ~]# mount.glusterfs 10.0.0.211:/striped-volume /mnt/striped/ [[email protected] ~]# mount.glusterfs 10.0.0.211:/distributed-replicated-volume /mnt/distributed-replicated/
1.5.3 查看挂载情况
[[email protected] ~]# df -h Filesystem Size Used Avail Use% Mounted on /dev/sda3 98G 2.3G 96G 3% / devtmpfs 228M 0 228M 0% /dev tmpfs 237M 0 237M 0% /dev/shm tmpfs 237M 4.6M 233M 2% /run tmpfs 237M 0 237M 0% /sys/fs/cgroup /dev/sda1 1014M 110M 905M 11% /boot tmpfs 48M 0 48M 0% /run/user/0 /dev/sdb1 10G 33M 10G 1% /data 10.0.0.211:/distributed-volume 20G 66M 20G 1% /mnt/distributed 10.0.0.211:/replicated-volume 10G 33M 10G 1% /mnt/replicated 10.0.0.211:/striped-volume 20G 66M 20G 1% /mnt/striped 10.0.0.211:/distributed-replicated-volume 20G 66M 20G 1% /mnt/distributed-replicated
1.6 测试卷
1.6.1 测试分布式卷
1.6.1.1 写入文件
[[email protected] ~]# echo "distributed01" > /mnt/distributed/distributed01.txt [[email protected] ~]# echo "distributed02" > /mnt/distributed/distributed02.txt
1.6.1.2 查看写入的文件
[[email protected] ~]# ll /data/distributed01/* -rw-r--r-- 2 root root 14 Dec 14 15:30 /data/distributed01/distributed02.txt [[email protected] ~]# ll /data/distributed02/* -rw-r--r-- 2 root root 14 Dec 14 15:30 /data/distributed02/distributed01.txt
1.6.2 测试复制卷
1.6.2.1 写入文件
[[email protected] ~]# echo "replicated" > /mnt/replicated/replicated.txt
1.6.2.2 查看写入的文件
[[email protected] ~]# ll /data/replicated01/* -rw-r--r-- 2 root root 11 Dec 14 15:34 /data/replicated01/replicated.txt [[email protected] ~]# ll /data/replicated02/* -rw-r--r-- 2 root root 11 Dec 14 15:34 /data/replicated02/replicated.txt
1.6.3 测试条带卷
1.6.3.1 写入文件
[[email protected] ~]# ll -h /var/log/messages-20171214 -rw-------. 1 root root 736K Dec 14 15:19 /var/log/messages-20171214 [[email protected] ~]# cat /var/log/messages-20171214 >> /mnt/striped/striped.txt
1.6.3.2 查看写入的文件
[[email protected] ~]# ll -h /data/striped01/* -rw-r--r-- 2 root root 384K Dec 14 15:40 /data/striped01/striped.txt [[email protected] ~]# ll -h /data/striped02/* -rw-r--r-- 2 root root 352K Dec 14 15:40 /data/striped02/striped.txt
1.6.4 测试分布式复制卷
1.6.4.1 写入文件
[[email protected] ~]# cat /var/log/messages-20171214 > /mnt/distributed-replicated/distributed-replicated01.txt [[email protected] ~]# cat /var/log/messages-20171214 > /mnt/distributed-replicated/distributed-replicated02.txt [[email protected] ~]# cat /var/log/messages-20171214 > /mnt/distributed-replicated/distributed-replicated03.txt [[email protected] ~]# cat /var/log/messages-20171214 > /mnt/distributed-replicated/distributed-replicated04.txt [[email protected] ~]# cat /var/log/messages-20171214 > /mnt/distributed-replicated/distributed-replicated05.txt [[email protected] ~]# cat /var/log/messages-20171214 > /mnt/distributed-replicated/distributed-replicated05.txt [[email protected] ~]# cat /var/log/messages-20171214 > /mnt/distributed-replicated/distributed-replicated06.txt [[email protected] ~]# cat /var/log/messages-20171214 > /mnt/distributed-replicated/distributed-replicated07.txt [[email protected] ~]# cat /var/log/messages-20171214 > /mnt/distributed-replicated/distributed-replicated08.txt
1.6.4.2 查看写入的文件
# gluterfs01节点查看 [[email protected] ~]# tree /data/distributed-replicated0* /data/distributed-replicated01 ├── distributed-replicated01.txt ├── distributed-replicated02.txt ├── distributed-replicated03.txt └── distributed-replicated04.txt /data/distributed-replicated02 ├── distributed-replicated05.txt ├── distributed-replicated06.txt ├── distributed-replicated07.txt └── distributed-replicated08.txt 0 directories, 8 files # gluterfs02节点查看 [[email protected] ~]# tree /data/distributed-replicated0* /data/distributed-replicated03 ├── distributed-replicated01.txt ├── distributed-replicated02.txt ├── distributed-replicated03.txt └── distributed-replicated04.txt /data/distributed-replicated04 ├── distributed-replicated05.txt ├── distributed-replicated06.txt ├── distributed-replicated07.txt └── distributed-replicated08.txt 0 directories, 8 files
我的微信
如果有技术上的问题可以扫一扫我的微信
