Centos7.1 – install a deployment ceph0.94

A: about CEPH
II: CEPH cluster hardware choices
Three: cluster installation configuration environment
1: System:
more /etc/redhat-release
CentOS Linux release 7.1.1503 (Core)
2: kernel:
uname -a
Linux localhost.localdomain 3.10.0-229.el7.x86_64 #1 SMP Fri Mar 6 11:36:42 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
3:CEPH versions:
ceph -v
ceph version 0.94.3
4: the deployment of cluster service distribution:
Running a service host name
ceph-deploy ceph-deploy
node241 mon osd
node242 mds osd
node243 osd
5:IP corresponds to the host name ceph-deploy node241 node242 node243
Four: host system configuring the cluster ready
All Ceph cluster nodes using CentOS version 7.1 (CentOS-7-x86_64-Minimal-1503-01.iso), all of the XFS file system recommended by Ceph official,
All nodes on the operating system are installed in a RAID1, other hard drive alone, without any RAID.
CentOS is installed on each node we need (including CEPH-deploy) to do the following configuration:
1: the system host name;
CentOS using the following command:
Hostnamectl set-hostname host name
2: Add the cluster hosts files between host names and host names can be resolved (add a host file do not use FQDN host name)
[root@ceph-deploy ceph]# more /etc/hosts localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 ceph-deploy node241 node242 node243
3: every SSH-copy-ID completed the server from SSH password;
ssh-copy-id node###
4: turn off the firewall, or open 6789/6800~6900 ports, turn off SELINUX;
# Turn off SELINUX
sed -i ‘s/SELINUX=enforcing/SELINUX=disabled/g’ /etc/selinux/config
setenforce 0
# Modify the SELINUX mode
sed -i ‘s/SELINUX=enforcing/SELINUX=disabled/’ /etc/sysconfig/selinux
sed -i ‘s/SELINUX=enforcing/SELINUX=disabled/’ /etc/selinux/config
# Ceph requires open ports
firewall-cmd –zone=public –add-port=6789/tcp –permanent
firewall-cmd –zone=public –add-port=6800-7100/tcp –permanent
firewall-cmd –reload
systemctl stop firewalld.server
5: Configure the NTP service, time service, guarantee the cluster server time;
Install NTP synchronization time
yum -y install ntp ntpdate ntp-doc
ntpdate 0.us.pool.ntp.org
hwclock –systohc
systemctl enable ntpd.service
systemctl start ntpd.service
Or add a crontab, automatic time synchronization;
[root@admin yum.repos.d]# more /etc/crontab
# For details see man 4 crontabs
# Example of job definition:
# .—————- minute (0 – 59)
# | .————- hour (0 – 23)
# | | .———- day of month (1 – 31)
# | | | .——- month (1 – 12) OR jan,feb,mar,apr …
# | | | | .—- day of week (0 – 6) (Sunday=0 or 7) OR sun,mon,tue,wed,thu,fri,sat
# | | | | |
# * * * * * user-name command to be executed
20 */1 * * * root ntpdate s1a.time.edu.cn && –systohc
6: System Optimization class
#set max user processes
sed -i ‘s/4096/102400/’ /etc/security/limits.d/20-nproc.conf
#set ulimit
cat /etc/rc.local | grep “ulimit -SHn 102400” || echo “ulimit -SHn 102400” >> /etc/rc.local
# Modify the maximum number of open file handles
cat /etc/security/limits.conf | grep “^* – sigpending 256612” ||
cat >>/etc/security/limits.conf<<EOF
* soft nofile 102400
* hard nofile 102400
* soft nproc 102400
* hard nproc 102400
* – sigpending 256612
Five: softwarebased package initialization:
1: installing the EPEL repository:
rpm -Uvh https://dl.fedoraproject.org/pub/epel/7/x86_64/e/epel-release-7-5.noarch.rpm
yum -y update
yum -y upgrade
2: commonly used packages, common tools, such as (non must be, recommended)
yum -y install gcc gcc-c++ autoconf libjpeg libjpeg-devel libpng libpng-devel freetype freetype-devel libxml2 libxml2-devel \
zlib zlib-devel glibc glibc-devel glib2 glib2-devel bzip2 bzip2-devel zip unzip ncurses ncurses-devel curl curl-devel e2fsprogs \
e2fsprogs-devel krb5-devel libidn libidn-devel openssl openssh openssl-devel nss_ldap openldap openldap-devel openldap-clients \
openldap-servers libxslt-devel libevent-devel ntp libtool-ltdl bison libtool vim-enhanced python wget lsof iptraf strace lrzsz \
kernel-devel kernel-headers pam-devel Tcl/Tk cmake ncurses-devel bison setuptool popt-devel net-snmp screen perl-devel \
pcre-devel net-snmp screen tcpdump rsync sysstat man iptables sudo idconfig git system-config-network-tui bind-utils update \
arpscan tmux elinks numactl iftop bwm-ng
3: fill in advance, according to the previous configuration deployment experience, install and configure the deployment process will continuously be wrong, partly because some dependent packages not installed causes, installed in advance to avoid unnecessary mistakes;
Centos6.x system configuration summary extraction method of pool, centos7. X version without installation, have successfully completed the cluster configuration, configuration if prompted to install missing packages can be taken into account.
for ip in $(cat /workspace/ceph/cephlist.txt);do echo —–$ip—–;ssh root@$ip yum install *argparse* redhat-lsb xfs* -y;done
3: install the deployment host (CEPH-deploy) installation preparation
4: create a CEPH working directories for all cluster hosts, later operations in the list below:
for ip in $(cat /workspace/ceph/cephlist.txt);do echo —–$ip—–;ssh root@$ip mkdir -p /etc/ceph ;done
5: establish the list of hosts
vim /workspace/ceph/cephlist.txt ceph-deploy node241 node242 node243
6: transfer key, hosts files to other nodes in the server
–Delete delete those files that SRC DST without
# Synchronize key files
for ip in $(cat /workspace/ceph/cephlist.txt);do echo —–$ip—–;rsync -avp –delete /root/.ssh/* $ip:/root/.ssh/;done
# Sync hosts file
Running a service host name
ceph-deploy ceph-deploy
node241 mon osd
node242 mds osd
node243 osd
1: install using CEPH-deploy to create the deployment section, MON information is generated:
Command format:
cd /etc/ceph/
ceph-deploy new node241
After a successful implementation would increase the three files in the directory
[root@ceph-deploy ceph]# ll
total 16
-rw-r–r– 1 root root 229 Aug 28 14:30 ceph.conf
-rw-r–r– 1 root root 5878 Aug 28 14:30 ceph.log
-rw——- 1 root root 73 Aug 28 14:30 ceph.mon.keyring
2: deploying cluster packages are installed:
CEPH packages are installed in all nodes CEPH-deploy install{CEPH-node}[{CEPH-node} ]
(Note: If you have yum installed on each machine the CEPH, this step can be omitted)
ceph-deploy install node241 node242 node243
3: Add initial monitor CEPH-deploy Mon create-initial and collect the secret key of the node {node}
ceph-deploy mon create-initial
4: collect keys
ceph-deploy gatherkeys node241
In the local directory, see the following key ring file:
1. {cluster-name}.client.admin.keyring
2. {cluster-name}.bootstrap-osd.keyring
3. {cluster-name}.bootstrap-mds.keyring
5: transport key configuration files to other nodes in the server
for ip in $(cat /workspace/ceph/cephlist.txt);do echo —–$ip—–;rsync -avp –delete /etc/ceph $ip:/etc/;done
–Delete delete those files that SRC DST without
This Monnode configuration basically come to an end;
6: Verify that the cluster node Mon installed successfully
[root@node241 ceph]# ceph -s
cluster 3adc4d9d-587e-424f-ab67-dd4d50493782
64 pgs stuck inactive
64 pgs stuck unclean
no osds
monmap e1: 1 mons at {node241=}
election epoch 2, quorum 0 node241
osdmap e1: 0 osds: 0 up, 0 in
pgmap v2: 64 pgs, 1 pools, 0 bytes data, 0 objects
0 kB used, 0 kB / 0 kB avail
64 creating
7: login Mon node to view a process has been started
ps -ef|grep ceph
[root@node241 ceph]# ps -ef|grep ceph
root 7087 1 0 15:04 ? 00:00:00 /bin/bash -c ulimit -n 32768; /usr/bin/ceph-mon -i node241 –pid-file /var/run/ceph/mon.node241.pid -c /etc/ceph/ceph.conf –cluster ceph -f
root 7097 7087 0 15:04 ? 00:00:00 /usr/bin/ceph-mon -i node241 –pid-file /var/run/ceph/mon.node241.pid -c /etc/ceph/ceph.conf –cluster ceph -f
root 9179 5080 0 16:59 pts/0 00:00:00 grep –color=auto ceph
Eight: install the cluster service OSD
1: perform OSD initialization commands
Each has 1 disk,/dev/sdb
Hard disk check Ceph storage nodes:
ceph-deploy disk list node241 node242 node243
Initializes the Ceph hard disk, then create OSD storage nodes (perform this step, OSD is automatically activated, do not perform activation);
for ip in $(cat /workspace/ceph/cephlist.txt);do echo —-$ip———–;ceph-deploy –overwrite-conf osd prepare $ip:/dev/sdb;done
2: verify:
root 13531 2 0 21:32 ? 00:00:00 [xfsaild/sdb1]
root 13844 1 0 21:32 ? 00:00:00 /bin/bash -c ulimit -n 32768; /usr/bin/ceph-osd -i 0 –pid-file /var/run/ceph/osd.0.pid -c /etc/ceph/
root 13848 13844 1 21:32 ? 00:00:00 /usr/bin/ceph-osd -i 0 –pid-file /var/run/ceph/osd.0.pid -c /etc/ceph/ceph.conf –cluster ceph -f
[root@ceph-deploy ceph]# ceph osd tree
-1 0.26999 root default
-2 0.09000 host node241
0 0.09000 osd.0 up 1.00000 1.00000
-3 0.09000 host node242
1 0.09000 osd.1 up 1.00000 1.00000
-4 0.09000 host node243
2 0.09000 osd.2 up 1.00000 1.00000
[root@node241 ceph]# ceph-disk activate-all
=== osd.0 ===
Starting Ceph osd.0 on node241…already running
[root@ceph-deploy ceph]# ceph health
Lsblk have a look if the SDK SDL is mounted on the/var/lib/CEPH/OSD/CEPH-0 CEPH, # no problem
[root@node241 ceph]# lsblk
sda 8:0 0 50G 0 disk
├─sda1 8:1 0 500M 0 part /boot
├─sda2 8:2 0 2G 0 part [SWAP]
└─sda3 8:3 0 47.5G 0 part /
sdb 8:16 0 100G 0 disk
├─sdb1 8:17 0 95G 0 part /var/lib/ceph/osd/ceph-0
└─sdb2 8:18 0 5G 0 part
sr0 11:0 1 1024M 0 rom
3: Finally, we profile generated from the CEPH-deploy synchronization deployment to several other nodes, and the CEPH consistent configuration of each node:
ceph-deploy –overwrite-conf admin ceph-deploy node241 node242 node243
Nine: Deploying MDS service is installed
1: Add the metadata server
cd /etc/ceph
ceph-deploy –overwrite-conf mds create node242
Command successfully executed without error to oK
2: State view. Has not been created when States.
For an MDS service that you just created, while the service is running, but its status until you create pools and file systems becomes Active.
[root@dn-5-221 ceph]# ceph mds stat
e1: 0/0/0 up
[root@ceph-deploy ceph]# ceph mds stat
e1: 0/0/0 up
[root@ceph-deploy ceph]# ceph mds dump
dumped mdsmap epoch 1
modified 2015-08-28 15:04:31.857259
compat compat={},rocompat={},incompat={}
metadata_pool 0
inline_data disabled
3: the following actions to create Filesystem
[root@ceph-deploy ceph]# ceph osd pool create cephfs_data 10
pool ‘cephfs_data’ created
[root@ceph-deploy ceph]# ceph osd pool create cephfs_metadata 10
pool ‘cephfs_metadata’ created
3: increases Mon node
A Ceph requires at least one storage cluster Ceph monitor. In order to ensure high availability, Ceph often run more than one storage cluster Ceph monitor, any single Ceph monitor downtime does not affect the entire Ceph storage clusters function properly. Because Ceph Paxos algorithm, which requires more Ceph monitor Quoram (such as 1,2:3,3:4,3:5,4:6).
To increased 2 sets of Ceph monitor your cluster.
ceph-deploy mon create {ceph-node}
ceph-deploy mon create ceph-node2 ceph-node3
As you increase the Ceph monitor, Ceph will begin to synchronize the monitor and the formation of Quoram. You can check the status Quoram as follows:
ceph quorum_status
ceph quorum_status –format json-pretty
Ceph offers 3 types of usage scenarios:
CephFS 1, distributed file systems.
Multiple client mount CephFS to the local, CephFS follows the POSIX interface, experience local like ext4 file system.
Similar to other distributed file systems, all CephFS clients share the same namespace.
2, RadosGW (rgw) object store.
Rgw using scenes is similar to Amazon S3, according to personal understanding is similar to seven cattle cloud storage.
3, block RBD (Rados Block Device).
Rbd Ceph provides virtual block devices, users of like using a SATA disk physical block device using the RBD.
Rbd is exclusive, each RBD block device is user private, relatively, how CephFS is shared.
Prime of the development of virtualization and cloud computing, IaaS combines RBD block device that uses wood fire, so one of RBD is Ceph community development center. This article is mainly from the perspective of RBD Ceph.
11: Configure the cleanup
################## Following CEPH-deploy node: ###########
During deployment, if any strange problems that cannot be solved, you can simply delete all over again:
All CEPH process sudo stop CEPH-all stop
CEPH-deploy uninstall [{CEPH-node}] uninstall all CEPH programs
CEPH-deploy purge [[CEPH-node} [{CEPH-node}] Delete CEPH related packages
CEPH-deploy purgedata {CEPH-node} [{CEPH-node}] Delete CEPH related packages
CEPH-deploy forgetkeys delete key
Clean your configuration file, rebuild the Setup configuration directory
for ip in $(cat /workspace/ceph/cephlist.txt);do echo —–$ip—–;ssh root@$ip rm -rf /etc/ceph/* ;done
for ip in $(cat /workspace/ceph/cephlist.txt);do echo —–$ip—–;ssh root@$ip mkdir -p /etc/ceph ;done
12: Troubleshooting:
The health checks
ceph health
Alert information is as follows:
# ceph health
HEALTH_WARN too few PGs per OSD (10 < min 30)
Increasing the number of PG, based on Total PGs = (#OSDs * 100)/pool size formula to determine pg_num (pgp_num should be set to the same as pg_num), 20*100/2=1000,Ceph refers to the number of times the official featured rounded to the nearest 2, so I chose 1024.
If all goes well, you should see HEALTH_OK:
# ceph osd pool set rbd size 2
set pool 0 size to 2
# ceph osd pool set rbd min_size 2
set pool 0 min_size to 2
# ceph osd pool set rbd pg_num 1024
set pool 0 pg_num to 1024
# ceph osd pool set rbd pgp_num 1024
set pool 0 pgp_num to 1024
# ceph health