ceph

一次因磁盘扇区损坏引起osd 出现down的问题

2016 年 12 月 14 日 admin 发表回复

当osd出现down的状态，日志信息显示为：

f049a9f5700 -1 os/FileStore.cc: In function 'virtual int FileStore::read(coll_t, const ghobject_t&, uint64_t, size_t, ceph::bufferlist&, uint32_t, bool)' thread 7f049a9f5700 time 
os/FileStore.cc: 2850: FAILED assert(allow_eio || !m_filestore_fail_eio || got != -5)

1 2	f049a9f5700 -1 os/FileStore.cc: In function 'virtual int FileStore::read(coll_t, const ghobject_t&, uint64_t, size_t, ceph::bufferlist&, uint32_t, bool)' thread 7f049a9f5700 time os/FileStore.cc: 2850: FAILED assert(allow_eio \|\| !m_filestore_fail_eio \|\| got != -5)

通过assert（allow_eio || !m_filestore_fail_eio || got != -5),非allow_eio且配置中fail_eio为true时，若有IO error则assert fail。

分析osd 所对应的磁盘信息：

# dmesg -T | grep sdh
sd 0:2:7:0: [sdh] 3904294912 512-byte logical blocks: (1.99 TB/1.81 TiB)
sd 0:2:7:0: [sdh] Write Protect is off
sd 0:2:7:0: [sdh] Mode Sense: 1f 00 00 08
sd 0:2:7:0: [sdh] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
 sdh: unknown partition table
sd 0:2:7:0: [sdh] Attached SCSI disk
 XFS (sdh): Mounting V4 Filesystem
 XFS (sdh): Ending clean mount
 sd 0:2:7:0: [sdh]
 sd 0:2:7:0: [sdh]
 sd 0:2:7:0: [sdh]
 sd 0:2:7:0: [sdh] CDB:
 end_request: I/O error, dev sdh, sector 1100294168
 sd 0:2:7:0: [sdh]
 sd 0:2:7:0: [sdh]
 sd 0:2:7:0: [sdh]
 sd 0:2:7:0: [sdh] CDB:
 end_request: I/O error, dev sdh, sector 1100294400
 sd 0:2:7:0: [sdh]
 sd 0:2:7:0: [sdh]
 sd 0:2:7:0: [sdh]
 sd 0:2:7:0: [sdh] CDB:
 end_request: I/O error, dev sdh, sector 1100294400
 sd 0:2:7:0: [sdh]
 sd 0:2:7:0: [sdh]
 sd 0:2:7:0: [sdh]
 sd 0:2:7:0: [sdh] CDB:
 end_request: I/O error, dev sdh, sector 1100294168
 sd 0:2:7:0: [sdh]
 sd 0:2:7:0: [sdh]
 sd 0:2:7:0: [sdh]
 sd 0:2:7:0: [sdh] CDB:
 end_request: I/O error, dev sdh, sector 1100294400
 sd 0:2:7:0: [sdh]
 sd 0:2:7:0: [sdh]
 sd 0:2:7:0: [sdh]
 sd 0:2:7:0: [sdh] CDB:
 end_request: I/O error, dev sdh, sector 1100294400
 sd 0:2:7:0: [sdh]
 sd 0:2:7:0: [sdh]
 sd 0:2:7:0: [sdh]
 sd 0:2:7:0: [sdh] CDB:
 end_request: I/O error, dev sdh, sector 1100294352
 Buffer I/O error on device sdh, logical block 1100294352
 Buffer I/O error on device sdh, logical block 1100294353
 Buffer I/O error on device sdh, logical block 1100294354
 Buffer I/O error on device sdh, logical block 1100294355
 Buffer I/O error on device sdh, logical block 1100294356
 Buffer I/O error on device sdh, logical block 1100294357
 Buffer I/O error on device sdh, logical block 1100294358
 Buffer I/O error on device sdh, logical block 1100294359
 Buffer I/O error on device sdh, logical block 1100294360
 sd 0:2:7:0: [sdh]
 sd 0:2:7:0: [sdh]
 sd 0:2:7:0: [sdh]
 sd 0:2:7:0: [sdh] CDB:
 end_request: I/O error, dev sdh, sector 1100294405
 sd 0:2:7:0: [sdh]
 sd 0:2:7:0: [sdh]
 sd 0:2:7:0: [sdh]
 sd 0:2:7:0: [sdh] CDB:
 end_request: I/O error, dev sdh, sector 1100294405
 sd 0:2:7:0: [sdh]
 sd 0:2:7:0: [sdh]
 sd 0:2:7:0: [sdh]
 sd 0:2:7:0: [sdh] CDB:
 end_request: I/O error, dev sdh, sector 1100294405

# dmesg -T | grep sdh

sd 0:2:7:0: [sdh] 3904294912 512-byte logical blocks: (1.99 TB/1.81 TiB)

sd 0:2:7:0: [sdh] Write Protect is off

sd 0:2:7:0: [sdh] Mode Sense: 1f 00 00 08

sd 0:2:7:0: [sdh] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA

sdh: unknown partition table

sd 0:2:7:0: [sdh] Attached SCSI disk

XFS (sdh): Mounting V4 Filesystem

XFS (sdh): Ending clean mount

sd 0:2:7:0: [sdh]

sd 0:2:7:0: [sdh] CDB:

end_request: I/O error, dev sdh, sector 1100294168

sd 0:2:7:0: [sdh]

sd 0:2:7:0: [sdh] CDB:

end_request: I/O error, dev sdh, sector 1100294400

sd 0:2:7:0: [sdh]

sd 0:2:7:0: [sdh] CDB:

end_request: I/O error, dev sdh, sector 1100294400

sd 0:2:7:0: [sdh]

sd 0:2:7:0: [sdh] CDB:

end_request: I/O error, dev sdh, sector 1100294168

sd 0:2:7:0: [sdh]

sd 0:2:7:0: [sdh] CDB:

end_request: I/O error, dev sdh, sector 1100294400

sd 0:2:7:0: [sdh]

sd 0:2:7:0: [sdh] CDB:

end_request: I/O error, dev sdh, sector 1100294400

sd 0:2:7:0: [sdh]

sd 0:2:7:0: [sdh] CDB:

end_request: I/O error, dev sdh, sector 1100294352

Buffer I/O error on device sdh, logical block 1100294352

Buffer I/O error on device sdh, logical block 1100294353

Buffer I/O error on device sdh, logical block 1100294354

Buffer I/O error on device sdh, logical block 1100294355

Buffer I/O error on device sdh, logical block 1100294356

Buffer I/O error on device sdh, logical block 1100294357

Buffer I/O error on device sdh, logical block 1100294358

Buffer I/O error on device sdh, logical block 1100294359

Buffer I/O error on device sdh, logical block 1100294360

sd 0:2:7:0: [sdh]

sd 0:2:7:0: [sdh] CDB:

end_request: I/O error, dev sdh, sector 1100294405

sd 0:2:7:0: [sdh]

sd 0:2:7:0: [sdh] CDB:

end_request: I/O error, dev sdh, sector 1100294405

sd 0:2:7:0: [sdh]

sd 0:2:7:0: [sdh] CDB:

end_request: I/O error, dev sdh, sector 1100294405

可以看到I/O Error信息，截取一段信息进行分析

badblocks -v -s -b 512 -o /root/badblocks.txt /dev/sdh 1100300000 1100290000
Checking blocks 1100290000 to 1100300000
Checking for bad blocks (read-only test): done
Pass completed, 8 bad blocks found. (8/0/0 errors)
[root@host20 ~]# cat badblocks.txt
1100294400
1100294401
1100294402
1100294403
1100294404
1100294405
1100294406
1100294407

badblocks -v -s -b 512 -o /root/badblocks.txt /dev/sdh 1100300000 1100290000

Checking blocks 1100290000 to 1100300000

Checking for bad blocks (read-only test): done

Pass completed, 8 bad blocks found. (8/0/0 errors)

[root@host20 ~]# cat badblocks.txt

1100294400

1100294401

1100294402

1100294403

1100294404

1100294405

1100294406

1100294407

发现1100294400 -4407被检测出来。但1100294352 – 1100294360 却没有（执行badblcoks命令仅使用了读模式检测）

ceph

docker 利用ipvlan 部署ceph 集群

2016 年 12 月 13 日 admin 发表回复

最新版本Docker网络部分所用的macvlan和ipvlan

$ su root

# rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org

# yum install http://www.elrepo.org/elrepo-release-7.0-2.el7.elrepo.noarch.rpm

# yum --enablerepo=elrepo-kernel install kernel-ml

$ su root

# rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org

# yum install http://www.elrepo.org/elrepo-release-7.0-2.el7.elrepo.noarch.rpm

# yum --enablerepo=elrepo-kernel install kernel-ml

下载插件

[root@ceph_docker ~]# git clone https://github.com/gopher-net/ipvlan-docker-plugin.git
Cloning into 'ipvlan-docker-plugin'...
remote: Counting objects: 874, done.
remote: Total 874 (delta 0), reused 0 (delta 0), pack-reused 874
Receiving objects: 100% (874/874), 22.40 MiB | 566.00 KiB/s, done.
Resolving deltas: 100% (225/225), done.
[root@ceph_docker ~]# git clone https://github.com/gopher-net/ipvlan-docker-plugin.git
Cloning into 'ipvlan-docker-plugin'...
remote: Counting objects: 874, done.
remote: Total 874 (delta 0), reused 0 (delta 0), pack-reused 874
Receiving objects: 100% (874/874), 22.40 MiB | 566.00 KiB/s, done.
Resolving deltas: 100% (225/225), done.

[root@ceph_docker ~]# git clone https://github.com/gopher-net/ipvlan-docker-plugin.git

Cloning into 'ipvlan-docker-plugin'...

remote: Counting objects: 874, done.

remote: Total 874 (delta 0), reused 0 (delta 0), pack-reused 874

Receiving objects: 100% (874/874), 22.40 MiB | 566.00 KiB/s, done.

Resolving deltas: 100% (225/225), done.

[root@ceph_docker ~]# git clone https://github.com/gopher-net/ipvlan-docker-plugin.git

Cloning into 'ipvlan-docker-plugin'...

remote: Counting objects: 874, done.

remote: Total 874 (delta 0), reused 0 (delta 0), pack-reused 874

Receiving objects: 100% (874/874), 22.40 MiB | 566.00 KiB/s, done.

Resolving deltas: 100% (225/225), done.

运行该插件，并指定接口enp5sf0
[root@ceph_docker binaries]# ./ipvlan-docker-plugin-0.3-Linux-x86_64 –host-interface “enp5s0f0″ -d

创建docker的网络模块
[root@ceph_docker ~]# docker network create -d ipvlan –subnet=192.168.19.0/24 –gateway=192.168.19.1 -o ipvlan_mode=l2 -o parent=enp5s0f0 db_net
a4bfe81593960c9a42ed7a9c89f5de8100f2efbec46aadb0bf10b8d1c6a33ea3
查看当前网络模块
[root@ceph_docker ~]# docker network ls

NETWORK ID          NAME                DRIVER
a4bfe8159396        db_net              ipvlan              
b7e4d3265869        bridge              bridge              
fc5183b62ec3        none                null                
466a69268805        host                host

NETWORK ID NAME DRIVER

a4bfe8159396 db_net ipvlan

b7e4d3265869 bridge bridge

fc5183b62ec3 none null

466a69268805 host host

开始部署容器，

docker run -d --net=db_net --ip=192.168.19.103 -v /etc/ceph:/etc/ceph -v /var/lib/ceph/:/var/lib/ceph/ -e MON_IP=192.168.19.103 -e CEPH_PUBLIC_NETWORK=192.168.19.0/24  ceph/daemon mon

 docker run -d --net=db_net --ip =192.168.19.104 --privileged=true -v /etc/ceph:/etc/ceph -v /var/lib/ceph/:/var/lib/ceph/ -v /dev/:/dev/ -e OSD_DEVICE=/dev/sdb -e OSD_FORCE_ZAP=1 ceph/daemon osd

[root@ceph_docker ~]# docker run -d --net=db_net --ip =192.168.19.105 --privileged=true -v /etc/ceph:/etc/ceph -v /var/lib/ceph/:/var/lib/ceph/ -v /dev/:/dev/ -e OSD_DEVICE=/dev/sdc -e OSD_TYPE=disk  ceph/daemon osd

docker run -d --net=db_net --ip=192.168.19.103 -v /etc/ceph:/etc/ceph -v /var/lib/ceph/:/var/lib/ceph/ -e MON_IP=192.168.19.103 -e CEPH_PUBLIC_NETWORK=192.168.19.0/24 ceph/daemon mon

docker run -d --net=db_net --ip =192.168.19.104 --privileged=true -v /etc/ceph:/etc/ceph -v /var/lib/ceph/:/var/lib/ceph/ -v /dev/:/dev/ -e OSD_DEVICE=/dev/sdb -e OSD_FORCE_ZAP=1 ceph/daemon osd

[root@ceph_docker ~]# docker run -d --net=db_net --ip =192.168.19.105 --privileged=true -v /etc/ceph:/etc/ceph -v /var/lib/ceph/:/var/lib/ceph/ -v /dev/:/dev/ -e OSD_DEVICE=/dev/sdc -e OSD_TYPE=disk ceph/daemon osd

查看部署后的状态

root@6d6aa76ce897:/# ceph -s
    cluster 7f12aae0-fa5c-4ec3-934a-d51969fabefe
     health HEALTH_OK
     monmap e3: 3 mons at {43d5248d080c=192.168.19.102:6789/0,6d6aa76ce897=192.168.19.103:6789/0,87f923e5c26c=192.168.19.101:6789/0}
            election epoch 6, quorum 0,1,2 87f923e5c26c,43d5248d080c,6d6aa76ce897
     osdmap e13: 3 osds: 3 up, 3 in
            flags sortbitwise
      pgmap v31: 64 pgs, 1 pools, 0 bytes data, 0 objects
            115 MB used, 5582 GB / 5582 GB avail
                  64 active+clean

root@6d6aa76ce897:/# ceph -s

cluster 7f12aae0-fa5c-4ec3-934a-d51969fabefe

health HEALTH_OK

monmap e3: 3 mons at {43d5248d080c=192.168.19.102:6789/0,6d6aa76ce897=192.168.19.103:6789/0,87f923e5c26c=192.168.19.101:6789/0}

election epoch 6, quorum 0,1,2 87f923e5c26c,43d5248d080c,6d6aa76ce897

osdmap e13: 3 osds: 3 up, 3 in

flags sortbitwise

pgmap v31: 64 pgs, 1 pools, 0 bytes data, 0 objects

115 MB used, 5582 GB / 5582 GB avail

64 active+clean

ceph

ceph的手工分区

2016 年 12 月 7 日 admin 发表回复

ceph的手工磁盘格式化，10.2中。采用了udev的方式，所以typecode的id不能出错
sgdisk –zap-all /dev/$DEV
sgdisk –clear –mbrtogpt — /dev/$DEV
journal_uuid=$(uuidgen)
sgdisk –new=’2:0:5120M’ –change-name=2:’ceph journal’ –partition-guid=${journal_uuid} –typecode=2:45b0969e-9b03-4f30-b4c6-b4b80ceff106 –mbrtogpt — /dev/$DEV
osd_uuid=$(uuidgen)
sgdisk –largest-new=1 –change-name=1:”ceph data” –partition-guid=1:${osd_uuid}4–typecode=1:4fbd7e29-9d25-41b8-afd0-062c0ceff05d — /dev/$DEV

#partx -a /dev/$DEV
partprobe /dev/$DEV

mkfs.xfs /dev/${DEV}1 -f

SELinux+

月度归档：2016年12月

一次因磁盘扇区损坏引起osd 出现down的问题

docker 利用ipvlan 部署ceph 集群

ceph的手工分区

基础分享安全

2016年十二月
一	二	三	四	五	六	日
« 四				一 »
			1	2	3	4
5	6	7	8	9	10	11
12	13	14	15	16	17	18
19	20	21	22	23	24	25
26	27	28	29	30	31

基础 分享 安全

基础分享安全