MountVolume.SetUp failed for volume "pvc-78cf22fa-d776-43a4-98d7-d594f02ea018" : mount command failed, status: Failure, reason: failed to mount volume /dev/rbd13 [xfs] to /var/lib/kubelet/plugins/ceph.rook.io/rook-ceph/mounts/pvc-78cf22fa-d776-43a4-98d7-d594f02ea018, error mount failed: exit status 32 Mounting command: systemd-run Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/plugins/ceph.rook.io/rook-ceph/mounts/pvc-78cf22fa-d776-43a4-98d7-d594f02ea018 --scope -- mount -t xfs -o rw,defaults /dev/rbd13 /var/lib/kubelet/plugins/ceph.rook.io/rook-ceph/mounts/pvc-78cf22fa-d776-43a4-98d7-d594f02ea018 Output: Running scope as unit run-r2594d6c82152421c8891bfa8761e8c05.scope. mount: mount /dev/rbd13 on /var/lib/kubelet/plugins/ceph.rook.io/rook-ceph/mounts/pvc-78cf22fa-d776-43a4-98d7-d594f02ea018 failed: Structure needs cleaning

$ kubectl exec -it -n rook-ceph rook-ceph-tools-69f996589-ps4k7 bash
[root@twtpedev013 /]# rbd map --pool=replicapool pvc-78cf22fa-d776-43a4-98d7-d594f02ea018
[root@twtpedev013 /]# lsblk
NAME                                                                MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda                                                                   8:0    0   16G  0 disk
|-sda1                                                                8:1    0  731M  0 part
|-sda2                                                                8:2    0    1K  0 part
`-sda5                                                                8:5    0 15.3G  0 part
  |-ubuntu--vg-root                                                 252:0    0 14.3G  0 lvm  /etc/hosts
  `-ubuntu--vg-swap_1                                               252:1    0  976M  0 lvm
sdb                                                                   8:16   0 1000G  0 disk
`-ceph--4b83cf83--445d--42cc--8eeb--f7febf99fcab-osd--data--ee3fbf0e--4521--46fd--9879--e73048f7a697
                                                                    252:2    0  999G  0 lvm
sr0                                                                  11:0    1 1024M  0 rom
rbd0                                                                251:0    0   10G  0 disk
[root@twtpedev013 /]# mkdir test-mysql
[root@twtpedev013 /]# mount /dev/rbd0 test-mysql/
mount: mount /dev/rbd0 on /test-mysql failed: Structure needs cleaning
[root@twtpedev013 /]# fsck /dev/rbd0
fsck from util-linux 2.23.2
If you wish to check the consistency of an XFS filesystem or
repair a damaged filesystem, see xfs_repair(8).
[root@twtpedev013 /]# xfs_repair -L /dev/rbd0
Phase 1 - find and verify superblock...
- reporting progress in intervals of 15 minutes
Phase 2 - using internal log
- zero log...
ALERT: The filesystem has valuable metadata changes in a log which is being
destroyed because the -L option was used.
- scan filesystem freespace and inode maps...
agi unlinked bucket 35 is 8227 in ag 0 (inode=8227)
sb_ifree 228, counted 227
sb_fdblocks 2398405, counted 2426486
- 02:26:05: scanning filesystem freespace - 17 of 17 allocation groups done
- found root inode chunk
Phase 3 - for each AG...
- scan and clear agi unlinked lists...
- 02:26:05: scanning agi unlinked lists - 17 of 17 allocation groups done
- process known inodes and perform inode discovery...
- agno = 0
- agno = 15
- agno = 16
Metadata CRC error detected at xfs_bmbt block 0x2d9ff8/0x1000
data fork in ino 8226 claims free block 1112
correcting imap
Metadata CRC error detected at xfs_bmbt block 0x2d9ff8/0x1000
btree block 2/48127 is suspect, error -74
bad magic # 0 in inode 8227 (data fork) bmbt block 572415
bad data fork in inode 8227
cleared inode 8227
- agno = 1
- agno = 2
- agno = 3
- agno = 4
- agno = 5
- agno = 6
- agno = 7
- agno = 8
- agno = 9
- agno = 10
- agno = 11
- agno = 12
- agno = 13
- agno = 14
- 02:26:05: process known inodes and inode discovery - 384 of 384 inodes done
- process newly discovered inodes...
- 02:26:05: process newly discovered inodes - 17 of 17 allocation groups done
Phase 4 - check for duplicate blocks...
- setting up duplicate extent list...
- 02:26:05: setting up duplicate extent list - 17 of 17 allocation groups done
- check for inodes claiming duplicate blocks...
- agno = 0
- agno = 6
- agno = 7
- agno = 8
- agno = 9
- agno = 10
- agno = 11
- agno = 12
- agno = 13
- agno = 14
- agno = 15
- agno = 16
- agno = 1
- agno = 3
- agno = 4
- agno = 5
- agno = 2
- 02:26:05: check for inodes claiming duplicate blocks - 384 of 384 inodes done
Phase 5 - rebuild AG headers and trees...
- 02:26:05: rebuild AG headers and trees - 17 of 17 allocation groups done
- reset superblock...
Phase 6 - check inode connectivity...
- resetting contents of realtime bitmap and summary inodes
- traversing filesystem ...
- traversal finished ...
- moving disconnected inodes to lost+found ...
Phase 7 - verify and correct link counts...
- 02:26:05: verify and correct link counts - 17 of 17 allocation groups done
Metadata corruption detected at xfs_bmbt block 0x2d9ff8/0x1000
libxfs_writebufr: write verifer failed on xfs_bmbt bno 0x2d9ff8/0x1000
Maximum metadata LSN (82836:5824) is ahead of log (1:64).
Format log to cycle 82839.
releasing dirty buffer (bulk) to free list!done
[root@twtpedev013 /]# mount /dev/rbd0 test-mysql/
[root@twtpedev013 /]# ls test-mysql/
#ib_16384_0.dblwr binlog.000009 client-cert.pem ib_logfile1 performance_schema sys
#ib_16384_1.dblwr binlog.index client-key.pem ibdata1 private_key.pem undo_001
#innodb_temp binlog.~rec~ eams ibtmp1 public_key.pem undo_002
auto.cnf ca-key.pem ib_buffer_pool mysql server-cert.pem
binlog.000008 ca.pem ib_logfile0 mysql.ibd server-key.pem
[root@twtpedev013 /]# umount /dev/rbd0
[root@twtpedev013 /]# rbd unmap /dev/rbd0