For the last couple of days, I had a VM sitting on an LVM volume with bad sectors. Here’s how you find out what LVM volume is causing the problem so that you can migrate off of it. Firstly, we want to look at the dmesg output to find out what LBA is bad:
[1381493.960771] Buffer I/O error on device sda5, logical block 380731029 [1381493.960771] Buffer I/O error on device sda5, logical block 380731030
This means the bad LBA is 380731029. so, we need to determine the partition offset next. To do this, we do the following:
# sfdisk -luS /dev/sda Disk /dev/sda: 1430807 cylinders, 64 heads, 32 sectors/track Warning: extended partition does not start at a cylinder boundary. DOS and Linux will interpret the contents differently. Warning: The partition table looks like it was made for C/H/S=*/255/63 (instead of 1430807/64/32). For this listing I'll assume that geometry. Units = sectors of 512 bytes, counting from 0 Device Boot Start End #sectors Id System /dev/sda1 63 19535039 19534977 83 Linux /dev/sda2 19535040 21494969 1959930 82 Linux swap / Solaris /dev/sda3 21494970 2930288129 2908793160 5 Extended /dev/sda4 0 - 0 0 Empty /dev/sda5 21495033 1475891549 1454396517 8e Linux LVM /dev/sda6 1475891613 2930288129 1454396517 8e Linux LVM
We subtract 21495033 from 380731029 to yield 359235995. Next, we determine the extent size in kilobytes:
# pvdisplay /dev/sda5 --- Physical volume --- PV Name /dev/sda5 VG Name aurora-1 PV Size 693.51 GB / not usable 2.55 MB Allocatable yes PE Size (KByte) 4096 <-- 4096KB. Total PE 177538 Free PE 2114 Allocated PE 175424 PV UUID mz2kUU-h19J-CpBc-Q0lX-Spq7-D3hg-qoexGa
A kilobyte is 1024 bytes. An LBA block is 512 bytes. So we multiply 4096 by 2 (8192), and divide 359235995 by 8192 giving us 43852, which is the extent number on the physical volume. The next step is obvious, read through /etc/lvm/backup/vgname until you find a volume that extent 43852 falls between! Then recover the data off of that volume and stop using it (or maybe try badblock recovery or something).