atom feed10 messages in org.freebsd.freebsd-fsOdd file system corruption in ZFS pool
FromSent OnAttachments
Andrew ReillyApr 24, 2012 7:29 am 
Peter MaloneyApr 24, 2012 7:37 am 
Andrew ReillyApr 24, 2012 4:21 pm 
Bob FriesenhahnApr 25, 2012 6:58 am 
Peter MaloneyApr 25, 2012 8:36 am 
Andrew ReillyApr 25, 2012 8:32 pm 
Andrew ReillyApr 25, 2012 8:44 pm 
Bob FriesenhahnApr 26, 2012 7:23 am 
Peter JeremyApr 26, 2012 2:06 pm 
Andrew ReillyApr 28, 2012 5:05 am 
Subject:Odd file system corruption in ZFS pool
From:Andrew Reilly (arei@bigpond.net.au)
Date:Apr 24, 2012 7:29:50 am
List:org.freebsd.freebsd-fs

Hi there,

Had a power outage followed by an electrician visit, and the power supply to my server room yoyo'd a few times before I pulled the plugs. When I powered up again this afternoon I had ZFS pool corruption, according to zpool status.

Several zpool scrub/reboot cycles later, I have the odd situation of a directory entry that shows up in glob expansion, but which claims not to be there when ls does a stat, or when I try to move or delete it. Any suggestions?

Ls shows:

$ ls ls: .Suppliers.2010: No such file or directory

(Yes, this is in a Maildir managed jointly by qmail and dovecot.) There are also a bunch of message files in a different directory that find reports "unknown error 112" for, which doesn't look good.

Zpool status -v tank says:

pool: tank state: ONLINE status: One or more devices has experienced an error resulting in data corruption. Applications may be affected. action: Restore the file in question if possible. Otherwise restore the entire pool from backup. see: http://www.sun.com/msg/ZFS-8000-8A scan: scrub in progress since Wed Apr 25 00:12:48 2012 50.9G scanned out of 1.03T at 106M/s, 2h42m to go 0 repaired, 4.81% done config:

NAME STATE READ WRITE CKSUM tank ONLINE 0 0 853 raidz1-0 ONLINE 0 0 3.33K gptid/b06b6337-e511-11e0-9d62-00270e0fb8e9 ONLINE 0 0 0 gptid/b6c7d5b0-e511-11e0-9d62-00270e0fb8e9 ONLINE 0 0 0 gptid/bbf6c485-e511-11e0-9d62-00270e0fb8e9 ONLINE 0 0 0 gptid/bf86a966-e511-11e0-9d62-00270e0fb8e9 ONLINE 0 0 0

errors: Permanent errors have been detected in the following files:

tank/home:<0x0>

Every time I run zpool scrub and reboot the CKSUM column goes to zero, and the errors go away and the status says that the previous scrub found no errors. Every time I look at the missing directory the CKSUM column gets larger...

I do have a backup (I hope): on another zfs filesystem on a usb drive, constructed by sending and receiving snapshots in an incremental fashion. What I don't know is whether I actually need to use it or not. Everything seems to be working fine apart from this mysteriously untouchable Maildir directory and a few mail files, all of which I could recover from the backup if only I could remove the original. Should I just zfs destroy tank/home then create it again and send the last backup snapshots back? Would that remove the corruption?

Oh, I'm running 9-stable, and my Zpool and zfs versions are :

FreeBSD johnny.reilly.home 9.0-STABLE FreeBSD 9.0-STABLE #15: Sun Apr 22
11:37:17 EST 2012 ro@johnny.reilly.home:/usr/obj/usr/src/sys/GENERIC
amd64

ZFS filesystem version 5 ZFS storage pool version 28

Cheers,