[p4] Backup procedure for the journalfile? - Justin Hahn - com.perforce.perforce-user

6 messages in com.perforce.perforce-user[p4] Backup procedure for the journal...

From	Sent On	Attachments
Peter Steiner	01 Mar 2002 00:51
Schaible, Jorg	01 Mar 2002 02:31
Russell C. Jackson	01 Mar 2002 06:57
Jeff A. Bowles	01 Mar 2002 09:29
Justin Hahn	01 Mar 2002 09:48
Jeff A. Bowles	01 Mar 2002 10:26

Subject:	[p4] Backup procedure for the journalfile?
From:	Justin Hahn (je...@profitlogic.com)
Date:	03/01/2002 09:48:38 AM
List:	com.perforce.perforce-user

I'd like to suggest a couple slight changes to Jeff's extremely thorough suggestions.

[ snip lots of details I'm not going to disagree with. ]

b. In fact, find OUT how long it takes IS to pull files off backup tapes. b1. Send the following e-mail to them, "I need to recover a copy of /u2/perforce from machine 'xyzzy' from this morning's backup, could you please pull it down and put it somewhere on machine 'plover' for me?" and look at the clock. Write down the time. Every time you get e-mail on the subject in the thread, write down the time. Record how long it takes to the get files back.

If this time is unacceptable you really should be interacting with higher management to make it an acceptable time. Also, Jeff hasn't addressed the issue of whether IT (if they are a seperate group than perforce administration which may or may not be the case in some organization) is actually backing up what they say they are backing up.

Prompt backups and restores are crucial in any serious organization. If you can't get them, you can't survive a disaster. Likely other divisions, VPs and what not would be interested to hear about this sort of thing.

[ snip lots of good recovery ideas ]

It's not "to the second" backups, which aren't quite possible, but can get you closer and closer. Especially if you do that fileserver approach and do incremental copies to it during the day.

This is actually not quite true. If you're using veritas volume manager or veritas filesystem you can make filesystem or volume snapshots. Similary with EMC and Hitachi storage arrays. The methodologies differ (some involve syncing up and then later detaching a RAID mirror, others involve tracking changed block). NetApp filers also have filesystem snapshot functionality (though I'm not familiar with the NetApps...)

Your best bet is to integrate this into your backup procedure. If you can afford one of these very fancy and powerful storage "solutions", (and i highly recommend vxfs if any of you are using solaris or any of the other systems that it supports). The idea is to checkpoint the server, then take the snapshot. Do your backup from the snapshot. This way you can more likely guarantee internal consistency between the journal, the checkpoint and the versionned files. (You won't end up in the case that you have version skew between them.) This will make recovery a whole lot easier.

[Aside: There are several things in here that I mention but don't spend lots of time on. One is that you need to really know the weak points of your recovery process before you need them; it does NOT do to find out that the IS folks don't keep backups for more than two weeks, or that they take four days to respond to restore requests, when you have people breathing down YOUR neck to get the server back. Know those exposures. (Another is that "p4d -jj" trick. There are other ways into that sort of thing, which others will elaborate on.) Lastly, if you have multiple depots, you'll have to read "all depot trees on the server" for each time I wrote "depot/* above.)]

I'd like to stress that if you are in charge of a perforce server and you aren't part of IT, you should really do your damnedest to interact with IT. Backups should be of the highest importance to an IT group. Anyone who isn't taking them seriously is looking to get burned. (but I've heard some of Jeff's horror stories and had a few of my own, so I know this is a very common problem.)