Home Server Projects
I've always been interested in installing servers for home use. The ability to customize Linux based servers to the degree that surpasses what is economically available from any other source makes Linux the obvious choice. Linux also offers the home user more latitude in that it will support practically any other desktop operating system with basic services.
Sunday 20 April 2008, 10:53 PM
Looking for True Permanence in File Storage
I guess it had to happen. I haven't been in any IT situation yet where users on their own managed to backup everything they wanted to keep permanently. Why should my home network be any different? My son's computer boot drive running Windows XP Pro cratered and totally trashed all of his digital pictures he's taken in the last year or so. Some of them are on Debbie but he didn't save all of them on her.
The publicized reports recently of the Google and the Carnegie-Mellon hard drive studies basically reinforced my prior experiences with hard drives. Years ago in the 60's, permanent storage was considered only to be possible on paper or on paper punch cards. IBM and Univac labeled magnetic memory storage on disc, drum or tape as temporary. It seems it still is. So the question becomes, what do you use to save family pictures and videos so that they are available for later family generations to look at?
Everybody seems to think of CDs and DVDs as permanent. Are they?
CD±RW and DVD±RW disks depend on laser pits recorded in layers of dye that gets “flipped” to another state to indicate a change in data state. Since I'm pretty uneducated about CDs and DVD technology, I have to do some study to find out what the materials engineers think of the “permanence” values of the various dyes used.
Kodak engineers did a study of their CDR's that uses fairly conservative methodology and they claim a lifetime of 100 to 217 years for their CDRs. But one thing stuck out, they stated that different manufacturers' CDR dyes had widely differing long-term stability. I suppose the trick will be to find out which manufacturer has the most stable dye layers.
My guess is that a CD should last longer than a DVD simply because the laser pits in the dye layer will likely be the biggest and therefore the signal to signal+noise level will be the best. Since the DVD packs more information in a smaller space, the data density is higher than a CD and the data bit size has to be smaller. This all ignores the differences of the laser light wavelengths used in the different optical disk recorders. At the same data density, blue laser light, a relatively short wavelength, used in DVDs has the ability to enable a better signal/signal+noise ratio compared to the longer red laser wavelength used for CDs. At the data density used on DVDs, there has to be some serious data signal processing used to extract reliable data.
Flash drive technology might be another choice. Yet another study area! Personal experience has shown me that flash drives are surprisingly pretty tough. I've accidently washed and dried a few USB flash drives that still work even after going for a hot ride in the electric dryer for a couple of hours. I've had one though that didn't survive its unexpected Magical Mystery Tour.
There are others looking into Digital Permanence and the big bugaboo is how long a time is defined as permanent. None of this really addresses directly the question I have about how to keep from losing permanently pictures my son thinks are cool (or whatever is the current meaning for: “Excellent”!). Even Debbie might suffer a hard drive failure induced lobotomy. As Carnegie-Mellon has shown, even hard drive Raid5 arrays are not really a viable answer.
The actual process I'll use to enforce a backup policy isn't anything more than a VB script run as a shutdown task or a scheduled task. So that part of the equation is easy enough. The other part, how to implement a reasonable storage scheme to push the data onto the selected “permanent” storage media is going to be tougher.
One possible scheme might be to use a 4 GB USB flash drive as the Samba 3 shared drive on Debbie. Have the scripts running on the Windows XP clients copy the saved files to a named folder on the shared flash drive. When the folder size gets large enough to fill a CDR, make it, verify the files and then delete them off the flash drive. Or just create a new sub-folder and rename it to match the folder name used in the scripts and keep the copied folder. When the USB flash begins to approach 95% filled, copy the entire flash off to a DVDR and empty the USB flash and start again. That will give you two opportunities to keep it for a long time on relatively static media.
PreviousComments on this post
It is actually amusing to read this as you mention about the "permanence" of CD/DVD media. When I started at Lorien (another microfilm & microfiche scanning bureau here in Sunrise, FL) some 12 years ago, CDRs were still in their infancy. I remember going through at least 4 or 5 different vendors trying to find the "best" (most reliable) media and went through the whole gambit of CDRs. At the time, the best ones were the gold-backed CDRs. I still have some of my backups from cerca 1997 and they still read perfectly. Granted, I have kept them stored in sleeves in binders, and all of my binders are kept in the house where there is plenty of air conditioning. Good storage of your media definitely promotes its longevity.
With DVDs, on the other hand, I have had a mountain of problems. They will burn and verify here at the office, but the client can't read them. Sometimes they read on one of their PCs but not others. One of the main reasons, at least as I have found, is the different media types (+R vs -R) and whether or not your DVD reader supports them. Most, if not all, of the newer drives support just about every media type that exists, but even up to a few years back you had to be very careful about which media you bought for your burner. Not to mention the problems with actually burning the discs in the first place. I have tried Roxio, Nero, and a few others, and all of them have had their quirks. The burning software I use now on Windows is "B's Recorder Gold5", version 5.50. It was already installed when I came onboard here, and once I started using it I found it to be the best software for burning media, especially DVDs. It has a very simple interface and does not install a lot of other miscellanea like Roxio and Nero.
If you want to check out the software, here is the link to their site:
B.H.A. Corporation
They have a much newer version which I have not yet tried, but if it is like their 5.50 version it will help if you want to backup your files to DVD.
Thanks for the link. Looks like its all Windows. I guess its my fault I didn't really make it clear that that Debbie is my Debian 4.0 Home Server, or DHS with Samba 3.0 on it for the Windows client machines to have storage. So the trick is to use a Debian compatible CD burner software in some sort of script that's run when the folder gets to a certain filled state close to a full CDR size. There's nothing that's really difficult about that, I just don't want to have to do it just yet. Just a case of the "lazies"!
I stayed up till midnight last night updating my son's P3 XP SP2 box after installing the SP2 image I got the last time his cpu cratered back in November 2007. 112 updates, mostly security patches, it took 2 hours to download and install using Windows Update. I've been waiting for SP3 and I can't wait till it gets released. I'll slipstream it into the image and hopefully be able to speed up the process for the next time.
Funny thing, I've been running Debian 4.0 on the server and 2 other systems and none of them have crashed since I installed them back in June last year. I've updated them now and then without any interruption in service. Gee what did I do wrong?
This comment has been deleted at the users request
