Out with the old...

Posted by Grant Mongardi on Wed, Jan 20, 2010 @ 08:55 AM

Often, Archiving of Digital Assets and other legacy digital files is an afterthought. Production just runs out of live working storage, and needs to make room, so they drag "a bunch of files" off to an Archiving location that frees the working storage space for use with new projects. There is often little thought put into the process, and the files are just forgetten - until you need them again. If the original plan is lacking, it's possible that you'll find the system lacking.

So what do you need to think about?

 Lots of things, it turns out.

  •  How long will you need to keep the data?
  • Will you need to keep all of it that long?
  • How much can I spend to do implement?
  • How much can I spend to maintain it?
  • How much time do I need to implement it?
  • How often will I need to do it?
  • How quickly do I need to restore it?
  • What regulations or standards are already in place?

Wow. And that's just the big items. There are also lots of little things to consider as well. I hope you have some time, because this could take awhile.

What's first?

Probably the first thing you should consider is whether there are any regulatory or standards bodies that will have jurisdiction over your Archiving process. Many organizations are subject to rules set forth by standards and regulatory practices that institute rules or guidelines that you are required to follow. If you plan your Archiving scheme beforehand, you should be able to meet nearly all of the rules and regulations that these organizations represent. Some things you may encounter in you're foray into Archiving might be:

  • Sarbaines-Oxley
  • ISO 9001/9002/10012
  • ANSI
  • FDA
  • Military/Government Regulations

All of these have at the very least "good practices" guidelines that they expect you to follow. I won't get into them here, but if you're subject to any of those organizations oversight, you should look into the requirements that they impose upon you. Often, they will offer you nearly everything you need to make a well-informed decision.

How long will you need to keep the data?

That's just the simple question. If you're like most of our customers, you have lots of different types of digital assets. In addition to actual WIP, supplied artwork, and purchased artwork, you probably have fonts, process documents (.doc, .xls, etc), templates, etc. If so, you may want to manage these separately, as the need for maintaining (or even Archiving them) may vary depending upon their type and your requirements for storage. For instance, fonts and process documents might only need to be kept for a 2 year period, where actual artwork might need to be stored indefinitely. If you choose to manage these separately, it will make the archiving process more complex, but will save you on storage requirements and tape costs in the long run. In addition, you may decide that for some of your storage needs, charging the customer to maintain their assets is an option. In the WIP case, this is difficult to do, however once you've gotten into Archival data, then it becomes a much more viable option. Archival data requires Real Estate for storage and man-hours to maintain and manage. Those are line items that are difficult to argue. It's not like WIP storage, where storing the data is necessary to produce a product. Once the data is gone from WIP, it becomes a logistics issue, where tapes need to be stored either Nearline, Offline, or Offsite. Nearline storage is the most intrusive and expensive. It requires using up a valuable tape slot, making it unavailable to use for Disaster Recovery or other Archives. Offline means that the tapes need to be stored in a vault onsite, with climate controls and a person to manage those tapes. Offsite means just that - buying storage offsite in a climate controlled vault.

What media options do I have for Archive storage?

This depends upon the prior question and your desire or need for longevity. Generally speaking, for most organizations the choice is limited to magnetic tape, DVD/CD media, or nearline disk. There are other options, which you're welcome to investigate, but these three are the most popular. There are also promising technologies in the works, such as holographic storage and optical tape, but those are at least 5 years away.

Over the years, magnetic tape seems to be the most popular choice for long-term Archival storage of digital assets, although some do use DVD/CD storage, and more organizations are asking about the potential of nearlining assets to disk storage. Some of this decision does depend upon your budgets and the amount of data that you are Archiving, but only to a lesser extent. Nearly any organization can afford to purchase at the very least a stand-alone tape drive for archival purposes, and then just manually write files to tape. If you're still writing Archives to DVD/CD manually, you should probably consider the possibility of adding a stand-alone tape drive. It will make your process much more efficient, and allow you to store Archives in a more supportable and reliable fashion.

Magnetic tape is the most popular choice among our clients, as most require a Disaster Recovery backup, and have already chosen to have an attached library for that purpose. This makes the choice for tape much simpler. In general, magnetic tape stored properly will last 15-20 years or more. Tape degradation is possible, and over extended periods of time data loss can occur, however assuming that Archival tape is only written once, and the tapes are then stored (once full) in the proper environment, you should be perfectly fine assuming that the data will be there when you need it. That also assumes you remember where you put it in 20 years. NAPC generally recommends that any Archival data is written to two separate tapes. This ensures that whatever happens to one copy of an Archive, you will still have a second option for retreival. Tape, being a magnetic media, is vulnerable to magnetic fields and humidity, and to a lesser extent humidity and temperature variations, so proper storage is essential to longevity. Probably not a good idea to store it on top of a large electric motor, or inside the Large Hadron Collider room.

Anectdotally, DVD/CD media appears to be more prone to data degradation than does tape (or disk for that matter). DVD/CD media is very sensitive to bright light and temperature variations, as well as air pollution and humidity. Also, the quality of manufacture seems to dramatically effect this media's longevity, as does the age _before_ writing to the media. You can purchase all different types of writable DVD/CD media, some even with a gold media layer. The manufacturers may tell you it lasts as long as tape, however our experiences, and the experiences of our customers seems to contradict that. Mind you, duplicate DVD/CD Archives, which we also recommend with Tape-borne Archives, can alleviate most of these concerns, as having a duplicate copy of your Archives ensures that the files would need to be damaged on both media before you would ever lose a file. It's very difficult to guage an effective life expectancy however, as we've seen DVD/CD media that is unusable after only a year, and then some that has lasted 10 years.

Nearlining Archives to disk has been gaining ground lately. The idea is that you present inexpensive, slower disk to your Xinet server, and move Archives to that disk, and then just running a tape backup or mirror/snapshots of that data for Disaster Recovery purposes. Given the relatively small expense of adding a large quantity of disk to and existing Enterprise storage device and the simplification it provides, this has been becoming more attractive to larger organizations, as it allows them to run Enterprise-level backups along with their existing backup strategies. For some organizations, this makes documenting and managing these Disaster Recovery schemes simpler for the purposes of explaining them to standards and regulatory inspectors. That alone can sway the powers-that-be to allocate monies for the expense. The scenario is typically more expensive than tape, however, and unless there are other more enticing rewards for doing so, it can be a less viable option. The reliability is generally the same as that for live data, in that only if the device fails in such a way that data on the disk is lost, will the data be lost. Most enterprise-level storage maintains the data it stores via various error-checking schemes, so the chance of data loss even over lifetimes greatly exceeding that of tape are slim. However most organizations would replace the underlying device(s) long before that. I hope.

So which should I chose?

That's not an answer I can offer. It definitely requires some study on your part, and depends largely on what sort of work you do, and what your actual needs are. You'll probably spend a lot of time in meetings discussing the actual needs of your customers and end-users, and what exactly their requirements are, as well as what your needs and resources are. If you need any assistance or guidance in how to best approach this, you should feel free to call your NAPC Account Representative, and they will be happy to help point you in the right direction.

Where can I get more information?

The Council on Library and Information Resources has some good resources on media:

CD Media

Magnetic Tape

And as always, feel free to contact NAPC for any help in defining your Archiving strategy.