Blogs

By Mary Mack

Recently, a new email-archiving challenge has arrived: as regulatory retention periods only get longer, many of these time frames are now longer than the average technical life of most commercial email archiving and enterprise information archiving solutions. As a result, one should incorporate at least one and in many case more conversions in the long term financial planning related to archiving electronic information. This is not as trivial as it looks, since almost all commercial archiving systems for electronic data – or vaults – are using proprietarily internal data structures and electronic data formats resulting in 100% vendor lockup when migrating data from one generation of a product to another.

For this reason, open and sustainable long-term archiving technology, already long in use by the federal government, should also be on the radar of commercial organizations.

Single instance storage may look tempting to save disk space in the short term, but in the long term it may cause serious other problems.  While single instance storage can save a great deal of money on high availability, high cost storage, the savings are not as great on lower availability, low cost storage. 

Most non-technical people are not familiar with the fact that single instance storage requires an internal (proprietary) database or lookup table to support the e-mail-archiving system. This not only causes huge processing delays when dealing with large data volumes and limits scalability of the archiving solution but also requires the original vendor to assist when the archives need to be migrated at a certain moment in time (for instance when the original manufacturer no longer supports the product).

This can result in vendor lockup or huge specialized conversion costs. As the price of disk space halves every 18 months, single instance storage is also not that relevant in the long term. This is why preventing vendor lock-up is a more important objective when selecting an e-mail archiving solution. Almost every commercial e-mail-archiving solution has a limited technical lifetime, which is often shorter than the (constantly extended by regulatory agencies) retention periods for sensitive e-mail communication. The issue of an expensive and time-consuming data migration is not an if-question but more and more a when-question. In the past, many e-mail archiving vendors have been acquired by large platform vendors, which resulted in in the end-of-life of the original product and a mandatory migration to the new product.

Gartnerstates that “Customers should expect the process of migration to be lengthy and potentially expensive, depending on the amount of data to be migrated. The alternative is to leave previously archived data in the original system; most organizations elect not to do this, as the costs and risks to maintain the original system over the life of the archived data are high.” 

Estimating (future) migration costs

In order to properly estimate future migration costs, many factors must be included.  According to an article by George Crumpon migration costs for storage (which can be extrapolated to application software migration costs) migration cost should include the following elements: 

•             Time and labor for migration and testing

•             Redundant operation of both old and new archives

•             Downtime and lost productivity

•             Software or hardware appliances to assist

•             Decommissioning the older archive

Altogether, migrations are very costly, and typically cost as much as 1 year of operation of the storage system.” 

Mailbox migration costs can be high even from one version to the next version of the same archive vendor.  Upgrading within Symantec’s Enterprise Vault, according to Transvault, a migration service, requires exporting mailboxes to PST from the old version and reimporting them into the new version. 

A recent costing exercise based on hiring a temp (at $18 per hour) to manually migrate PST files, and an average mailbox migration time of 3 hours indicated a cost of around $54 per mailbox. Bearing in mind this is just part of the process (the contents of PST files then need to be ingested into the target archive) manual migration costs can reach over $100 per mailbox. .

If your organization has a records management filing plan, the longest retention on the plan will certainly not be the requirement for e-mail archives. Retention periods are different for officers, heads of certain departments, and those under legal hold.  Assuming a 7-year retention requirement, almost all proprietary e-mail archive systems will require one conversion, and some will require two conversions.

The objectives (and benefits) of proper e-mail archiving:

• Preservation of valuable data and the protection of Intellectual Property
• Regulatory compliance and a tamper-proof data trail for e-mail for litigation, legal discovery and legal hold.
• E-mail backup, disaster recovery and records management.
• Ability to respond quickly to investigations, litigation and info requests.
• Reduction of future liability for unnecessary eDiscovery costs.
• Lower volume to collect (faster, cheaper).
• Deep Index (OCR, Embedded objects, metadata).
• Supports an efficient iterative process.
• Higher fidelity without unnecessary conversions and reconversions of native attachments and e-mail bodies.
• Documented with defensibility in mind.
• The format of the archive allows retrieval and analysis without requiring a closed proprietary system.
• The ability to recover corporate assets when employees are leaving.

There are commercial e-mail archives that store data on schemes based on XML for extremely long term availability and also in a SQL database for high speed, dynamic availability.  To use the system as deployed, the proprietary Archive software reads and writes to these two open formats. This allows our clients to use the archives in any process they wish.

NARA only accepts datain sustainable open file formats. One of those formats is XML, which has been audited and approved by NARA as part of the NSC e-mail archiving of both the Bush and the Obama administration. Many agencies in the US Government are using the types of processes advocated here, and are supporting them with XML, to great effect. These agencies are saving money and getting better performance and efficiency because their processes are thorough and their XML platform ensures them of the following:

• Sustainable, secure and enduring infrastructure, regardless of how much information is stored.
• Cost containment, due to no required conversions or high upfront costs.
• An array of supporting tools that greatly enhance capabilities for eDiscovery and e-Disclosure.

When a vendor upgrades or goes out of business, decisions are sometimes queued up to look at other technology.  Should you decide not upgrade to the current vendor’s new version or go with another vendor,  you can end up completely locked-in for the historical archive, because the original vendor may not exist or will not allow competitors to access their archives via an developers API’s.

Once an archive is scheduled for “End of Life”, it is an opportune time to get off the migration merry go round, and search for an archive that will stand the test of time, without handcuffing your IT and legal department.

In a new whitepaper E-mail Archiving: Your First Step towards Enterprise Information Archiving, we have done some calculations based on these studies to help with a Total Cost of Ownership (TCO) exercise.  We have a nice grid contrasting retention periods with the useful life of an archive to determine the number of migrations, and then a grid with costs depending on the number of migrations and number of mailboxes.  Click here to obtain the whitepaper..