As a profession we are very proud of our principles.  If you ever discuss technology with a group of records managers one of us is bound to say  'records management principles are timeless, regardless of how much or how quickly technology changes'.

But what exactly are these principles?

The time when you most need a set of principles is when new ideas, tools, technologies or approaches come onto being.   By definition we have little or no practical experience of these new ideas/tools/ technologies.  We need a set of principles which distills our past experience of what has and hasn't worked, in order to predict whether these new approaches are likely to work.

NARA's report on automated approaches to records management

In March 2014 the  US National Archives (NARA) issued a report on the different ways in which records management could be automated.   The approaches have little in common with each other except that they all aim reduce the burden of records management tasks on end users as compared with the more established electronic records management system approach.

The approaches discussed in the report were extremely varied, but can be boiled down to the following:

• an in -place approach based on holding a records classification and related retention rules in one application, and applying them to content held in the various different native applications of the organisation (SharePoint, e-mail accounts, shared drives etc.)

• a workflow approach -  the definition of workflows for each activity, that include provision for the capture of records at particular points of the process

• a defensible disposition approach - the definition of policies to apply to aggregations of content around the organisation.    For e-mail accounts you might have a policy that the e-mail accounts of individuals deemed to have important roles in the organisation are kept permanently, whilst the e-mail account's of less important individuals are kept for six years.  You might set a policy that content in SharePoint sites/shared drives is deleted after five years/fifteen years. or kept permanently depending on the importance of the functions carried out by the team concerned.   There is no retention schedule, and no records classification, just risk based decisions.

• automated filing by a rules engine  through the definition of rules to enable a rules engine to recognise which content arises from which particular activity

• automated filing by an auto-classification tool  the  use of machine learning to learn the attributes of content that arises from particular activites,  without the organisation having to write a rule set.

(I have taken the liberty of changing NARA's categories slightly - this summary  sticks more closely to NARA's definitions)

 These new approaches are very different from the precious electronic records management system approach, but there has been no change in records management theory in between the two approaches (unless you count the new ideas from the information governance world - such as 'everything held is a record'  'big buckets are better than granular hierarchies' 'end-users should not have to bear the burden of records management')

NARA gives the pros and cons of each automated approach, without favouring any one or other of them, and without stating whether or not they believe each approach will keep records to an acceptable standard on a corporate scale.    This is not NARA's fault - it is simply a recognition of the fact that at the moment we as a profession have no handy set of criteria to evaluate these approaches against.

In this blogpost I am going to nominate what I think is the  most useful and concise set of criteria for judging a records management system or approach- namely the five characteristics of a reliable records system that were developed in Australia in the early 1990s and ended up as section  8.2.2 of  the International Records Management Standard ISO15489.

The five characteristics of a reliable records system

Reliability is the most important characteristic of a record system.  A record system exists to perform a paradoxical function.  It exists to both:

•  enable external stakeholders to scrutinise an organisation AND

•  to enable the organisation to defend itself against external scrutiny

This paradox is fractal - it is present at any level of aggregation:

• a record system enables an organisation to scrutinise a team, and it enables the team to defend itself from scrutiny

• it enables an individual to be scrutinised and to defend him/herself from scrutiny.

The only way that a records system can resolve this paradox is by being trusted by all parties - in other words for all parties to consider the system to be reliable  - the individuals carrying out a  piece of work, their immediate colleagues and line management, the rest of the organisation, and external stakeholders.

Section 8.2.2.of the ISO 15489 standard states that in order to be considered reliable a records system must:

• routinely and comprehensively capture all records arising from the activities that it covers

• act as the main source of reference for the activities it covers

• link records to the activities from which they arose

• protect records from amendment or deletion

• preserve access to records over time

Why the five characteristics of reliability are essential, rather than merely desirable

These characteristics are not 'nice to have' they are 'must have'.

Think what would happen if a records management system did not meet even one of these characteristics:

• If records were not consistently captured into the record system then external stakeholders could wonder wether there was a bias in record capture, for example whether content that could be incriminating/embarrassing was deliberately kept off the record system

• If a record system does not comprehensively capture all records arising from an activity then this will leave gaps in the record and weaken the organisation's ability to defend or learn from the way it carried out that activity.  It will also lead to external stakeholders looking to other sources of information outside of the records system - sources that may be more time consuming and more embarrassing for the organisation to search.

• if a record system does not serve as the main source of reference for the activities within its scope then any gaps in the record will go unnoticed, and uncorrected.   It is particularly important that the record system is used as source of reference by colleagues carrying out the piece of work itself - because they are the only people in a position to know that there is content missing from the record.

• if records are not linked to the activity from which they arose then the organisation will find it impossible to apply a precise retention period to those records.   Retention periods are specific to a type of activity:  managing staff,  designing a building, managing a project, bringing a pharmaceutical product to market,  adjudicating on a citizen benefit claim etc..  The trigger point for that retention period to start is even more specific, it is specific to a particular instance of each activity:   the date a particular member of staff left employment;  the date a particular building ceased to exist; the date a particular project finished, the date a particular pharmaceutical product was withdrawn from the market, the date a particular citizen ceased receiving benefit etc.

• If records are not protected from amendment or deletion then an external stakeholder will have cause to doubt whether or not content detrimental to the organisation, or to a particular team or individual, has been deleted from the system prior to them viewing the record

• If access to records is not preserved over time then an organisation cannot be sure that if will be able to defend itself from scrutiny/challenge if that scrutiny or challenge comes at a date in the future.  By the same token external stakeholders cannot be sure that they will be able to scrutinise the organisation at a future date.

Why have we lost site of these reliabilty criteria?

At first site it seems odd that we as a profession have lost site of these criteria for judging a records system.

These criteria are not obscure.    They were a fundamental part of the drive by the records management and archives professions to manage the transition from the paper age to the networked digital age by expressing the attributes of good recordkeeping systems in an abstract, non-format specific way.

Although these criteria are trying to be as timeless as possible, they are also very much of their time.  They were written at the start of the 1990s, just before the mass adoption of e-mail and networked computers with shared drives  by organisations in what were then called developed economies. They were embedded in the Australian records management standard which later became the International Records Standard (ISO 15489, published in 2001).

The reason why we have lost site of them is that we have not been able to implement systems that meet all five of these criteria,  on the scale we want to work at (the corporate scale) since the mass adoption of e-mail.  Nor is there a realistic likelihood that any of the five automated approaches discussed by NARA will meet all five of these criteria.

Evaluating existing record keeping systems against the five reliability criteria

When you compare existing systems within organisations to these reliability criteria you find that:

• line of business systems (case file systems and sector specific systems such as insurance claim systems) can be set up to routinely meet all five reliability criteria.   They can use functional e-mail addresses, and/or web forms, to divert correspondence related to that function away from private e-mail accounts directly into the system, which means they can routinely and comprehensively capture correspondence arising from  the activities that they cover.   However each line of business system can only cover one area of the business.    Organisations carry out  so many different types of work that it is impossible to have a line of business system for each of them.

• generic document management repositories such as shared drives, electronic records management systems, and SharePoint cannot routinely and comprehensively capture business correspondence sent and received through e-mail - they are dependent on individuals exercising their own judgement on which e-mails (if any) they upload or push to the system.    None of these three types of repository serve as the main source of reference to all or most of the activities that they cover.    Individuals tend to use their own e-mail accounts as the first place  to go to find records of their work.   Of those three repositories electronic records management systems and, to a lesser extent SharePoint, do a better job than  shared drives at protecting records and linking them to the business activities that they arose from.

• e-mail archives routinely and comprehensively capture electronic correspondence.  However they do not relate records to a business activity which leads to problems with applying access and retention rules .  They cannot be used as the main source of reference for activities by anyone except Legal Counsel because private and sensitive e-mails are undifferentiated from other correspondence

Evaluating automated approaches against the five reliability criteria

The automated approaches described in the NARA report  either don't meet all five reliability criteria or they don't scale across an organisation

• the in -place approach - currently lacks an answer to the question of how  to routinely and comprehensively capture important e-mails and relate them to business activities.  It is dependent on either individuals dragging e-mails into folders (which is not routine or comprehensive because many individuals never file e-mail into folders) or one of the automated filing approaches (see below)

• a workflow approach - works well, meets all five criteria, but cannot scale across an organisation because of the time taken to analyse processes and define workflows

• a defensible disposition approach - does not protect records.   A disposition rule is applied to a part of a shared drive or a SharePoint site but indviduals can delete or amend content before the time at which the disposition rule is applied.

• automated filing by a rules engine  - the time spent to write the rules means this approach does not scale to a whole organisation

• automated filing by an auto-classification tool -  a typical organisation carries out a great many different types of activity.  For each type of activity it carries out a great many different instances of that activity.  An auto-classification tool  has to be trained in each container/category it is asked to file records into.   The more containers/categories you have the higher the cost of training.  Organisations trying to implement auto-classification corporate wide have had to compromise and ask the tool to file into 'big buckets' rather than into a container for each instance of each activity (project/case/matter/ relationship etc.).  This means that on a corporate scale auto-classification does not currently meet the criteria of linking records to the business activities that they arose from.  I explained above that although retention rules may be set on broad swathes of activites (e.g records of all our projects are kept for ten years after the project ends) to apply that rule you need to have each record allocated to a particular project so that the ten years can be triggered by the end date of that particular project.   The buckets also end up being too big to be navigable or useable by end users, meaning that the system ends up not being used as the main source of reference for the activities it relates to.

Conclusion

At this point in time we need to be honest and say that we have no approach to implementing systems on a corporate scale that will routinely and comprehensively capture business correspondence , protect it, link it to the business activity it arose from , maintain access to it over time, and  serve as the main source of reference for the activities it covers.  Nor have we the prospect of such an approach any time soon.

Until we get such an approach organisations will suffer problems with their records management/information governance.

We have seen organisations establish electronic records management systems/SharePoint implementations and hope that access to information requests/ e-discovery can be confined to those systems, only to find themselves searching e-mail accounts, shared drives and maybe even back up tapes.  This is because their official records system does not routinely and comprehensively capturing records.  Their external stakeholders have responed in effect by treating all the organisation's applications as being part of their records system.

We need approaches to records management that result in systems that are both reliable and relied upon.   If end users do not rely upon a records system,  but instead refer mainly to content outside the scope of the record system, then they will neglect to point out omissions in the record system, and there will be a disconnect between the records available to the individuals carrying out the work, and the records available to those wishing to scrutinise or continue their work.