Blogs

Are folders still necessary?

By Lisa Ricciuti posted 10-25-2013 14:58

Recommend

Folders and SharePoint 2013

Last month I attended a crash course on SharePoint basics. As we were setting up our library*, the instructor informed us that creating folders was not advantageous as they could interfere with the effectiveness of metadata assigned to the documents.

Initially I accepted this as a customization one has to make when setting up the SharePoint platform. As I thought more about this idea, I started to wonder if we still need folders now that so many options exist with metadata. Are folders the best option for organizing electronic information?

A folder is an intellectual construct that allows us to group like things together, or an aggregate of related things, so that we can find them easily later. This system worked in the physical world where few other options existed as finding aids to locate items. Creating systems to find and locate items could also be extremely time consuming in the physical world. However, in the digital world where we have more of everything to contend with, folders seem antiquated.

Metadata is King

I’ve heard this expression a number of times. Metadata is King, but actually I think it only deserves this exalted status when it is done properly. Things like hashtags often exemplify what I consider to be poorly executed metadata. When metadata is properly and consistently defined, it can be extremely effective for both searching and browsing depending on how it is set up and used.

When I worked with Content Server, folders were seen as advantageous because they could be prepopulated with metadata. Any document saved in the folder would automatically inherit the metadata. In a perfect world, because that’s the best place to do records management, the folders would be set up and prepopulated with about 50 - 80% of the metadata. Then the user only has to save the document to the right folder and fill in a minimal amount of information. Obvious advantages are that:

The document is described properly and can be searched for.
The user has virtually nothing to fill out.
And it seems to be kind of a win-win.

Seems to be…except for this idea with SharePoint that folders hinder the effects of the metadata. And with the Content Server method, the potential for documents to get buried in layers of folders exists. How can folders and metadata function in a way that is advantageous to us as users and information professionals?

Browsing vs. Searching

I am a browser. This means that I prefer to find content by clicking through hierarchical levels to find what I need. Searching means that people find content by running queries on keywords or metadata elements. Then the user must scroll through the list of results to find what s/he needs.

I’m so accustomed to browsing through folders that initially my brain couldn’t figure out how to browse without folders even though I experience it all the time. Folders are a convention that came from the paper world. This method was maintained in the electronic environment as a way to group things together, the same as it was supposed to be doing in the paper world, but folders can be problematic for some of the following reasons:

They lengthen the document name causing problems later.
Documents get buried in layers and layers of folders making them difficult to find.
Users stop putting things there because they can’t be bothered with all the clicking.
Security and access can be difficult to monitor and maintain.

So why do we still use them? If I could figure out a way to browse through all my documents with metadata on my laptop, I wouldn’t bother creating folders at all. The folder name could easily be a piece of metadata, which hopefully could be turned into a browseable facet.

If you’re still working in a shared network environment I can see the rationale for folders because options are somewhat limited. But if you’re in an ECM (or EDRMS or EIM system) I think properly defined metadata with browseable facets and established search queries could be just as effective while obviating the need to maintain an extra layer of stuff (i.e. folders) in the system.

Users can browse with facets. This is already a popular feature on a number of websites, like libraries, or as part of an app like iTunes. I browse through my music completely by metadata elements attached to the albums such as artist and genre.

I know in the end setting up folders and metadata is entirely dependent on the platform being used. And I can already hear my readers thinking, “users would never fill out that many metadata fields.” I say, let the users figure it out. A lot of people are happy to slap hashtags on posts and pictures. They would quickly see the benefit of filling in metadata in a professional capacity when they can’t find their stuff.

It’s a good exercise to consider the unique advantages the digital age offers us in organizing and managing information more effectively. Maybe if we started thinking about how to organize electronic information based on its properties, we could get away from replicating systems used in the paper world because clearly better options now exist.

*SharePoint name for the area containing documents.

15 comments

1127 views

Comments

Howard Frear

11-19-2013 08:55

I'm firmly behind the idea that you should have the best of both - speed and accuracy of the metadata with modern search technology and a virtual presentation of folders based on this metadata. At EASY we've had the challenge of just search in the past but I believe and have seen evidence of users being much more comfortable with this over the last few years.
Just recently we were asked to come up with a solution to indicate missing and incorrectly indexed documents in a folder structure. We did this with a simple traffic light mechanism, however only after pointing out that presenting duplicate file systems for projects from their predecessors caused the empty folders problem in the first place. Also users presented with no simple (and mandatory where necessary) metadata choices together with a straightforward submission process for what were highly controlled documents, were bound to get into trouble!
I guess the debate will roll on.

Andrew Wilson

11-08-2013 05:25

Marc - apologies.
I can agree with all the responses here and also think that the answer is as using a mixture of the capabilities provided.
Whilst I don't think folders are bad, I wouldn't condone uncontrolled end user creation of them and wouldn't want to see more than 2 levels at most.
This isn't just about not using folders though, it is about what the other features can bring. The way that metadata in SharePoint allows personalised views of the same set of documents for example and the automation that content types can provide.We need to use all of these tools to aid findabilty and data quality. SharePoint does have its limitations and as with every technical solution you have compromises to make.

Joao Assuncao

11-07-2013 05:26

Great discussion :)
I think you need everything and it depends when you need one thing more than other. You need metadata because is the best to have content with great findability but you need folders to have security. Do you know you can't have permissions attached to metadata? One of the things I really miss is to have some permission model assigned to some metadata I put in my content. Let's say I want that every document with metadata value "AIIM" is private only to me and the other documents are public. The only way to do that is to have folders or else you have to put permissions in every document - and that is a mess.
You have the metadata navigation - a really good feature. You have folder navigation. You have filtering. Have content types and so one. I think you need it all.
And we need metadata assigned permission in SharePoint.

Marc Fresko

11-06-2013 16:01

Andrew,
The mitigation is slight. In a large system you will have tens or hundreds of thousands of folders. Would you support that many content types? No, I didn't think you would.
The general tenor of the anti-folder arguments above seems to be that the important thing is to do things in a way that complies with SharePoint's limitations rather than in a way that complies with business requirements. How strange.
Marc (not Mark, please)

Andrew Wilson

11-06-2013 03:57

@Mark.
In SharePoint the use of content types mitigates some of the issues you talk about. Content types can be used to define and enforce information management policies and content routing can help with restructuring and document filing. I agree with your point regarding access though, this is an area where folders are still useful.
@All.
My approach has been to restrict the end user from creating their own folders but not banning them altogether. This has helped embed the mindset of using metadata instead and the benefits it can bring. Once users see it they generally get it.
From personal experience there are occasions when you have no alternative but to use folders, due to the technical limitiations of the the list view threshold (set at 5000 by default), in order to work with a library of 50k document in the way we wanted we had to organise content into folders even though they aren't displayed by default.

Marc Fresko

11-05-2013 23:51

No, no, no.
Or, to put it more diplomatically, Yes - everything you write is correct, Lisa. But the argument you present is missing an enormous point.
Yes, metadata alone supports searching (subject of course to the comments from some users noted in some of the earlier comments). And yes SharePoint, can have "issues" as you note. But there is more to it.
There is only one important word in the phrase "Enterprise Content Management", and that is "Management" (I write this because "Enterprise" is frequently misleading and misunderstood in this context, and "Content" is just unnecessary jargon that means "information"... but I digress). Note that ECM is not "Enterprise Search" (nor is it "Enterprise Content Search"... oh dear, I'm digressing again).
The "Management" aspect of ECM includes managing things like access permissions, retention, etc. Metadata really does not support that, "folders" do. Imagine managing access or retention object by object in a large collection using metadata alone: it is tricky at best, impossible in a very large repository, and worse than impossible over long periods of time. You really need to "aggregate and conquer" (I'm trademarking that phrase) to stand a chance of succeeding.
You could say that electronic folders don't really "exist". At a very low level, classifying an object to a folder means assigning the folder ID to the object as a metadata value, so in that sense we are agreeing rather than disagreeing. But what folders bring in addition are some conventionally-expected behaviours, notably binding the objects they contain together, allowing wholesale management of their contents, and a few others. We can lump these behaviours together and call them "inheritance" if we choose to. The important thing though is that they make management practical for large collections over long periods of time.
To put it (yet) another way, if you configure folders properly, they have the effect of ensuring that every time you open a folder, move it, change its permissions, delete it, search it, place it on hold - anything really - you will affect exactly the same set of records. Whereas no matter how you configure metadata I can almost guarantee the opposite - over time, if you rely on metadata searches to move, change permissions, manage retention and so on you will get different and unpredictable results - and this for several reasons, which would be the subject of a separate post entirely. And I won't even start to think about performance implications.
In some situations I daresay metadata without folders will do the job - smaller setups, shorter retention periods, information for which good retention practices don't matter, perhaps. And it better had, because that is the direction of so many implementations. But please don't promote the idea that metadata without folders is an acceptable mechanism for all cases - I just don't believe it is.

Richard Porter-Roth

11-05-2013 18:37

Lisa, really good post but could you provide and example mapping for a folder and documents and their associated metadata? I like Tara's example, and I actually user Amazon, but would your metadata provide "build your own folder" structure like Amazon does? Also, not everyone uses SharePoint, what other systems allow you to do this?

Gene Edmiston

11-05-2013 15:42

@Russ Herald, SP2010 has the ability to create groups on lists/libraries as well as define filters and views to present the items in unlimited fashions. Also, as @Sebastiaan Mindreau mentioned, with the term store, you get the left navigation with the search features.
@Luke Vansanten, there are instances where having a thin folder hierarchy is preferred over metadata columns. The key here is to keep the folder levels to a minimum.
@All, there is a perception that having folders in document libraries in SharePoint are disadvantageous. Many of the SharePoint gurus have stated that folders don't have a negative impact. I tend to agree with them as the search engine really has no problems with it from an architecture perspective. Although, you won't be able to take advantage of some of the SharePoint search features like term store filtering, in list/library pulldown searches, etc.

Luke Vansanten

11-05-2013 11:47

While I agree with the article and most of the comments, I was wondering - has anyone ever seen or heard of someone saying the opposite? Specifically, that folders are better than metadata?
If so, please share link(s)?

Nikolaus Maack

11-05-2013 09:56

Our EDRMS does not have folders. It's strictly a metadata game. Users find this no-folder-world utterly baffling. I've tried to come up with metaphors to explain it, and they often fall flat:
"Picture a cluster of cows, and you can brand them with different marks -- circles, triangles, etc. Some cows may have more than one brand. You have a machine, and you can tell it, show me all the circle cows, and it pulls them all out."
Okay, that one might need some work.
Part of my difficulty is that clients still have access to shared drives. So they can be non-complient, and use their familiar folder-based shared drives. Alternatively, they can march into the future and a new system (that they are already a little frightened by) and try to wrap their heads around a world without folders.
Of particular concern to me are those wonderful days when the EDRMS suffers glitches. Access rules change with a re-org as we lag behind doing a data fix. The search function breaks due to a technical issue.
Suddenly the client searches for content, and nothing comes up. This causes user panic:
"Is the error with me, or is the error with the system? Did I check that item in? Did I use the proper metadata? Am I an idiot?"
They search and search and search and find nothing and become frustrated.
When they dicover the error is with the system, not themselves, you can be certain a little voice in their head says, "I never have this problem with my shared drive folders."
And it's very difficult to argue with that.
(Phew. I'm done venting. Thanks.)

Andrew Wilson

11-05-2013 07:15

Russ
Have you tried using the 'group by' option in the view? This allows you to essentially create 2 layers of virtual 'folders' by grouping the documents using the metadata set. The real power here is that user a can have a different view to user b which is just not possible with folders. Meta data navigation is also available in 2010.

Tara Dragon

11-01-2013 13:39

One of the "light bulb" moments that I frequently see my clients have is when we talk about metadata in a non-document context. There are all sorts of internet examples that I pull on to show metadata in action. The Home Depot website is always one that I use - I can "Search" for a fridge. Or, I can browse to appliances, and then pick my style/manufacturer/size/color/etc. until I get what I want. I find that when people see that sites they use everyday are examples of metadata in action, it makes the jump from folder to folder-less less daunting for documents.

Russ Herald

11-01-2013 05:52

Sebastiaan: Thanks for the heads up on SharePoint 2013. I'm still hoping to find a way to integrate this into 2010, though, as we'll be busy with that version for the coming time.

Sebastiaan Mindreau

10-31-2013 08:25

Hi Russ,
In SharePoint 2013 you have the option of metadata-based navigation, which gives you a left filter-pane with the term store hierarchy to search your documents/content.
Best regards,
Sebastiaan Mindreau

Russ Herald

10-31-2013 03:48

One of the difficulties I've seen, is that SharePoint (2010, at least) natively doesn't really offer a friendly way to filter via facets. In a document library I get a kind of spreadsheet view, in which I can filter and sort the columns. But this is confusing to many users, who think (as you mention) "folder" as soon as documents are involved. Presenting the various facets, say, on the left would be a help already, and would let users see the benefits of filtering based on multiple values and perspectives.
It is important to as how users want to find information. Some want to have everything presented orderly before them (folders of catgeories), some want to see everything and browse, some want to start by searching. I think a hybrid design that lets the user choose from various options might work, but this can also get confusing for some users.