46 responses to “The Folder-Less SharePoint Paradigm”

  1. Great outline of the metadata for files.

    We are at a transition within our company to take our existing files off our our local domain drives(Drives and folder heirarchy which does not make any sense to anyone). We have been given SharePoint repository’s for specific product subteams.

    When over-hearing the “plan” the director was going to copy the folders over to sharepoint.…JOY!

    I totally understand the point behind the basics of sharepoint and file properties to manage content retrieval. What I am trying to understand is how to preapre our product development teams for putting data into the library and having them leave the appropriate trail of attachment to the related documents without having to make too much of a chore of it.

    Example…We receive a product complaint(web-portal workflow manager for compliance with our business policy). We then take the information and get photo’s, descriptions from related sales and product engineers etc…and then the folders begin to be created. One for photo’s one for an anlysis of a product freeze effect on current inventory and mitigation to get the production up and running again, RAIL document, etc.…

    We would have a project property such as ComplaintNumber assigned to the document…Good start. Potentially the phase of the investigation (DMAIIC possibly), but the discipline needed rom people is just nonexistent.

    One tool we use that is very helpful is a requirement management system that is also for compliance to business process. We understand that because we have tracelinks and connections from one document object to another. Is this kind of linking etc supported in some way by sharepoint?

    Another example could be a component I need to make will require three quotes from three suppliers and they must be archived for retrieval as well as data around price and delivery needs to be extracted and entered in for calculation into lead time and product cost. Again, attaching it to the process of “Go out and get three quotes for component (1…N)” Is there some way to effectively attach (and store) related documents to say a CAD file?

    I have been trying to get this going within our IT department, but they are a little too understaffed to get it going. They give me blank stares and “why would you do that” kind of looks.

    Sorry for the rant…Any suggestions to a mechanical engineer who knows just enough about applciations to really muck it up quickly?

    Regards,
    Dave

  2. @Dave,

    I’ve a feeling you’re still thinking in folders.. hope I can help you a bit.

    but the discipline needed from people is just nonexistent.”

    First thing that comes to mind is: How is this discipline managed today?

    In SharePoint you have the possibility to make columns required. That way you can “force” product development team to fill in the required information. If it is not filled in the document can’t be added to the library/list.

    On with these columns ’cause that is what it’s all about with SharePoint… As you state you can add a property such as ComplaintNumber to a document. So why not add some more properties like document category (photo, analysis, complaint, email, etc).

    The linking from the req. management system you are revering to can achieved with grouping in a library/list, two levels deep. You can group for instance on Complaint Number and will find all documents grouped, that belong to one number.

    Requiring quotes from suppliers can be done via (Nintex) workflow, send out emails and store their replies , related documents AND the CAD file in on library or list, grouped by their common divider.

    The lesson here is, don’t use the CAD file as a starting point, use it’s meta data to structure your content.

    In this case I think you will need the library and not a custom list since attachments in lists won’t be searchable.

    HTH,
    Mike

  3. 1) More items may be stored in lists when folders are used.

    2) They are more intuitive to use for simple business users.

    3) They can be manipulated VIA Explorer view not some less than spectacular Microsoft web interface.

    4) They may be controlled using Event Receivers.

    This so called “discipline” is brought about by crappy application design: EX// SharePoint … Folders could work perfectly if Microsoft made them work … infact, I’ve built a massive folder based list with meta data stored in a different list … My lookups are much faster because I only traverse files in one folder and have not a need to search them (image data) .. plus I can save as many images as I want in one list without any hastle.

    Grouping makes the UI slow and ruins the user experience because column data is now hierarchical.

  4. Stumbled across your site while researching a problem I am having with … guess what? Folders and views.

    One point, just to follow on @Matt’s post bullet #1, is that we were told by Microsoft that one way to address the impact of a large number of list items was to organize them into folders, which we are in the process of doing now.

    I would assume that was the reason for Matt’s first point, but I would certainly be interested in your comments on this tactic. Thanks!

  5. I have also found your blog while trying to develop a simple document storage solution. A tree of folders seems to be the only valid approach.
    We want users to be able to checkin and checkout multiple documents associated with a single project. We’ve created a project list with appropriate columns: customer, description, date, etc. When a user creates a new document, we want them to select the appropriate project and upload it.
    But because SP lists are not relational, there’s no way to link the fields! A user could upload a document, selecting one customer, then selecting a description for *another* project. I’d end up with a row in the library, with the appropriate document, but mismatched data in the customer and description columns.

    I know there are ways to code around this, but a custom application is not an option.

  6. I found an interesting way around this which allows a little more content for lookup purposes.

    I use it for a part number and description field combination all of the time. I have a list that has the part number and description and a lot of the specifics for a part. I create a formula(calculation field) using the typical excel alpha combination scheme using “&” symbol. To create a “PartNum_Name” field. I then use this combined field as a lookup within the other lists or doc.library you may want to reference.

    This is especially helpful for issues, taasks, etc, so people know where the problem is attached.

    IS this what you were trying to address?
    –Dave

    I found this useful, since some people know something by name, some know it by number, etc.

    Its the composite fielddifferent lists.

  7. Regarding Folders.…I found an interesting way to break people of this habit by inserting the external lookup field lists, as well as showing how the documents could look via the document library views that disable the folders, but group or filter the web part vies.

    Let them make it in folders, but you also have to force them to assign the meta data to the document by requiring a field for example.

    You can get what you want, you have to meet them half way. What really helps is when you show them how useful the field properties are.

    Also to avoid confusion, have the property fields you wish to track as a separate lists that is hidden or leave it exposed if it is something usful.

    –Dave

  8. Folders will never ever go away. The sooner we all except this and stop the folder vs meta-data debate, the better.

    In my opinion we as an IT professional community has failed to not recognise the potential merger of both. It is not a one or the other debate. Current attempts at making them co-exsist range from desperate to hopefull.

    We seem to think users do not understand meta-data and then go on about their dicipline (or lack of).
    I have also done this untill I realised that users love meta-data, they just have difficulty agreeing on it. The legacy of folders…

    …User created folders imply meta-data. The user does not realise it, but he/she is is creating implicit meta data for the documents. The typical file-based collaboration problem then becomes to create alignment on the folder structure. Point is… users have been fighting over meta-data for as long as file servers have been available to them. Pre-dating formal document management etc. They just called it something else.

    Tools such as Sharepoint has largely failed to understand the huge resistance to change they will have to overcome when introdicung meta-data driven systems.

    Here is my attempt at a solution:

    If e.g. Sharepoint could present its meta-data as folders the problem will go away. Lets introduce a conceptuall view called: Folder View (as opposed to Grid View, Normal View, etc.)

    The setup for this view would be essentially ordering the metadata in a specific order rendering a view which can be presented as folders. Done deal, everone is happy. Ofcourse they would not be able to create ad-hoc folders anymore, but they will not care if all the meta-data is there. Adding meta-data is easy and the problem is well solved in Sharepoint.

    The following advantages now become evident:
    1) Meta Data (real) is seemlessly used to facilitate traditional click through searching. Your user is completely unaware of the difference.
    2) When a user creates a file in a specific folder, all the meta-data for the new document is implied by the parent structure of the folder. This meta-data could automatically be attached to the document, thereby removing the need to populate it. Only the difference will be presented for capture. This is a good thing because meta-data can become very laborius to capture.
    3) Obviously, multiple folder views can be created, thereby bringing home the advantage of meta-data to the user. Yah!

    The current sharepoint (up to 2007, since I do not know the 2010 spec) comes kinda close with the option of a Group by view. But no cigar and here is why (in order of severity):
    1) For reasons beyond my understanding you may only have two levels. Epic Fail.
    2) It does not look like folders so users feel uncomfortable. Fail.
    3) If you add a documents the advantage as per 2 above is obviously not present. Minor Fail.

    All my searches on this topic leads me to threads similar to the one, which is essentially a folder vs meta-data only standoff.

    The reality is that the folder paradaigm can never be removed. It is well and truly entrenched. It can however be hijacked by meta-data (as described above).

    I would be interested to here this groups thoughts on this?

    regards,

    CHM

  9. I’m confused on your statement about security. What do you do if you have content within a list that requires varying security levels? Do you implement item level security? That would force you to set the security on every document every time one is uploaded.

    We use folders so that through inheritance, users can upload as many documents as they want to a folder with a given security scheme without having to modify the security on every document they upload.

    You could write a receiver to set permissions on things as they are uploaded, but have fun with that as users invent new lists with new permission schemes that force you to constantly update the code.

    Performance wise folders are better as well. Imagine a library with 10,000 documents contained in 1,000 folders. Whats better for performance, 1000 folder permission sets or 10,000 item level permission sets?

    I have implemented a lot of document library lists at my company and while I agree with your basic premise that excluding the folders makes the list cleaner and easier to work with, my experience is that this is only true for small focused lists with a limited audience. As soon as you start adding lots of different types of documents and start including a larger audience, then you have to use folders.

  10. I have to say I find the discussion regarding Folders vs. Metadata always interesting. Keeping it brief, I must say I am in complete agreement with Chris.

    I base this on the fact that there seems to be 2 big issues that users deem folders are necessary for: Security and Large list issues. Now we all may not agree on usage and settings and the little eccentricities of SharePoint I have to be clear about Security and Large Lists.

    You can have an extremely large number of files/items in a library or list. The most I have personally seen is in the range of 157,000 items (Library). The default view presents 50 of the most recent documents only. The results populate across the WAN half a world away in about .34 seconds. All the documents may be retrieved based on Folder Names (not real folders but what their team decided to call their metadata column), dates, authors, and up to 6 other metadata items. Their longest list returns 4,500 items (only 3 exceed the recommended 2,000 item limit) in just under 4 seconds across WAN, half a world away. I think that mystery is solved.

    Finally — security. Why an organization would find it necessary to manage the security of 10,000 separate items is beyond me. I guess it happens though. The same goes for folders. Why would yo want to manage information on that level? Document management, collaboration, processes, and Information management are but a few issues that come into play with SharePoint. IF a company has developed some sort of folder structuring system where many of their documents are managed and secured at folders levels then they have not been led down the right path by their consultants. OK, I will admit, I have used folders, but for very limited reasons, and never for security purposes. It has been mostly for archival purposes. How I overcome the security problem is through information architecture. What are the items, who uses the items, how often are they used, when are they used, how often are they reviewed, appended, changed, and finally when is it considered archivable.

    If an organization plans their document structure company wide, I will guarantee you that the structure will not lend itself readily to folders, but to Sites and Libraries structured around security, usage, need and the type of information. As an example, I finally had a department of a particular client come to me after using SharePoint for 2 years with their information stored in several libraries under several sites and THOUSANDS of folders. In some cases the structure was 10 levels deep. At the time it was what they wanted and there was no arguing them out of it. When they came to me they begged for restructuring. Security was bombing out because they couldn’t keep ahead of the changes. People were storing documents anywhere they thought was right. They had 5 levels of security but based on their design you would hav ethought there were 500. When we finished, they had 7 Sites, each secured at the site level only as libraries inhereted from the site. 2 of the Sites were Portals where other users could accesss cross department information. No more than 7 security groups used to manage security and at the site level only. No woperational for 6 months with no permisisons issues reported and not 1 single folder in use. So here is to metadata, long live metadata.

  11. Many of us are still thinking in terms of folders. We have to move forward.
    Folder concept is inferior to metadata concept because with conventional folders you can have only ONE view, whereas with metadata you can have multiple views to sort your files.

    Another problem you will face if you start creating multiple levels of folders, is that the URL length will become too long.

  12. @Greg — fantastic!

    @Nate — I’m Shocked!

    SP 2010 is point and case, to be quite frank it should put this debate to rest.

    But let’s ignore that for a moment: in 2007, a lot of the issues people experiece *without folders* are due to bad design rather than *having folders* being better! If you give users a folder structure 27 levels deep with nothing findable — they would not know better — thats what they’ve always had — so when you deliver a ‘folder solution’ in many ways you can’t fail. It takes good design and planning to make a new ‘paradigm’ work.

    1. Enforce the population of metadata — they’ll get use to it;
    2. Filter items on list at entry point
    3. No one needs a single point of access 10’s of thousands of documents — split up libraries
    4. If after filtering you return more than say 5000 items (nice round number) (excluding an archive which is accessed less often) you may need to re-think your filters

    Job done

  13. OK, I’m going to get slain here, no doubt. I’m not the techy that the people are in this discussion. I’m the person who runs the company and relies on the IT people to provide sound advice for a proper solution. I’m computer literate, so I constantly suffer from not knowing how to keep away from being cut by the “bleeding” edge vs. the cutting edge of technology.

    I understand folders very well, and we have designed a folder system that has worked very well over the years with automated security because we are very disciplined; however, the effort to enact that discipline is, well, hard to do. If there is a better way, then I’m all ears. Unfortunately, I have a hard time getting my arms around this issue. It’s not that I don’t see the value of meta data and new thinking. What I mean to say is that the ideology behind meta data is sensical and causes me pause. The difficulty is in the “goodness” we have achieved in our file structure. It works. It makes sense. It provides discipline of information flow. It is sound in practice.

    Here is where my problem is and why I am joining this discussion. I can’t stand what I’ll call the Microsoft person. This goes hand in hand that I love the Google person. Don’t get me wrong. I define the Microsoft person as a techy software engineer type. The Google person is represented by a blank white screen with a data entry box in the middle. Many billions of dollars later you see the Microsoft person telling the Google person how stupid they are, and how there solution doesn’t have all of the power, and how blah, blah, blah. I’m reminded here of the Apple vs. PC ads on TV. The Microsoft person is probably right, but who really cares.

    If I have to go through HELL to get to easy, then I think I’ll stay stupid. In the very least I’ll wait until the Microsoft person creates a really easy point and click thingy that gives me what I want without having to get education like I am now. Consultants, engineers, etc. just don’t get that business owners want solutions that don’t require them to give the keys to their business into the pocket of the IT guy. The solution is only as good as the design created by the IT guy. If there is a good IT guy out there, then, well, you can see what a bad situation that is.

    Why do we have to have this debate? Why can’t the Microsoft guy create a tool that forces good design regardless of meta data or folders. I’m looking at coming back to Microsoft (Office 365), and here I am having to learn computer again. Over at Google everything happens. I’m not sure how, but I get productivity quickly. Google docs just doesn’t cut it, so here we are at Microsofts door. The door opens, and there is that stubborn arse forcing me to be smart. Turns out I like being stupid.

    Let the attacks begin,

    Scott

  14. Chris,

    You hit the nail on the head. This arguement is, I think, solved for you if you can show what you mean. For me I will then learn and understand more appropriately what people mean behind the rhetoric. I’m a show me kind of guy. The key would be to having the show encompass the opposing sides concerns and responses; otherwise, the “show” would appear to be a spin.

    How exciting that my comfortable solution would be as caveman-like a comparison of opening a can of beans with a steak knife. I’m more than willing to experiment, but I’m very, very tired of having the engineering sort talk out of their other mouth and not listening to the realistic concerns of the end users. Forcing me/us to rely on a good designer isn’t and never will be the answer because you simply can’t trust the designer to be good or understand your business.

    Let the video begin,

    Scott

  15. […] and folders in Sharepoint 2010  It is instigated by the discussion in his blog about the Folder-less Sharepoint Paradigm. This discussion is very enlightening because of the divers views of participants of using […]

  16. just wanted to say, this is a great post, and a great conversation. Lots of valid points throughout.

  17. […] SharePoint users and consultants, and I think the folder debate is probably the most common. Some, like myself, have gravitated in an extreme manner away from folders, and others have tried to tow the line […]

  18. I’ve to agree with the post of Carl-Hein Mostert, in short
    folder structure = meta data
    (OK, meta data might do more than the strict tree structure of folders, but since (hd) folders can be links its almost the same)
    Which automatically leads to the fault of the original post:
    People that did not care about a file structure on the HD will also not have the discipline to fill in the META data.
    But required field…
    Just that a field is required does not mean, that it will be filled with meaningfull data.
    And not all sharepoint owners do have admin rights and are allowed to change required fields.
    It’s one thing which is the academical best solution, but often another thing what the user wants or is even able to understand. And users are already frightend by the word MetaData, because it sounds already quite abstract.
    While I’m using categories for my private blogs (the articles are natively in a flat structure, no matter if Joomla or WP) I’m forced to use folders for sharepoint. pragmatically.

    I you write the metadata yourself and especiall if you are admin, it’s a fine thing, I guess.
    I would probably spend 30% of my working time, training the sharepoint members, if I would not used folders.

  19. Interesting to see that you’re still keeping up with visitors talking about the folderless paradigm.

    I’m in the position where I recognize the benefits, but sometimes it just cannot or should not be done.

    This isn’t to say that there’s a lack of discipline on the part of the users, or that I don’t have the will to make it work for them. I know what the benefits are, and it’s just that metadata doesn’t fix everything.

    I’ve commonly seen project documentation organized in the fashion where at the root level, you have the phases of the project, and then varying kinds of documentation underneath, and subfolders under those subfolders, etc. The fact is, you simply can’t cast this as a problem that metadata fixes. You can certainly do it at the root level, indicate that documents fall into certain project phase categories, but what happens after that? You have metadata that only applies to the sublevels of particular folders, and doesn’t make sense within the context of the others. For example, why would I need requirements documents to reside in the release instructions folder? Why would I need dataflow diagrams to be in the QA? You end up with a design where people are directed to skip the metadata where it doesn’t apply, thus leading to them generally ignoring it on the whole, and you waste a ton of time, energy, DB space/performance, and resources to put a square peg into a round hole.

    And to what end? Yes yes, you can make views that filter on the metadata. But now your views list is a mile long, security management is a nightmare, and you’ve not actually helped anyone do anything better.

    People understand the folder motif because it organizes things. They might be made to learn other paradigms, but it’s not part of their workflow, and does the training and effort result in a good ROI? In my experience, it really does not.

    Rather than flattening everything out, I believe the superior way to handle this would have been to allow Document Sets to nest. Then you can maintain your organizational hierarchy without having your hands tied by it. Too bad Microsoft missed the boat on that one.

  20. I think the same direction like Drew (except this nesting thing).
    If you have a folder structure the owner predefines a fixed structure. A member can follow this structure and see, which subfolders exist and where a dead end is. The member needs initiaaly zero knowledge of the structure.
    If you convert the folder structure to metadata, the member does not only need to know which metadata is valid (i don’t know if there is a possibilitey to get a drop down box of existing metadata for filtering), but especially the member does not know which combination of Metadata is valid.
    So I have to refine my agreement with Carl-Hein Mostert: Yes you can convert a folder structure in Metatags (like tag Project Phase instead of folders Project Phase xx-zz in Drews example) but for the user it’s not really equally.
    Pragmatically I would say it is good to use Metadata if you have a flat structure (like a blog with articles of different categories), but if you have a system where you have already an underlying tree structure, often the tree structure itself is an important part of the information. Then it’s probably better to go with the folder structure.
    This might change if MS improves the MetaTag filtering/navigation.

  21. I’m tech-savvy, but have limited experience with SharePoint, and I’ve read the benefits of the new 2010 version, but am still not sure about a few things. Our organization is looking to migrate our information from the older version, which means we’ll have all documents in one library– no folders. It’s great, and it forces you to shorten file names and make the Titles the lengthy part (this may seem trivial, but for everyone to do this, it’s a major change). We have created a document library to house Timesheets… and we are creating/designing a Site. Some of my colleagues have gone into the library to add folders for each staff person’s name, and transferred over the corresponding documents into them. Others felt that it would be better to have a different document library on this site for each staff. I’m not sure about which one is the best approach. Any suggestions? We need the documents to be checked in and out, without being able to see everyone else’s timesheet, so permissions is an issue as well. Oh, and keep in mind that we do not have access to SharePoint Design.

  22. Dear Chris and all

    I have an enquiry that is related to but not directly regarding the above discussion but there seems to be lots of expertise here so hopefully you can help me.

    I am not a techy person but I am computer literate and have had some SharePoint training and been charged with the task of constructing a SP site for my company to replace the old, bloated one which got out of hand quickly due to the excessive use of folders. One thing I am not familiar with is SP designer so I am trying to build the site just using the functions available through full authority with the Site Actions.

    My issue is mentioned briefly above by Amir Kashmiri and it is one of navigation/organisation of the site without using excessive layers of folders. Basically the site I want to produce will necessarily have many layers of organisation due to it’s size and scope. There are a number of sub sites at the top end and document libraries at the bottom (some of which will contain folders but they should only ever be one layer deep.) However on most of the sub sites users will need to pass through 3 layers of navigation before they reach the document library holding what they want or providing the correct space for an upload. What function does SharePoint provide to produce these layers without uising folders?

    I am sure this must be very obvious to an expert but I am struggling. I already have enough sub sites and they are not suitable for the purpose, you cannot populate lists/libraries with other lists/libraries therefore they are no use until the actual storage stage and I have not found an effective way to do it with pages that would protect the end user and site from unwanted alterations. This has left me at a bit of a blank.

    The whole point of this redesign was to make navigation to the correct document/storage space very simple (without layer after layer of folder) but I cannot achieve this without having the freedom to add as many layers as is necessary and the only way I have found to do so is folders but that comes with all the above problems attached (and more..!)

    I hope this question is not out of place in this thread and any help would be greatly appreciated. Thanks for taking time even to read. Best Wishes

    H

  23. I really agree with the whole idea of driving views etc by metadata.

    The real gotcha for me in my organisation is libraries with a large number of documents (30,000+) and the use of Explorer view and the performance problems that view causes.

    The Explorer view reminds me of the quote ‘Just when I try to get out, they pull me back in’.

    The Explorer view seems to be the only way (out of the box) to change the file extension of a file. Similarly when you create a document from a template, and click Save it shows the Explorer view and all documents in that library with the same file extension.

    **sigh**

  24. Both approaches have their problems and benefits. However, it seems that metadata approach leans on benefits side more than the folder approach. But I am not convinced that the metadata approach is as intuitive as folders even though it’s more flexible. Ofcourse one can learn a new paradigm and train, but I still don’t see it being as intuitive as folders. Folders put users in an appropriate hierarchical context, which makes it easier for us to understand where the user is. The problem with nesting folders is that it’s not easy to see an overview and find the right one. Right now a folder represents only a single possible hierarchy for a document, however a document could belong to two logical branches. That’s where I would like the flexibility of metadata. But right now metadata approach in SharePoint does not present an intuitive way of navigating and defining right hierarchical contexts for a document. I think if we come to a marriage between intuitiveness of folders and flexibility of metadata we will have a much better solution. I like the wordpress tagging feature where I can define a hierarchy and also assign multiple tags to a single post. And the flexibility can come from the ability of metadata to be presented hierarchically as well flat.

  25. This might have been said already, but there are cases where you want to email a shortcut to a set of documents to someone. This can be done on a folder, not on a view. It all depends on how users are using the data and how they need to access it (and what kind of content is in a library).

Leave a Reply