Enterprise Content Management - Document Retention

Document retention is an important part of content management.  SharePoint provides out-of-the-box support for managing document retention using the built-in expiration policy feature of the information management policy framework.  The built-in expiration policy feature includes the ability to define expiration formulas based on document metadata, but there are many scenarios in which the expiration date depends on conditions external to a given document.  In this session, we'll explore the information management policy architecture in detail and learn how to extend the expiration policy feature by writing custom document expiration formulas that calculate the expiration date based on data pulled from elsewhere in the SharePoint farm.

Speaker: John Holliday, MVP Office SharePoint Server

There is a new standards effort in the works that promises to unify the disparate ECM platforms currently available into a common set of interfaces.  The Content Management Interoperability Services (CMIS) specification is the result of a collaboration between Microsoft, IBM, EMC, Alfresco, OpenText, SAP and Oracle.  Click here to download a preview copy of the spec.

First impressions:

The specification tries to define the "core" components of any ECM system.  On the one hand, this is good because we at least need to agree on common terms.  For example, the whole "content type" versus "object type" debate is important to get resolved.  On the other hand, with such a young paradigm, it's not yet clear what functionality will be the winning market differentiator for any given vendor, thus thwarting further attempts to refine the specification.  To that end, the group has submitted the draft specification to the OASIS consortium so that others can join the effort.  This is a good move and will hopefully lead to faster refinement.

1. The primary goal of the spec at this stage seems to be to provide a common set of SOA endpoints for the different ECM offerings from the vendors involved.  There are SOAP and REST/Atom protocol bindings for its proposed "domain model", each of which are explicitly required for conformance with the standard.
2. The spec does not cover administration, configuration or security, and includes within its definition of "administration" the modification of Object Types.  This is a bit of a disappointment, but not surprising given the different interpretations of that term.  Yet, the explicit exclusion of such a fundamental set of methods again reflects the newness of the paradigm and the difficulty of defining a common object model.
3. The data model focuses on the ability to create and access a repository, but explicitly excludes anything defined as "transient", which includes such important concepts as compound document, workflow, event and subscription.  Here again, we see what appears to be disagreement on first principles.  The result is that the data model describes only low-level components, such as object, property, document, folder, relationship and policy.
4. CMIS supports versions for document objects, and also includes a standardized query model based on a subset of SQL.  This is very nice, because it means that clients will be able to submit SQL SELECT statements to retrieve objects from multiple ECM vendors.  This will include searches based on object properties and folder membership as well as full text searches.

The spec at this point is pretty low-level, similar to the Open XML Formats packaging API, but offers a strong first step toward cooperation among the ECM vendors in providing a common set of tools for accessing disparate content repositories. 

The query specification is interesting, and includes a few sample queries that I'm anxious to dig into.  There are also some good examples of how versioning, checkin and checkout will work.  Hopefully, I'll get a chance to go into these in some detail in the coming weeks.

Stay tuned.

When it comes to content management in SharePoint, half the battle is figuring out the best angle of attack.  Why?  Well, because there are so many options and combinations of options that come into play when designing a solution.

Take the SharePoint permissions architecture, for example.  There are 33 different permission masks divided into 3 categories: personal, list and site permissions.  These are combined into role definitions (permission levels) which may be assigned to users and groups and then associated with sites, lists and list items.   Often, you need to create a custom role definition for a particular user or group.  When doing so, it is important to understand permission dependencies, because that will determine the effective permissions being granted.

With the exception of the "Open" permission, which grants the ability to open Web sites, lists and folders to access their contents, every SharePoint permission depends on one or more of the other permissions.  This means, for example, that if you grant the "Manage Alerts" site permission, you are also automatically granting the "Create Item Alerts", "View List Items", and "Open List Items" list permissions as well as the "View Site Pages" and "Open" site permissions.

If you're like me, with so much information to digest and process, and so little time to do it in, a simple diagram can go a long way towards sharpening your focus.  To that end, here is a little chart I created for my ECM401 course (all praise to the Visio gods!) that you may find useful.

Enjoy.

Let's face it, SharePoint is a huge development platform no matter what angle you approach it from.   As most of you know, I've spent the past year or so traveling and teaching SharePoint developer courses for the Ted Pattison Group, and my focus has been pretty broad, covering topics ranging from features to content types to workflow and everything in between. 

Now, as the platform continues to mature and as developers become more and more familiar with the basics of SharePoint development, it is increasingly evident that more specialized training is needed, particularly in the area of Enterprise Content Management (ECM).  To that end, I'm pleased to announce the immediate availability of two new courses I've developed that are being offered through the Ted Pattison Group:

• ECM401 - Enterprise Content Management with SharePoint Server 2007 (hands-on)
• WC-ECM401 - Enterprise Content Management with SharePoint Server 2007 (online)

Both versions of the course are available for immediate registration.  The online version is a great option for those unable to attend the hands-on course.

• The hands-on version includes 12 modules instead of 10 with a deeper treatment of content modeling and using workflow to drive ECM solutions.
• The hands-on course is 4 full days with lectures, demos and labs.  The online version is 5 days with about 3 hours of lectures and demos each day plus labs assigned as homework.
• The online version is conducted via Live Meeting and students may submit questions during the lecture which are answered at the end of each session.  Questions may also be submitted directly to the instructor via email or live chat.
• Students attending the hands-on course receive a student workbook with slides and labs.  Online students receive the workbook in electronic format.

Feel free to contact me directly with any questions you may have about either version of the course.  For additional details and to review the course outline, please visit the Ted Pattison Group website using the following links.

Here is the upcoming course schedule, with links to the registration form.

Well, we're just a couple weeks away from TechEd 2008.  This year, I'm giving four presentations - three breakout sessions and one TLC session, so it should be a lot of fun.  Here are the dates and times:

• OFC10 - TLC 6/3/2008 3:00PM - 4:15PM
  Building Custom Routers for Records Management in Microsoft Office SharePoint Server 2007

  This talk discusses developer aspects of records management in Microsoft Office SharePoint Server 2007. Custom routers are an important extensibility point for records management and this talk details the requirements for these components. It provides a demo of building a custom router, including deploying it to a SharePoint Server environment.
  

• OFC318 6/4/2008 8:30AM - 9:45AM
  XOML, XOML, Everywhere: Building Custom Workflow Editors for SharePoint
  

  XOML is the markup language behind Windows Workflow Foundation and is the same language used by Microsoft Office SharePoint Designer 2007 to attach declarative, no-code workflows to SharePoint lists and document libraries. This session takes a deep dive into Workflow Markup and explores how you can use it to generate custom workflows from any client application, including Office clients like Word and Excel. The session also shows how to attach custom workflow markup to SharePoint lists using the websvcWebPartPages Web service provided by Windows SharePoint Services.
  

• OFC309 6/4/2008 4:30PM - 5:45PM
  Building Document Management Solutions Using SharePoint Content Types
  

  Windows SharePoint Services 3.0 introduces a new concept called "Content Types" that will dramatically alter the Enterprise Content Management landscape. On the surface, Content Types make it easier to encapsulate field definitions into a nice neat package, but there is a lot more going on under the covers. In this session, we take a deep dive into the world of content types and see how to declare them using XML and how to create them programmatically. Then we use content types to create a real-world document management solution with custom policies that we can use to control every stage of the document lifecycle. At the end of this session, you will have a thorough understanding of what content types are, with a deeper appreciation of the value they bring to document management solution development.
  

• OFC404 6/5/2008 10:15AM - 11:30AM
  Turning SharePoint Data into Microsoft Office Documents: A Deep Dive into SharePoint Document Assembly Using Open XML
  

  Windows SharePoint Services 3.0 introduces a new concept called "Content Types" that will dramatically alter the Enterprise Content Management landscape. On the surface, Content Types make it easier to encapsulate field definitions into a nice neat package, but there is a lot more going on under the covers. In this session, we take a deep dive into the world of content types and see how to declare them using XML and how to create them programmatically. Then we use content types to create a real-world document management solution with custom policies that we can use to control every stage of the document lifecycle. At the end of this session, you will have a thorough understanding of what content types are, with a deeper appreciation of the value they bring to document management solution development.

See you in Orlando!

As the new year approaches... Wait! Is it 2008 yet? So much is happening - so fast! Do you ever get that feeling that you'll never catch up? I sure do. Time flies...

For the past few weeks, I've been busy writing chapters for a new book (more on that later), developing and experimenting with new toys and project ideas (lots to talk about) and getting ready for my ODC2008 sessions.

I'll be delivering three sessions this year. One in Karthik Ravindran's architecture track, one in Steve Fox's client track and one in Jerome Thiebaud's server track.

• ARC303 - Building Collaborative BPM Solutions

This session will cover best practices for leveraging the Microsoft BPM platform offerings (Windows Workflow Foundation, BizTalk, SharePoint Workflow, and Windows Communications Foundation) to integrate comprehensive Business Process Management (BPM) capabilities into Business Solutions built on the Microsoft Office Platform/System. This session will help the audience gain a broad perspective of the BPM scenarios targeted by each component and will offer prescriptive guidance on choosing the right option(s) to address common solution requirements. This session will also show how the various components can interplay and add value to Office Business Applications.

• CLI307 - Generating Enterprise Content using InfoPath 2007, SharePoint and the Microsoft SDK for Office Open XML Formats

This session will introduce the Microsoft SDK for Open XML formats and show how developers can use the SDK and InfoPath 2007 to generate Microsoft Office documents on the server from data stored in SharePoint form libraries.

• SER401 - Building Document Management Solutions using Windows SharePoint Services 3.0 Content Types

Windows SharePoint Services 3.0 introduces a new concept call "Content Types" that will dramatically alter the Enterprise Content Management (ECM) landscape. On the surface, Content Types make it easier to encapsulate metadata, but there is a lot more going on under the covers. In this session, we'll take a deep dive into the world of content types and see how to declare them using XML and how to create them programmatically. Then we'll use content types to create a real-world document management solution with custom policies that we'll use to control every stage of the document lifecycle. At the end of this session, you will have a thorough understanding of what content types are with a deeper appreciation of the vale they bring to document management solution development.

I'm grateful for the opportunity to present these sessions (thanks to Karthik, Steve and Jerome), and it looks like this is shaping up to be a great Conference. I'll be arriving early with my friend Andrew Connell if you want to just hang out. Otherwise, I'll look forward to seeing you at the Conference!

I'm always on the lookout for ways to work smarter, not harder. When developing SharePoint solutions, that means finding ways to reduce the overall "surface area" of the platform so I don't have to spend so much time on the little details. You know - the things that should have taken only a few minutes but end up taking several hours? And by the time you're done, you've forgotten the original train of thought!

Writing CAML queries is one of those things I always feel I'm spending too much time on. I guess I've been burned enough times by making simple little mistakes in the CAML - typos, or misplaced parameters - that I tend to over-examine the CAML when something goes wrong. I keep thinking "wouldn't it be great if we had a CAML compiler or some other tool that would help ensure that the CAML is correct every time?". That way, we could just write the CAML, compile it and then use it just like any other component. One less moving part => reduced surface area => greater productivity => mission accomplished. Working smarter.

I know what you're thinking. We already have some great CAML generators. What about the CAML Query Builder from U2U? What about the Stramit SharePoint 2007 CAML Viewer? With these tools, you just point them at your SharePoint site, enter a few parameters, and voila! - instant CAML that you can validate right there in the tool.

Well, those tools are great when you're working against an existing site, and the query parameters are relatively static, but there are a lot of situations where you need something more flexible - and more strongly typed.

I have to confess I've never been a big fan of cutting and pasting large blocks of generated CAML code. I'd rather define the query programmatically and add it to my arsenal of tools that I can reuse over and over. But there are a few other reasons why I think it's problematic as a general development practice.

• CAML is hard to read. When converted to a literal C# string, it's even harder. Even simple parameter substitutions are prone to errors that can be costly to track down, and the person doing the cutting and pasting is often not the same person tasked with finding them.

• CAML is not strongly typed. It's easy to supply the wrong attribute to a CAML element, thereby causing the query to fail. This problem is reduced when using a CAML generator, but creeps back in when the generated CAML is applied to a different site.

• CAML generators require that you first create a mock-up of the lists and site columns on which the query will be based. If you already have a good idea of what you want the query to do, creating a site and defining the lists and columns is distracting and just feels like a waste of time.

So what to do? After playing around with it for awhile, I've come up with a simple technique that addresses many of these problems at least for the C# developer. It also works for VB, but certain keywords (like "and" and "or") are a problem. I call this approach CAML.NET.

[ Note: I was going to call it "Dromedary", but I felt that was a bit much. A dromedary is a specially-bred Arabian racing camel - faster than a normal camel with one hump instead of two - get it? racing camel? never mind ]

The basic idea is to leverage the power and flexibility of the .NET Common Language Runtime (CLR) to build CAML queries dynamically in code while preserving the syntactic structure of the native CAML language.

For example, the following CAML query retrieves a list of items grouped by title, whose content type is "My Content Type" and whose description field is not empty. It also specifies that the result set should be ordered by the _Author, AuthoringDate and AssignedTo fields and that the AssignedTo field shall be in ascending order.

To use this in C#, you would have to convert it to a literal string, taking care to handle the embedded quotation marks, as in:

string simpleQuery = @"<Query><Where><Or><Eq><FieldRef Name=""ContentType"" /><Value Type=""Text"">My Content Type</Value></Eq><IsNotNull><FieldRef Name=""Description"" /></IsNotNull></Or></Where><GroupBy Collapse=""TRUE""><FieldRef Ascending=""FALSE"" Name=""Title"" /></GroupBy><OrderBy><FieldRef Name=""_Author"" /><FieldRef Name=""AuthoringDate"" /><FieldRef Ascending=""TRUE"" Name=""AssignedTo"" /></OrderBy></Query>";

To reuse this query for another content type, or to add more fields to the result set, or to change the ordering, etc., you would have to either generate a new query, or painstakingly decode the string by hand, possibly introducing typos or other errors along the way.

Here is the same query written in CAML.NET:

This is not only easier to read and modify, but it has a number of other advantages.

• Avoids hand-editing of literal XML strings in your code.
• Eliminates query failures caused by typos and improper casing of elements and attributes.
• Each query component is processed as a separate statement with strongly-typed parameters.
• Operator and method overloading greatly simplifies the raw CAML schema.
• Enables the use of variables instead of literal text to specify query components.
• Visual Studio intellisense support is available while writing queries.
• Simplifies the construction of reusable CAML component libraries.

Note the CAML.SortType enumeration which allows you explicitly specify the desired sorting for a given field. Other similar enhancements make it easy to construct your CAML queries. And on top of that, you get intellisense, with helpful comments that provide additional context while writing your queries.

Once you've assigned the string to a variable, you can use it in the same way you would use any other XML query string. For instance, the following code snippet applies the query to retrieve matching list items from a site collection:

SPSiteDataQuery q = new SPSiteDataQuery();
q.Query = simpleQuery;
q.ViewFields = CAML.FieldRef("Title") + CAML.FieldRef("ID");
q.RowLimit = 10;
DataTable t = SPContext.Current.Web.GetSiteData(q);

Another Example

Here is another example, taken from the ECM team blog. It lists all .bmp files in the site collection that have a height or width over 200 pixels, in descending order by file size.

Again, "stringizing" for C# you have:

string queryBitmapImages = @"<Query><Where><And><Eq><FieldRef Name=""DocIcon"" /><Value Type=""Computed"">bmp</Value></Eq><Or><Gt><FieldRef Name=""ImageWidth"" /><Value Type=""Integer"">200</Value></Gt><Gt><FieldRef Name=""ImageHeight"" /><Value Type=""Integer"">200</Value></Gt></Or></And></Where><OrderBy><FieldRef Ascending=""FALSE"" Name=""FileSizeDisplay"" /></OrderBy></Query>";

And in CAML.NET:

Note the use of the overloaded CAML.NET Value constructor, which simply takes an integer value as an argument. The generated CAML sets the "value" attribute automatically depending on the type of argument you pass in. There are lots of other places in the CAML schema where the .NET CLR can help to reduce the overall surface area and greatly improve developer productivity.

A beta version of the CAML.NET assembly is available now on CodePlex at http://www.codeplex.com/camldotnet. Please feel free to download it - give it a spin - and provide feedback either here or in the discussion forums. The CAML query schema is the first step. Planned future releases will include the other CAML schemas. My goal is simply to make it easier to write CAML code and thereby increase the productivity of SharePoint developers.

Stay tuned.

Technorati : .net, c#, caml, csharp, moss, sharepoint, wss
Del.icio.us : .net, c#, caml, csharp, moss, sharepoint, wss

Wrox Press has posted a promotional excerpt from the Professional SharePoint 2007 Development book due out in June. The topic is "Creating Content Type Metadata for SharePoint 2007 Document Management Solutions" from one of my chapters. To read the excerpt, go to http://www.wrox.com/WileyCDA/Section/id-305185.html. Comments and feedback are welcome.

Technorati : development, moss, sharepoint, wrox
Del.icio.us : development, moss, sharepoint, wrox