As every SharePoint developer knows, the CAML is an ornery beast.  Get too close - it'll spit at you, and as soon as you get knee-deep in the hot sands of SharePoint-land, it's just a matter of time before it bites you in the ass!

But wait!  There's hope for all you erstwhile CAML-geeks!

Announcing the CAML.NET IntelliSense project

I've added a new component to the CAML.NET toolset that aims to help you tame that sometimes finicky, but ever-popular and oh-so-useful CAML language.

Click here to download the CAML.NET.Intellisense installer.

You probably already know how to add the SharePoint schemas to Visual Studio so you get IntelliSense when editing CAML files.  If not, here's a link to a great article by my friend Andrew Connell that walks you through the procedure.

This is a good first step, but we can take it even further by copying the core schema files and then extending them so they provide more information.  Specifically, we can add the detailed guidance from the SDK that is scattered all over the Internet and that developers seem to have a hard time finding. 

Let's face it - even with an up to date set of bookmarks, who wants to go searching all over the place for that missing tidbit that tells you just what that obscure attribute is for when you're right in the middle of building the CAML for your feature?  If you're like me, this quickly becomes a major distraction.  Hard enough to stay focused in the first place!

Using Annotated Schemas

The main idea behind the CAML.NET IntelliSense project is to extend the core schemas in two ways:

1. Gather as much information as possible about each element and attribute and place it into xs:annotation elements so that it pops up in context while editing; and
2. Identify and replace as many xs:string types as possible with enumerated types so the valid values for each attribute also popup in context while editing.

For example, using the core WSS schemas directly, we get a drop-down like the following while editing a Field element:

That's better than nothing, but it doesn't say what the attribute is for, so unless we're already familiar with it, we have to refer back to the SDK either online or in the CHM file.  On the other hand, using the enhanced annotated WSS schema, we can provide more information in context.  We only need to refer to the SDK documentation when we really need it.  Most of the time, we just need a hint, which is what Intellisense is all about, right?

Extending the Core Schemas with Enumerated Types

In addition to annotating the schemas with the corresponding SDK documentation for each element, we can go one step further and change the plain vanilla xs:string types to custom enumerated types so we get the list of valid choices for the attribute values.  This comes in very handy and also helps to eliminate those nasty little typos that can be *really* hard to find (and the CAML gets mighty snippy about).

There are lots of places in the SDK documentation where the expected attribute values are listed in the description, but are not enumerated in the schema.  This again means that we have to go digging through the documentation to discover what SharePoint expects the attribute values to be.  This can be an enormous waste of time - even if you have a pretty good idea of what the values are.  Who can keep all that detail in their head?

So, I've tried to find and enumerate as many attribute values as I could so they are available at your fingertips when you need them.  Each enumeration is also annotated, so you get contextual help for the available values as shown below.

Here's another example that shows the enumeration of list template types when adding a ListInstance element to a feature.  This is a biggie.  How many times have you had to hunt down the table of list template type identifiers?

Installing the Enhanced Schemas

The CAML.NET IntelliSense download package includes an MSI installer that copies the extended schemas into the local Visual Studio schema cache, so all you have to do is run the installer and then close and re-open Visual Studio.  The current version handles both Visual Studio 2005 and 2008.

Note: If you're already using the wss.xsd schema from the 12 hive, you should remove the previous catalog file from the Visual Studio schema cache.  You may also have to fiddle with your IDE settings a bit because Visual Studio somehow remembers the previous schema settings and adds both the old and the new schemas for files that have already been edited, event after you have removed the old catalog file.  This can interfere with the new catalog file you've just installed.  In the worst case, you will have to choose "don't use this schema" from the "schemas" dialog for files you have already started editing.

As you can imagine, there are a LOT of elements and attributes to deal with.  I've made a pretty good start at addressing the most frequently used items, and I'll continue to add more annotations as my schedule permits.  If you find missing or incorrect annotations, please drop me a line or leave a comment and I'll update the schema. 

Now you can ride that CAML high in the saddle and head on out to the SharePoint.  You know - the oasis?  Right next to the 12 hive!  See you there!

"The camel has a single hump,
The dromedary two;
Or else the other way around,
I'm never sure.  Are you?"

- Ogden Nash

Stay tuned for CAML.NET 2.0!

For the past several months, I've been experimenting with ways to use .NET reflection with the SharePoint API.  As I've mentioned before, I believe that .NET attributes can not only make it easier to build great SharePoint applications, but can also provide a foundation for building better development tools.

In this series of posts about the evolving SharePoint Reflection Framework, I'll continue to share the results of my research and I'll include links to sample code directly within each post.  As the code matures, I'll consider making it available on CodePlex so that others may contribute.  For now, let's just treat it as a "thought experiment" as my friend Andrew Connell likes to call it.

The sample code and the framework assembly can be downloaded here:

http://www.johnholliday.net/downloads/JohnHolliday.SharePoint.ContentTypes.zip.

There is also a short screencast (27 minutes) that shows how to use the framework which you can view here:

 

Now let's take a closer look at how we can apply .NET attributes to simplify working with content types. 

Some have asked, and you might also be wondering "Why do we need this?   Why not just use CAML to declare the content types and then simply publish them using a feature?  Why go to all the trouble of using this non-traditional approach?"

Well, there's nothing wrong with the traditional CAML-based approach.  There is ample support for it and with a few simple tools, it's pretty straight-forward once you get the hang of it.  But there are a number of limitations with the "code + markup" paradigm when it comes to reusability and extensibility.  The markup is "brittle" in the sense that subtle mistakes are hard to find.  True, you can preload the schema to help you remember what attributes go where, but there is no true intellisense and no strong typing to prevent those mistakes.  If you want to extend an existing content type, you have to  visit multiple files and switch back and forth between your event handler code and the markup.  You have to spend a good deal of time making sure that the code matches the properties that have been defined in the markup.

The unfortunate reality is that working directly with XML still requires a significant paradigm shift in the way most developers think.  With all the buzz about improving the SharePoint "developer experience", I keep thinking there must be a better way.

So, there are several goals for this experiment:

1. We want to make it easier to declare content types in code.
2. We want to eliminate the need to work directly in CAML or XML.
3. We want to use the same coding idioms we are used to when working with other code.
4. We want to make it easier to write custom event receivers for our content types.
5. We want to make it easier to associate sub-components like document templates and XML documents with our content types.
6. We want to be able to build libraries of reusable components that can be leveraged across multiple projects easily.
7. We want to enable DRM for content type components at the assembly level so that developers can protect their intellectual property.

The SharePoint Reflection Framework

For this experiment, we'll construct a content type for an expense report in C#.  But we also want to work with the expense report content type in the same way we might work with other C# classes.  For instance, we might want to create a library of reusable content type classes for use in financial solutions. 

To achieve this, we need to have a set of attributes that we can attach to a class to "declare" the content type and its properties.

We can start simply by deriving a ContentType class from System.Attribute.  To control where and how it is used, we can mark it so that it can only be applied to classes and only once per class.

[AttributeUsage(AttributeTargets.Class, AllowMultiple = false)]public class ContentType : Attribute{    public string Name { get; set; }    public string BaseType { get; set; }    public string Description { get; set; }    public string DocumentTemplate { get; set; }    public string Group { get; set; }    public bool Hidden { get; set; }    public bool Sealed { get; set; }    // ...}

namespace JohnHolliday.SharePoint.SampleContentTypes{    [ContentType(Name = "Expense Report",        BaseType = "Document",        Description = "A worksheet for recording expenses.",        Group = "SharePoint Reflection Framework Samples",        Hidden = false]    public class ExpenseReport    {    }}

Notice that we are basically specifying the same properties we would have to add to a <ContentType> element in CAML.  The difference is that here we get strong typing and intellisense support.  Notice also that we don't have to worry about constructing the content type identifier.  Instead, we just supply the base type and let the framework handle the details.

Next, within the content type declaration, we need to specify the fields that make up the type.  Here again, .Net reflection comes to our aid to make things easier.  What we really want is to declare the fields of the content type in the same way that we might declare properties for any C# class.  So, if we wanted to add a description field to the expense report content type, we would naturally want to do something like this:

[ContentType(Name = "Expense Report",    BaseType = "Document",    Description = "A worksheet for recording expenses.",    Group = "SharePoint Reflection Framework Samples",    Hidden = false]public class ExpenseReport{    private string m_description=string.Empty;    [FieldRef("Description")]    public string Description {        get { return m_description; }        set { m_description=value; }    }}

Here, we can simply make use of a second attribute included in the framework to markup the properties we want to map to SharePoint fields.  The FieldRef attribute is provided for this purpose.  The framework supplies the necessary code to examine the ExpenseReport class, locate the mapped properties, determine their field types and link them automatically to the content type on creation.  For instance, if we add a DateTime property to the ExpenseReport class, we want it to surface in SharePoint not as text, but as a date field.   Similarly, the other field types can be automatically determined from the underlying property type.

Enumerations

We can extend this idea further to include built-in support for CHOICE fields if an enumerated type is mapped to a field.  For example, we might need to declare a ProjectTypeEnum enumeration and then map it to a ProjectType property as shown below. 

public enum ProjectTypeEnum{    Consulting,    Training,    Programming,    Sales,}private ProjectTypeEnum m_projectType;[FieldRef("ProjectType", DisplayName = "Project Type", Required = true,    Description = "The project type for which expenses are being submitted.",    Group = "Sales")]public ProjectTypeEnum ProjectType{    get { return m_projectType; }    set { m_projectType = value; }}

In this case, the framework detects that the underlying data type of the property is an enumerated type and it then generates the appropriate choices automatically, as shown here.

We can do a lot more with field references, but let's move on to the question of how to create the actual content type within the SharePoint environment.

Creating Instances

Typically, we'll be writing a FeatureActivated feature receiver or building a command-line utility, and we need to create a content type as part of the solution.  Ideally, we'd like to end up with an instance of SPContentType that we can work with.  So we really only need the framework to handle the dirty work of talking to the SharePoint API and then returning a properly constructed SPContentType object.  One way to implement this pattern is to provide a factory method on the ContentType attribute class that takes a reference to our implementation class and then gives us back an SPContentType object.  For the expense report content type, we might call it like this:

using (SPSite site = new SPSite("http://litwareinc.com")) {    using (SPWeb web = site.OpenWeb()) {        // Create the expense report content type.        SPContentType ctExpenseReport = ContentType.Create(web, typeof(ExpenseReport));        if (ctExpenseReport != null) {            // do something with it        }    }}

Note that the ExpenseReport class is completely independent of the SPContentType object it was used to create.  It can be implemented in a separate assembly.  Using this approach, we can easily build libraries of content types that can be reused across multiple solutions.  But there are still a couple of additional requirements we need to fulfill.  Namely, how do we handle item event receivers?  And what to do about the nested sub-components of a content type, such as document templates and XML documents?  These are critical requirements, because we want to apply the same coding paradigm we are used to for other C# classes.  We don't want to have to revert to using XML markup in order to specify event receivers and other elements.

Item Event Receivers

To specify event receivers for a content type in code, we need to add the appropriate entries to the EventReceivers collection of the SPContentType object.  When working with XML markup, this is done by creating a special XML document that specifies which event we want to capture and the assembly and class in which the event receiver method is implemented. 

To deal with event receivers in our ContentType attribute, we can take advantage of the fact that SharePoint provides an abstract class called SPItemEventReceiver that declares all of the event receiver methods.  By simply deriving our ExpenseReport class from SPItemEventReceiver, we get the correct method declarations in the base class and then we can selectively override the ones we want.  Instead of telling the framework explicitly which methods we are implementing as we would have to do with CAML, we can let the framework do the work for us and infer which methods have been implemented by examining the class via reflection.  So all we have to do to handle the ItemAdded event, for example, is add the derivation and override the ItemAdded method as shown in the following code.

[ContentType(Name = "Expense Report",    BaseType = "Document",    Description = "A worksheet for recording expenses.",    Group = "SharePoint Reflection Framework Samples",    Hidden = false)]public class ExpenseReport : SPItemEventReceiver{    // ...properties    public override void ItemAdded(SPItemEventProperties properties)    {        base.ItemAdded(properties);        string message = string.Format("Expense report added to list: {0}",              properties.ListTitle);        EventLog.WriteEntry("SharePoint Reflection", message);    }}

Still, no CAML.  We are working entirely within a C# class and we have captured all of the information needed to create the content type and to install an event receiver for the ItemAdded event.  The framework looks at the ExpenseReport class and sees that only one of the abstract classes was implemented, so it creates an event receiver for that method only.  It knows which assembly the class is declared in, so it automatically assigns the assembly and class names.  If we want to add an event receiver for another event, such as ItemUpdated, we can simply add another override and implement the method.  The framework automatically registers it for us.

Document Templates and XML Documents

The thing about document templates and other kinds of files that we might need to associate with a content type is that they are files.  Ideally, we'd like to work with them as files without having to jump through hoops just to get SharePoint to recognize them.  Fortunately, Visual Studio has a nice feature that lets us access and work with files easily from within our code.  These are embedded resources.  If we change the build action for any given file within our project to EmbeddedResource we can reference it using a path expression of the form <folder name>.<sub folder name>.<...>.<filename>.<extension>.  So if we have a project folder named "DocumentTemplates" and a file in that folder named "ExpenseReport.xlsx", and both are part of a project named "SampleContentTypes", we could refer to the resource using the path "SampleContentTypes.DocumentTemplates.ExpenseReport.xlsx".

By extending the ContentType attribute to recognize and locate embedded resources, we can easily add document templates and other resources directly to our class declaration without having to worry about how they actually get copied into SharePoint.  Here again, we can delegate that responsibility to the framework and just focus on the content type implementation.

For consistency and to distinguish document templates specified as embedded resources from those specified with legitimate urls, I've added a "res://" prefix which the framework recognizes.  It then treats the url as a reference to an embedded resource and copies the file into the proper location when the content type is created.  Since it already knows where the resource is located, we can drop the assembly name and just specify the assembly-relative path to the resource file.  We end up with a declaration that looks like this:

[ContentType(Name = "Expense Report",    BaseType = "Document",    Description = "A worksheet for recording expenses.",    Group = "SharePoint Reflection Framework Samples",    Hidden = false,    DocumentTemplate = "res://DocumentTemplates.ExpenseReport.xlsx")]public class ExpenseReport : SPItemEventReceiver{    //...}

Future Directions

As you can see, there is a lot of promise to this approach, but we've only begun to scratch the surface.  I'm currently working on one-way data binding of list items to properties in the implementation class whenever an event receiver method is called.  This would essentially let you reference the mapped properties just like other properties in your event receiver methods.  It would be one-way only because there might be times when you don't want those properties to get copied back into the list item automatically.  For that, you can always call into the SharePoint API.

There is a lot more we can do with content types as well as with other SharePoint objects that can really make our lives easier as SharePoint developers.  Give the code a spin and tell me what you think.  In the coming weeks, look for additional posts that will go into more capabilities of the framework. 

Stay tuned.

From the title of my blog, one might surmise that I'm kind of into reflection. (Ya think?)  Well, although I haven't blogged about using .NET reflection for SharePoint development since back in 2006 (see my article "Use .NET Reflection to Create SharePoint Sites and Site Definitions"), I've continued experimenting with the idea and may have actually come up with something useful.

My original article dealt with the problems associated with writing site definitions for WSS 2.0.  Since then, with all the new capabilities they've stuffed into WSS 3.0, the basic problem still remains - how to reduce the ever-increasing "surface area" faced by SharePoint developers?  I still find it somewhat disconcerting to write XML markup along with my managed code while keeping everything in sync.  Every time I conduct a training class for SharePoint developers, they all echo the same sentiment.

So, what if you could write a SharePoint Feature like this?

All C# - all the time.  No CAML to speak of. 

Of course, there's the question of how to get SharePoint to understand it.  There's also a bit of plumbing required under-the-covers to make everything work.  But the end result is somewhat compelling, isn't it?  

If you're intrigued, then click here to read the entire article.  I've basically created a SharePoint Reflection Framework that I've started to use in my own development work.  It's getting to the point now where I'm feeling the need to share the fruits of my labor.

And don't resist the urge to give me some feedback!  I'm always on the lookout for beta testers!

Stay tuned.

JFH

Consider the following scenario:

You're building a SharePoint Feature in Visual Studio and you need to add a couple of custom content type definitions to the project.  The first thing you have to do is specify the site columns for the content type.

Although SharePoint makes it easy to create a new content type from the user interface, building one from scratch requires a little more effort.  Adding fields through the user interface is just a matter of browsing through the available fields and selecting the ones you want.  Doing it in XML means you have to know the unique field identifier and field name.  This information can be difficult to find, even if you know which file it's declared in.

Let's say you want to define a new content type named "Resume" that includes some of the built-in site columns like "Birthday" and "CellPhone".  Your content type definition would look something like this:

One approach is to do it in two steps.  Step 1 - create the content type via the UI in a dummy site.  Step 2 - use a command-line utility or a custom STSADM command to extract the content type definition and then add the generated XML to your project.  Granted, this works, but it's a lot to go through when you just need to add a couple of fields to your content type definition.

Here is another approach that you might find useful, at least for the built-in site columns that come with the platform.

The built-in site columns are declared in an XML file named "fieldswss.xml", which is located in the 12\TEMPLATE\FEATURES\fields folder.  If you take a look at this file, you'll see that it has over 4000 lines of XML.  Even if you know the name of the field you're looking for, you'll spend a good deal of time searching through the file to find the ID and display name you need.

TIP: Transform the file using XSL so you don't have to fool around with the raw XML.

Here are the steps:

1. Start by COPYING this file to a separate folder, like C:\TEMP.  (It's always good to make a copy of files you are working with in the 12 hive!). 
2. Create a little XSL stylesheet like the following.  [ Instead of simply displaying the field identifier, you can generate the full declaration needed when adding the field to a custom content type. ]

   				<?xml version="1.0" encoding="utf-8"?>		<xsl:stylesheet version="1.0" 		xmlns:xsl=http://www.w3.org/1999/XSL/Transform 		   xmlns:wss="http://schemas.microsoft.com/sharepoint/"> 		<xsl:output method="html" version="1.0" encoding="utf-8" indent="yes"/> 		<xsl:template match="wss:Elements"> 		<html> 		<body> 		<h2>SharePoint 3.0 Built-In Fields</h2> 		<table border="0" width="100%" style="font-size:9pt;"> 		<tr bgcolor="#9acd32"> 		<th align="left">Field</th> 		<th align="left">Group</th> 		<th align="left">Type</th> 		<th align="left">Declaration</th> 		</tr> 		<xsl:apply-templates> 		<xsl:sort select="@Name"/> 		</xsl:apply-templates> 		</table> 		</body> 		</html> 		</xsl:template> 		<xsl:template match="wss:Field"> 		<tr> 		<td><xsl:value-of select="@Name"/></td> 		<td><xsl:value-of select="@Group"/></td> 		<td><xsl:value-of select="@Type"/></td> 		<td> <!-- Emit the Full Declaration Here --> 		             &lt;FieldRef ID="<xsl:value-of select="@ID"/>" 		Name="<xsl:value-of select="@Name"/>" 		DisplayName="<xsl:value-of select="@DisplayName"/>" /&gt; 		</td> 		</tr> 		</xsl:template>		</xsl:stylesheet>		
   		

3. Save the file into the same folder as your copy of fieldswss.xml.  In this example, I've named it fieldswss.xsl.
4. Open the fieldswss.xml in Visual Studio.
5. Open the Properties tool window, select the Stylesheet property and browse to the fieldswss.xsl file you just created. 
   
   
   
6. Finally, from the XML menu in Visual Studio, select "Show XSLT Output".
   
   

Voila!  You now have a nicely formatted table of built-in fields.  Now all you have to do is copy and paste the fieldref declaration into your content type definition.  What's nice is that you don't have to jump back and forth between the IDE and a separate application.

This is just one example of how you can use XSL effectively when building SharePoint solutions.  Other areas include site definitions, list definitions and control templates.  The ability to transform the templates from within Visual Studio makes it especially nice because you don't have to bounce back and forth between the UI and the IDE.  Once the transformation has been done, you can simply save the generated HTML file for future reference. 

And if you're feeling really lazy, here is a link to the generated table of built-in fields.  You can download the stylesheet from here.

Enjoy.

Boy, news travels fast on the Internet!  Awhile back, my fellow SharePoint MVP Carlos Segura Sanz came up with a pretty nifty tool that lets you create CAML queries using T-SQL syntax.  It's called YACAMLQT for "Yet Another CAML Query Tool", and you can read all about it here. 

Basically, it lets you write a query like this:

	WHERE ContentType="My Content Type"
	OR Description<>null
	GROUPBY Title DESC
	ORDERBY _Author, AuthoringDate, AssignedTo ASC
	

YACAMLQT then parses the text and produces a CAML query that looks like this:

Pretty cool, eh?

So I got to thinking, "Why not extend the tool to generate CAML.NET code?".  Then you would have the convenience of T-SQL with the power and extensibility of C#.

Well, wouldn't you know it?  Carlos was already way ahead of me.  In fact, he had already implemented it.  Here is a screenshot of the latest version of YACAMLQT, that now has an extra tab containing the generated CAML.NET code!

I'm working with Carlos to extend his tool to support some additional feature, like adding a SELECT clause so you can easily specify the view fields.  I'm also looking at ways to incorporate the YACAMLQT parser directly into the CAML.NET framework so you could do things like pass a T-SQL string to an existing CamlQuery instance to take advantage of the built-in data binding features of CAML.NET.

Look for updates to appear soon on both our blogs, and also on the CAML.NET Project on CodePlex.  In the meantime, you can download the updated YACAMLQT tool from the following link:

 YACAMLQT.zip (15,07 KB).

The true power of CAML.NET is the ability to create reusable queries that can be applied repeatedly to different scenarios, extended via simple inheritance and bound automatically to custom data types. 

Although the framework is being offered *FREE* to the SharePoint developer community, I initially thought it prudent to obfuscate the code so that the platform could evolve without breaking solutions being developed by others.

Probably not the best idea I've ever had.  :)

It seems the obfuscation succeeded only in creating a lot of frustration.  The issue is that many of the static string methods used to generate the underlying CAML are modified by the obfuscation process and rendered inoperable.  After spending many hours trying to come up with a work-around, I finally said to myself, "Hey! Who are you protecting, anyway?  The point is to have better tools to write better SharePoint solutions, right?  So what if a new version breaks a little code here and there, right?"

So, I've published the full source on CodePlex at http://codeplex.com/camldotnet for your downloading pleasure.  I'll continue to improve and extend the framework as well as the documentation set, so you might want to stay tuned here for updates on new features and the latest developments. 

I've got lots of ideas for how to improve the framework, and I'll bet you do too!  So let's hear them, either via comments to this blog, or on the codeplex site.  I'm particularly interested in building up a collection of useful queries that are general enough to be applied to many different kinds of solutions.

To support the growing interest in the CAML.NET Framework, I've created a new Community Server site just for CAML.NET developers.   You can join the community at http://johnholliday.net/camldotnet.  From there, you can share your feedback and experiences and collaborate with other developers.

The site includes links to walk-through videos, online documentation and related resources.  It also includes an area called the "CAML.NET Query Exchange", where you can upload your compiled CAML.NET queries.

I warmly invite you to join the discussion forums and share your thoughts about the CAML.NET approach to building SharePoint solutions.

Stay tuned.

The CAML.NET Framework documentation is now available online in the same format used for the MSDN docs at http://www.johnholliday.net/camldotnet/documentation.   This documentation set describes all of the CAML.NET static methods as well as the ICamlQuery and ICamlField interfaces along with the CamlQuery base class from which you can derive your own reusable query classes.

You can also download the compiled help file.

As always, your feedback is welcome.

This session will introduce the CAML.NET.NET framework in detail, showing how you can use it to take full advantage of the CAML.NET query schema in your WSS and MOSS applications.

• Presented by: John Holliday
• Date: Thursday, August 23, 2007; 6-8 PM
• Location: MODIS Building, Downtown Jacksonville; Conference Room A, 2nd FL
• Meeting Location Map: http://floridaofficegeeks.org/JOG/Pages/Meeting%20Logistics.aspx
• For more information or to RSVP: http://www.clicktoattend.com/invitation.aspx?code=119500