Crawled and Managed properties in MOSS 2007

By: Ved Mishra | Comments | Related: > SharePoint Configuration

Problem

I understand that SharePoint lists and libraries contain extra columns (i.e. Author, Description, Status etc) that can be used to group, sort and filter content items as needed. These extra columns are known as "Metadata".

I also understand that metadata is collected by SharePoint's search crawler and can be used in various ways by developers and administrators. I would like to understand more about this.

Solution

Crawled Properties: All the site columns(i.e. metadata) are picked by the Search Crawler based on the protocol handler used.

Notes:

Site columns with blank data are not picked by Search Crawler. The site columns must have some content in order to be picked by Search Crawler as Crawled Property. In case, you run into this issue, add some content (at least 1 item) and do an incremental or full crawl.
Search Crawls normally take a good amount of time if the amount of content is large.

Crawled properties can be seen in the Shared Services Provider’s Metadata Properties page. Launch SharePoint 3.0 Central Administration -> Select Shared Services Administration -> Select the Shared Services Provider -> In the Shared Services Provider, you can see the categories of the Crawled Properties by one of the following two ways:

Search Settings
Search Administration (if the Infrastructure Update is installed)

Click on any of those 2 links. Click on the "Metadata Properties" under the "Queries and Results" heading in the left navigation.

Click on "Crawled properties".

There are 11 crawled property categories that are available out of the box. You can see this to understand about all the categories. However, here we are interested in the SharePoint category.

Any crawled property that is prefixed with "ows_" comes from SharePoint. This is how it avoids conflict with crawled properties from other categories.

Content Sources: A content source specifies settings that define how and on what schedule content is crawled. It includes one or more addresses of a content repository from which to start crawling, also called "start addresses". These settings apply to all start addresses within the entire content source. Let us create a content source which can be used to crawl the site collection, which contains the Tips column data.

Click on the link "Content Sources" in the left navigation and then click on "New Content Source".

Create a content source. Give it a name and the start address of your site collection.

Click on the arrow next to the new content source you created and start a full crawl.

The crawl will take some time depending on the amount of data you have in your site collection

Keep refreshing the page until you see the crawl is complete and the status is set to "Idle".

Now, the crawler would have picked new site columns which has some content. Let us verify. Go to Crawled properties -> Sharepoint category. Search for the property "ows_Tips" (remember the crawler adds "ows_" as prefix to Sharepoint crawled properties.

Metadata Properties: The Metadata property mappings, map properties extracted from documents during crawls, called crawled properties, to managed properties that users can use in search queries. Now in order to use the "Tips" crawled property in a search, we need to create a metadata property and map it to the "Tips" Crawled properties

Click on Metadata properties in left nav. Click on "New Managed Property"

Give a name to Metadata property (in this example - Tips) and click on Add Mapping. Map this to "ows_Tips" Crawled property. Make sure that you check the box "Allow this property to be used in scopes". Doing so, will make this managed property available for search scoping (this will be explained later in this article).

Please note that after mapping the property, you need to do another crawl for the content to be picked. As you see in the figure below, the number of items found in this property shows 0.

After the crawl is complete, it will show the actual count as shown below:

Now the metadata property is ready for searching. There are various ways you can use the metadata property for search requirements. A few options include Basic search, Advanced search, search query etc.

Use in Search scope:

Go to Site -> Site Actions -> Site Settings -> Modify All site settings -> Click on Search scopes under Administration.

Click on create "new scope".

Create a scope (Tips).

Now we will have to create a rule for this scope. Click on Add rules.

You can use the property query option to set rules specific to the metadata. For example, let us say we want to search only content which has the Tips column value equal to "Indexing".

Please note that only those properties are shown in the drop down, which have its crawled property set to "Allow this property to be used in scopes" (as explained above).

After you click OK, the scope takes some time to get updated.

After the scope is ready, you can see the number to items available under this scope.

As you can see below, this search scope is available in the basic search scope dropdown.

Now going forward, whenever data is added in the list, only an incremental crawl is required for it to be available for searching. For example, let's add one record in the list as below.

Go to search SSP Admin -> content source and do an incremental crawl.

It should be quicker than a full crawl as it just has to pick the changed data from the last crawl instead of crawling the all of the content.

After the crawl is complete, you can go to the scope and see that the count has been increased to 2.

Use in "Advanced Search": Metadata properties can be used to narrow down search results in Advance Search. As you see in the figure below, the Tips scope is available under "Narrow the search" section.

We can also add a custom property restriction in Advanced Search. To do this, we need to modify the advanced search property XML statement.

Please remember to add both the PropertyDef and PropertyRef nodes. It is recommended to edit the XML in Visual Studio or some XML editor.

Use in a custom search query: These metadata properties can be used as columns when writing search queries (either through the API or the search web service). Below is an example for a search query using FullTextSqlQuery

Planning search is an important part of any Microsoft Office SharePoint Server 2007 deployment. If you devote time to plan how search is implemented, you can save time in the future and can reinforce the effectiveness of other business processes. For targeted and large environments, Metadata and Crawled properties are key pieces to SharePoint search.