> ## Content Index
> Fetch the complete content index at: https://www.studert.com/llms.txt
> Use this file to discover other available public pages before exploring further.

# Your first Sitecore Search Crawler
- URL: https://www.studert.com/your-first-sitecore-search-crawler/
- Published: 2023-08-08T07:35:40.000Z
- Updated: 2023-08-21T13:09:28.000Z
- Author: Tobias Studer
- Tags: Sitecore Search

You just got access to Sitecore Search. Full of energy, you login for the first time and you stare at the screen with absolutely no idea what to do next. This was me, just a few months ago. In the meantime, I spent quite some time with the tool and it turns out, configuring your first crawler does not have to be overly complex.

## Modeling your content

Sitecore Search uses [entities and attributes](https://doc.sitecore.com/search/en/users/search-user-guide/working-with-entities,-attributes,-and-features.html?ref=studert.com) to represent your content. An entity is simply a collection of attributes.

For our example, we will use an entity called `Content` and some simple attributes for `Title`, `Description` and `Content Type`.

![](https://storage.ghost.io/c/f9/f3/f9f3e6b7-8305-49f1-a106-1c2d25c09539/content/images/2023/08/entity_and_attributes-3.jpg)

You can access the attributes in the *Domain Settings* in the *Administration* section. We will go in more depth about the configuration of attributes in another blog post.

💡

**Tip**  
If you get an error while adding a new attribute, try a hard refresh of the page and add the attribute again.

## Defining a Source

To get content into the search index, you need to setup a source. You can find the *Sources* section in the main navigation.

![](https://storage.ghost.io/c/f9/f3/f9f3e6b7-8305-49f1-a106-1c2d25c09539/content/images/2023/08/sources.jpg)

We will name it `default source` and select `Web Crawler (Advanced)` as our connector. This will give us the ability to index content in multiple languages and write JavaScript to extract attributes from the website.

To get your first Sitecore Search Crawler setup, you only need to define a trigger and a document extractor.

### Setting up a Trigger

[A trigger](https://doc.sitecore.com/search/en/users/search-user-guide/configuring-triggers.html?ref=studert.com) is the starting point that the crawler uses to look for content to index. For our example, we assume you're using SXA Headless on an XM Cloud solution in combination with the sitemap feature.

![](https://storage.ghost.io/c/f9/f3/f9f3e6b7-8305-49f1-a106-1c2d25c09539/content/images/2023/08/trigger.jpg)

### Setting up a Document Extractor

A document extractor adds an index document from your source content for each URL. We create a JavaScript extractor with one tagger. 

![](https://storage.ghost.io/c/f9/f3/f9f3e6b7-8305-49f1-a106-1c2d25c09539/content/images/2023/08/document_extractor.jpg)

The tagger is your JavaScript code that reads data from the website and adds it to the index document. A very simple tagger for our example could look something like that:

```javascript
function extract(request, response) {
  $ = response.body;

  var pageContentType = $('meta[name="content_type"]').attr('content') || 'Content';

  var title =
    $('h1.intro__title').text() ||
    $('meta[property="og:title"]').attr('content') ||
    $('title').text();

  var description =
    $('meta[name="description"]').attr('content') ||
    $('meta[property="og:description"]').attr('content') ||
    $('div.intro__content').find('p').text();

  return {
    type: pageContentType,
    name: title,
    description: description,
  };
}

```

We read the `pageContentType` from the meta tags. The `title` we try to extract from the content itself, with fallbacks to other properties of the page. For the `description`, we do the opposite - check meta tags first and fallback to the content later. Then, we return a JavaScript object, where the properties must match the attribute names defined earlier.

### Publish the changes

Once you finished your configuration, you need to publish the source. Publishing will also add a recrawl job to the queue. The job will use the trigger as a starting point and use the document extractor to create the index documents.

## Check the Index

After the index job has finished, you can look at the indexed documents in the *Content* view in the *Content Collection* main navigation entry.

![](https://storage.ghost.io/c/f9/f3/f9f3e6b7-8305-49f1-a106-1c2d25c09539/content/images/2023/08/content_collection.jpg)

Clicking on an entry reveals the attributes and their values.

![](https://storage.ghost.io/c/f9/f3/f9f3e6b7-8305-49f1-a106-1c2d25c09539/content/images/2023/08/content.jpg)

## Outlook

With this very high-level introduction to Sitecore Search, we are only starting to scratch the surface of what it can add to your platform. In future blog posts we will look at other aspects of the tool and also go deeper in certain areas to start leveraging the full potential Sitecore Search.