Microsoft SharePoint Products and Technologies Administrators Pocket Consultant
Crawler impact rules can be used to calibrate the aggressiveness of your crawl efforts. The default settings ask for eight documents in a single request and then wait 0 seconds before asking for another eight documents after the first set of documents has been received. You can change both the number of documents in a single request (up to 64 documents) and insert a wait period between requests, configured in seconds.
You may need to calibrate how quickly and aggressively SharePoint Server crawls a content source to ease the burden on the target server. In addition, you may also need to configure one or more rules to help ease the burden of the crawl procedures on the Web front-end servers in your farm. This is because all of the crawl processes are actually passed to the Web front-end server, which then connects to the content source server on behalf of the index server and executes the crawl procedures. The index server does not connect directly to the target content source server. Instead, all of the crawl requests are proxied through your Web front-end servers. Therefore, as a side note, when monitoring the performance of the index server to assess the load placed on it by the crawl action, be sure to simultaneously monitor the Web front-end server to gain a full picture of how the crawl procedures are affecting the servers in your farm.
To create new crawler impact rules, perform these steps:
-
From the Applications tab in Central Administration, click the Search Service link. You are then presented with the Manage Search Service page.
-
Click the Crawler Impact Rules link. The Crawler Impact Rules page then appears.
-
Click the Add Rule link. The Add Crawler Impact Rule page, as shown in Figure 12-20, then appears.
-
Enter the site URL (wildcards are allowed on this page; refer to Figure 12-20).
-
Configure the request frequency in terms of both seconds and number of documents requested.
Note Because there are option buttons, you cannot configure both settings at the same time. If you want to change the number of documents requested and configure a wait time between requests, first make your document selection and then configure the wait time. The document selection configuration will persist when the wait time configuration option button is selected.