www.johntimney.com

Ok, this is a beauty! Deep searches, like advanced search queries or adding codeplex components like faceted search that make use of ordered search results between web front end servers and offset query servers appear to grind to a halt, deadlock the web app and force a recycle. The recycle may be an automatic one, or may have to be forced to get IIS to recover. However, this wasn't easily seen as the system was live and the users were just using the soltuion, when it would suddenly grind to halt. It was not something you could predict and we first thought it might be down to excessive load although the system was configured far in excess of the NFR requirements for transaction per second.

It took an age to track this one down and we were simply lucky that one of the chaps in the team spotted a slow down when another team member did something related to faceted searching, and data sorting. This only actually led us to realise it was connected to search somehow, but it at least allowed us to reproduce the problem and we could now regularly bring the platform to its knees. So, it was likely to be Query server related.

Taking a look at the even logs on the query servers, it was apparent one of them had dropped out of the MOSS farm. The error was "Source: Office Server Search EventID: 10038 Level: Error Query machine 'machinename' has been taken out of rotation due to this error. We had three of them and one missing from the rotation cycle was a clear indicator it was strugglng with something, although as the error was only on 1 server, it was not symptomatic of the problems we were having as it was the only one of the three query servers to be showing any event log errors. It merited further investigation however now we had some correlating evidence of root cause.

The rotation cycle error led me to a recently released KB description which had something in it that was very interesting: "the client computer stops responding for about two minutes". Our very symptom, as the WFE servers are clients to the offset role Query Servers! Even though it doesn't mention it could occur in this type of transaction scenario between a WFE server and a Query server.

Applying the hotfix solved this issue, and we suddenly stopped having server restarts due to deep search queries.

http://support.microsoft.com/kb/946487