MSCRM Server Performance Counter Errors

In the last week or two I've gotten bombarded with Performance Counter information. Some was voluntary, some was forced due to errors on client's servers. I am by no means an expert but I do have to say that I have gleaned a fair amount of information that I bet the typical MSCRM admin or consultant doesn't know about. There just isn't much information out there specifically about Performance Counters within MSCRM.

My post last week talked about running the lodctr /R command. I found a post that stated that running just lodctr /R will rebuild all system Performance Counters and thinking this would clean things up I ran it on a server...not smart. At least not smart in the MSCRM world. Everything worked just fine, including CRM, until I ran an IISRESET later in the day and then CRM just died. I found articles on how to manually rebuild Performance Counters (http://support.microsoft.com/kb/300956) and that works just fine for Windows Server Performance Counters just fine. But not MSCRM. There is even a KB article in PartnerSource about Performance Counters but it says that running a repair would work just fine and resolve the problem. Well, I'd tried that before I'd even found the article and about 70% of the way through the repair it would error out with (the all too familiar error to me by now):

The requested Performance Counter is not a custom counter, it has to be initialized as ReadOnly.

Only this was on the server and what I did to fix the client was what caused the problem. So, running it again (which I tried) was zero help.

After many, many hours of deliberation I finally spoke with Adam in Australia from MS Support and he helped clear up everything and demistify Performance Counters for me. There are 4 important registry keys that need to be set up correctly in order for Performance Counters to work properly in MSCRM. Oh, and for whatever reason, if the Performance Counters fail then the whole application fails (as if I didn't realize that by now). But most other applications just allow them to fail and move on. They aren't system critical, they just track the performance of the application. Adam has submitted the request of Performance Counters not being a system requirement in future versions, hopefully they listen to his request and/or read this post.

Anyway, back to the fix. In the registry there are 3 keys that need to exist, all are under the following paths:

My Computer\HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Services (add the last two hives of each hive listed out below to get to the actual keys)

1. CRM Async Service
2. CRM Authentication\Performance
3. CRM Discovery\Performance
4. CRM LocatorService\Performance
5. CRM OutlookSync\Performance
6. CRM Platform\Performance
7. CRM Server\Performance
8. CRM Sql Governor\Performance
9. CRM Sql Governor Multi Instance

(Basically and hive under the Services hive that has CRM in it is what you need to check.)
Oh and one more for workflow performance

10. Windows Workflow Foundation 3.0.0.0

The keys that must be there and their values are:

Key--------------------------Value
Library----------------------netfxperf.dll
First Counter----------------(any value is fine, it just needs to exist)
Last Coutner----------------(any value is fine, it just needs to exist)

Running a repair on CRM will fix all of the Performance Counters for the CRM hives but it won't fix the Workflow hive. So, if you get the error regarding Performance Counters in the middle of your repair I can almost guarantee that your problem is that you are missing keys in your Workflow Performance hive. That's what I had to do and according to Adam (who has written the majority of the Performance Counter KB articles for the MSCRM Support team) only the Workflow Performance Counters would cause the repair to fail since the repair rebuilds them.

Now that we know what needs fixed, how do we fix it? If its CRM Performance Counters, just run a repair on the server. If you error during the repair with a Performance Counter error then you will need to reload the Library, First and Last Counter keys for Workflow Foundation. In order to do this you must use the lodctr command. DO NOT recreate these manually. I haven't tried it so I can't speak to what bad things could occur if you did I just know that you should rebuild them using the lodctr command.

You need to open a command prompt and go to the following directory:

%SystemRoot%:\WINDOWS\Microsoft.NET\Framework\v3.0\Windows Workflow Foundation

Then run the following command:

lodctr perfcounter.ini

That will rebuild the Workflow Foundation Performance Counters (which MSCRM is dependant on). If you look at the registry you should now see the keys there with correct values. Continue your repair, or try it again and you should be all set. I would recommend a reboot after the repair in this case to get everything reset and you should be back up and running.

For those of you who don't know what Performance Counters are used for, they can be very useful in showing the performance of the server and where the majority of the load is so that you can appropriately manage the server/network and correct the bottleneck of CRM. To view them open a run prompt and type "perfmon". This will open the Performance Monitor. By default it shows Pages/sec (if IIS is installed), Avg. Disk Queue Length, and % Processor Time. You can add other Counters to this list by right clicking in the list of counters at the bottom of the main section of Performance Monitor. Choose "Add Counters..." In the new window that comes up expand the "Performance object" dropdown and pick the application/object you want to monitor. Below that will be all of the counters you can monitor. Take a look at those and you'd be amazed at what you can track for CRM. There are also a lot of really good SQL ones for your SQL server. If you are looking for metrics to show companies the performance increase by going to MSCRM 4.0, or moving to a 64 bit SQL box, this will give you a great comparison and actually show the big wigs that their decision made a big difference.

I have learned a lot about Performance Counters and hopefully my lengthy post has helped someone else glean a little more information as well.

David Fronk
Dynamic Methods Inc.