Plugins, Cached Memory and Global Variables

In the recent past I came across an issue where callouts that had been written for MSCRM 3.0 were behaving strangely in MSCRM 4.0. Whenever I would test the code it would work just as I expected it to. However, when users would test it values were not coming out as expected. Among different callouts I saw a number of different issues. Some being, the doubling of calculated fields, doubling of record creation (though one would get the data passed to it and the other wouldn't), among others. The trend that I noticed however was that things were doubling. I thought this was strange as other than the system being upgraded, nothing had changed.

After some testing I came to the conclusion that the new architecture was caching data. In a number of callouts I had used global (or class) variables, whether it was due to necessity or lack of time and a need to just get the piece done, and in every callout that I had written where I did not manage my global variables correctly I was getting strange results.

With that in mind I started changing how I coded and modified all code that was upgraded and used global variables. After discovering this I ran into a completely separate issue where the w3wp.exe process was running extremely high. It got past 1.5GB and was causing users to see an error stating "Out of memory exception." Only an IISRESET, or a reboot would remedy the problem, and that would only last a certain amount of time. I had not yet modified all of my code to not use global variables, I was under the assumption that if it wasn't broken, don't fix it. Well, it was broken users just weren't able to adequately inform me of the issues they were having because the errors were so sporatic. Needless to say the problem ended up in being the global variables. Once I removed them and/or managed them in my code the sporadic errors went away and the w3wp.exe process went down to about 400MB. That's a substantial change. Now, I will say that the client I was working on had about 35+ callouts running so they all helped to add to the problem.

So, all that being said, what point am I trying to make? The new plugin architecture that is used in MSCRM 4.0 utilizes caching to try to make the use of custom code more efficient by having it stored and more readily accessible to the application for when it is needed. What this caching means to those of us who use global variables is that global variables also get cached. So, if two instances of the same callout are running at the same time the values of the global variable are shared between the two instances. Let's assume we have a calculation running in a callout that is stored in a global variable. If two people were to trigger the callout at about the same time (within a few seconds of each other) the first callout would finish, cache the calculated value and pass that value into the next callout, starting it not at 0, or where one might expect it to be, but with the final result of the first callout. Get enough users clicking triggering the callout at the same time and you can imagine how quickly this issue will escalate.

I asked MS Support about this and here is what they said:

"Note For improved performance, Microsoft Dynamics CRM caches plug-in instances. The plug-in's Execute method should be written to be stateless as the constructor is not called for every invocation of the plug-in. All per invocation state information is stored in the context.

They will be adding a note that states we cannot use global variables with plug-ins for CRM 4.0. At this point, the code would need to be modified in either CRM 3.0 prior to the upgrade or the callout would need to be rewritten as a plug-in."

My suspicions now confirmed I made a stronger resolve to avoid using global variables. I thought I was the only one using global variables since I was the first one to bring this up (to my knowledge), either that or I was slow on the uptake and everyone else already knew about it. But MS Support also mentioned that they see code come in all the time with global variables and that this could be an issue that a lot of people bring to them in the near future. So, I know there are others like me out there, my hope is that I save you some of the time that I spent beating my head into the wall trying to figure this all out.

So, the moral of this story is, try to avoid global variables because they get cached. Use them is necessary but when you do be sure to manage them appropriately. And you might want to consider having two people trigger your custom code at the same time while you are still developing the solution that way you can make sure no reuse of values occurs where you don't want it to.

Hope this helps someone else out there.

David Fronk
Dynamic Methods Inc.