Category Archives: Issues & Resolution

Investigating Performance Problems using Memory Dumps

I recently had to brush up my WinDbg knowledge due to a performance issue that occurred in production environment.

Normally you don’t have to go to the memory dumps route to get an idea on what’s causing the performance bottleneck in your application, if you have an APM tool such as New Relic you would be able to tell the hotspots in your application – if you don’t have an APM tool then at a minimum you need to use Windows performance counters to gather metrics around hardware utilization (CPU, Memory, Disk I/O) and the ASP.NET performance counters.

I especially like the New Relic Thread Profiler feature, which provides a unique ability to run a profiler against your application.. even in production environment – no I didn’t made a mistake there. You can read more about it https://docs.newrelic.com/docs/apm/applications-menu/events/thread-profiler-tool

And oftentimes NR Thread Profiler analysis will provide me with the information that I need to pinpoint the application bottleneck and work with the team to come up with optimization plan.

But there are times where you just need more information on what’s going on with your application in a lot more detail manner, that extra information that would be able to help you validate your hypothesis. Enter the memory dump analysis..

Capturing memory dumps

There’s an existing Sitecore KB article that goes and explains how to capture memory dumps using different tools which you can refer to https://kb.sitecore.net/articles/488758

Memory dumps analysis

There’s a couple of tools that you can use to analyze your memory dump files, I’ll go through each one and explain what I use them for.

Debug Diag

This tool is very useful as it will find problematic areas of your application based on the memory dump file that you provided, you can also set multiple memory dump files which is taken from the same server consecutively to get better result on performance analysis.

Once you have the memory dump file available, what you want to do is to open it using Debug Diag, select the analysis rules and ran an analysis.

It will then generate a .mthml file that can be opened with Internet Explorer and when you open it you will able to see the summary of the analysis outcome and the stack trace of the offending threads.

WinDbg

WinDbg is the main tool in which you will spend most of your time doing memory dump analysis as it’s very powerful and has plenty of useful commands and extensions that can help you narrow down the cause of the problem.

There’s a new version of WinDbg called WinDbg preview available through the Windows Store which has more modern look to it, but the commands that I reference in this section should apply to the older version as well

The first thing that you want to do is to load the SOS extension

.loadby sos clr

And the Mex extension – which you can download at https://www.microsoft.com/en-us/download/details.aspx?id=53304

.load [fullPathToMexExtensionFolder]\mex.dll

And the Sosex extension – which you can download at http://www.stevestechspot.com/

.load [fullPathToSosexExtensionFolder]\sosex.dll

Useful commands

!threadpool

This command allows you to check CPU utilization when the memory dump was taken and the number of active worker threads and IOCP threads.

!runaway

This command will display all the threads that were running and order them by the execution time that it has taken until the memory dump was taken

~[threadId]s

This command allows you to set the context to the specified thread id which allows you to run commands against that thread

!clrstack

If the current thread execute a .NET managed code then it will dump the full stack trace – which is useful to help identify problematic code in your application

!do [address]

This command dumps the object information of the specified memory address

!syncblk

This command will display all the threads that own a lock, if there’s a thread that has a high MonitorHeld value then it suggests that you might have a deadlock or a lock contention issue. The value is either 0,1, or any odd numbers. For example, if the value is 89 then this thread owns a lock which is represented by a value of 1, and the remaining 88 count means that it is waited by other 44 threads – each waiting threads holds a value of 2

!sosex.dlk

This command will analyze whether you have a deadlock that occurred between the threads.

!mex.clrstack2

Similar to the !clrstack command above but better.

!mex.sqlcn

This command displays the information around .NET SQL Pool objects which is useful to find out how many SQL connections were open.

!mex.aspxpagesext

This command will display all the running and completed aspx http request that was processed during the time of the snapshot. You can sort the result by the time longest time it took so that you can isolate your area of performance problem.

Visual Studio

You can open your application source code files in Visual Studio and the memory dump file in order to be able to switch between the threads and see what line of code they are executing. I normally combine this with WinDbg where I use WinDbg to isolate problematic threads and have the convenience of Visual Studio to be able to explore the source code.

Jetbrains DotMemory

You can also open the memory dump with Jetbrains DotMemory to get an idea of what objects that was created by your application. You want to get a good idea of what’s normal for your application here, how many custom objects do you expect and whether there’s some anomaly that occurs.

If you’re investigating memory leak, you would spend more time investigating this area.

Sitecore Pipelines – Thread Safety

This might not be a widely common knowledge, but when creating your own Sitecore pipeline processors you need to make sure that your code is thread safe as by default the pipeline processors will be created as a singleton.

I’ve encountered it before a few years ago and recently stumbled upon it again. Thus this blog post serve as a reminder for my future self.

Take a look at the following example

When testing locally with a single user the issue won’t become apparent as there’s usually only a single request that will trigger the code block but given enough volume of incoming requests then you would start seeing a weird behavior where the result is being overridden by a different thread.

The reason why by default Sitecore choose to create the pipeline processor to be created as a Singleton most likely is due to performance, though I wish this was officially documented.

Another way to re-mediate the issue is by changing the object life cycle from Singleton to Transient, by doing this Sitecore will create a new instance of the processor class whenever the pipeline processor is executed.  This can be done by setting the reusable property to false.

Sitecore Commerce Server Connect.10.0.70.update package installation hangs

I had a strange issue where I previously able to successfully setup a couple of Sitecore Commerce 8.2.1 initial release instances then suddenly creating a new instance keeps failing.

To be exact, it was failing when trying to install Sitecore Commerce Server Connect.10.0.70.update package through the Sitecore update installation wizard page, it just keeps loading indefinitely. Changing the Sitecore log files to DEBUG mode and checking the Windows event log file doesn’t reveal anything. I do notice that memory consumption for the W3WP process is absurdly high in my local machine and it seems like something went wrong.

True enough, after debugging the W3WP process it reveals that the process throws a StackOverflow exception which stems from NewRelic Agent process that I’ve installed yesterday. Uninstalling the NewRelic Agent seems to fix this issue and I’m able to continue with the packages installation per normal.

Sitecore Server Role Checker Tool

When configuring Sitecore in a distributed environment, you typically have more than 1 server in the production environment configured as a different roles (CM, CD, Processing, Reporting Service, etc).

More often than not I’ve noticed that issues are raised due to misconfiguration rather than implementation itself. You then go to Sitecore documentation site and check the configuration for each of the server that you’ve setup to see if there’s anything that you miss.

In the 8.0 documentation you would need to read a long list of tables containing information which config files that you need to enable/disable.

Since the 8.1 release, these steps are simplified with Sitecore providing us an excel spreadsheet file as a guide to enable/disable the config files depending on the role(s) that you want to setup. These steps are manual though and highly likely that we will miss one or two config files and could cause some issues down the line.

With the number of projects that you need to review, this task will start eating up your time and should really be automated. Some might have already done so and create a little tool tucked away somewhere.

I’ve decided to create my own version of the tool called Sitecore Server Role Checker, can’t be more obvious than that 🙂

How does it work?

This tool would basically uses a converted csv format from the Sitecore official spreadsheet guide and read the configuration based on your selected roles. It currently supports the following Sitecore version

  • 8.1 update 3
  • 8.2 initial release
  • 8.2 update 1
  • 8.2 update 2

Only those version is supported as those are only the spreadsheet available for now.

How do I use it?

Follow this simple steps

  1. Choose your Sitecore version
  2. Choose your search engine provider
  3. Browse to your website folder
  4. Tick your intended role(s) for this particular Sitecore instance
  5. Click the Analyze button

It would then go through each of your configuration files and report if there’s any config files that should be enabled/disabled.

Through the tool you can also quickly disable/enable those config files.

What it doesn’t do

  • Check if your config files is updated accordingly, e.q:
    • Changing robot detection handler in CM
    • Configuring remote reporting service url
    • etc
  • Check the configuration files for WFFM, EXM.. yet
  • Make you coffee

Where can I get it?

The code is available in Github

Note that this tool is not deeply tested, if you have any issues or suggestions with the tool then raise a ticket in Github or do a PR

update: 22 February 2017

The tool is now available at Sitecore Marketplace

Issues encountered with Sitecore Azure Module

I was playing around with Sitecore Azure Module last week and would like to share my experience to get the site up and running.

I am using Sitecore 8.0 version, WFFM and EXM modules. Obviously you would need the Sitecore Azure module to help you setup the content editing and content deliry farm.

  • Sitecore 8.0 rev. 160115 (8.0 Update-7)
  • Sitecore WFFM rev. 151127 (Update-6)
  • Sitecore EXM  3.1 rev. 151213 (3.1 Update-2)
  • Sitecore Azure 8.0 rev. 150522

The target Azure setup would be

  • 1CM and 1CD server in the same data center, in my case I create the farms in the Singapore data center as it’s the closest one for me
  • Sharing the same databases for CM and CD; core, master, web, reporting. In the case of CD it will not have a reference to master and reporting
  • xDB cloud will not be used, instead MongoDB databases will be created using the free 500MB databases from http://mongolab.com

Before proceeding you would need an Azure Subscription in order to be able to create the necessary VM/Cloud Service, if you’re like me and just want to try out the Sitecore Azure Module you can sign up for a free trial for one month.

Installation

In the installing Sitecore Azure Module article above, it also explain about how to deploy to Azure to setup a Editing Farm and Delivery Farm which seems to be easy and straightforward. However I encountered some issues while trying to deploy to Azure.

Issues when deploying

Error saying that the storage account already exist in Azure

Solution:

 

The storage account service name must be unique in Azure across subscriptions, meaning that it has to be unique not only for you and for other subscriptions in the Azure as well. To fix this you can go to /sitecore/system/Modules/Azure/[project_name]/[region_name]/Editing01/Storages/Default and update the Service Name field.

 

Error saying that the database server already exist

Solution:

In the /sitecore/system/Modules/Azure/CampaignManagement-Evaluation/Southeast Asia/Editing01/Sql01 “Server Name” field empty the value. And then clean the unused Azure SQL database server.

 

Error saying that the vmsize does not exist

Solution:

This is because the vm tier name has been changed by Microsoft, which originally was D3 is now has been updated to Standard_D3. To fix this you would need to update the /App_Config/AzureVendors/Microsoft.xml

 

With the free trial subscription you might encounter an issue when the Sitecore Azure Module tries to create the database in SQL Database S2 tier

Solution:

To fix this you can temporarily change the /sitecore/system/Modules/Azure/[project-Name]/[region_name]/Editing01/Sql01/Set01/[database_name] Tier field and change it to “Basic – Basic”. This could also be caused by the database size Sitecore Azure Module requested to be created is not supported by the free trial subscription, I haven’t tested this however if you would like to try it yourself then update the /App_Config/AzureVendors/Microsoft.xml change the

to something smaller, say 100GB or 10GB

 

Another issue when trying to create the Cloud Service

Solution:

This issue seems to be related to the siteStorageSize configuration in /App_Config/AzureVendors/Microsoft.xml. I was trying to create a D3 instance which has the default size of 180GB, reducing the value to 10GB or more seems to do the trick

 

Some reference links that might helps:

exm-error

Issue when installing Sitecore EXM with different index name

Had an issue earlier today when trying to install EXM 3.2.0 rev. 151020 to Sitecore 8.1 rev 151003

Server Error in ‘/’ Application.

Could not find add method: AddIndex (type: Sitecore.ContentSearch.ContentSearchConfiguration)

Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code.

Exception Details: Sitecore.Exceptions.RequiredObjectIsNullException: Could not find add method: AddIndex (type: Sitecore.ContentSearch.ContentSearchConfiguration)

Source Error:

An unhandled exception was generated during the execution of the current web request. Information regarding the origin and location of the exception can be identified using the exception stack trace below.

Stack Trace:

[RequiredObjectIsNullException: Could not find add method: AddIndex (type: Sitecore.ContentSearch.ContentSearchConfiguration)]
Sitecore.Configuration.Factory.AssignProperties(Object obj, Object[] properties) +1920
Sitecore.Configuration.Factory.AssignProperties(XmlNode configNode, String[] parameters, Object obj, Boolean assert, Boolean deferred, IFactoryHelper helper) +643
Sitecore.Configuration.Factory.CreateObject(XmlNode configNode, String[] parameters, Boolean assert, IFactoryHelper helper) +275
Sitecore.Configuration.Factory.CreateObject(String configPath, String[] parameters, Boolean assert) +579
Sitecore.ContentSearch.ContentSearchManager.get_SearchConfiguration() +266
Sitecore.ContentSearch.SolrProvider.SolrContentSearchManager.get_Cores() +92
Sitecore.ContentSearch.SolrProvider.CastleWindsorIntegration.WindsorSolrStartUp.Initialize() +44

[HttpException (0x80004005): Could not find add method: AddIndex (type: Sitecore.ContentSearch.ContentSearchConfiguration)]
System.Web.HttpApplicationFactory.EnsureAppStartCalledForIntegratedMode(HttpContext context, HttpApplication app) +544
System.Web.HttpApplication.RegisterEventSubscriptionsWithIIS(IntPtr appContext, HttpContext context, MethodInfo[] handlers) +186
System.Web.HttpApplication.InitSpecial(HttpApplicationState state, MethodInfo[] handlers, IntPtr appContext, HttpContext context) +172
System.Web.HttpApplicationFactory.GetSpecialApplicationInstance(IntPtr appContext, HttpContext context) +402
System.Web.Hosting.PipelineRuntime.InitializeApplication(IntPtr appContext) +343

[HttpException (0x80004005): Could not find add method: AddIndex (type: Sitecore.ContentSearch.ContentSearchConfiguration)]
System.Web.HttpRuntime.FirstRequestInit(HttpContext context) +579
System.Web.HttpRuntime.EnsureFirstRequestInit(HttpContext context) +112
System.Web.HttpRuntime.ProcessRequestNotificationPrivate(IIS7WorkerRequest wr, HttpContext context) +712

The error message is a bit misleading as it turns out after looking into the Sitecore.EmailExperience.ContentManagement.config file, one of the index was referring to sitecore_suggested_test_index index which doesn’t exist (I’ve renamed it to something else as I was running multiple sitecore instances in my local machine).

There’s also a reference to sitecore_master_index in Sitecore.EmailExperience.Core.config file.

Rename those indexes to an existing indexes then you’ll be fine.

Hope this helps.

Sitecore 8 WFFM MVC – Form Report display no data

We’re using Sitecore 8 update 2 and WFFM 8.0 rev. 150224, the issue that we have is that no matter how often we input the form data and submits it, the form report in Sitecore backend keeps saying that there’s no data to display. Checking in to the analytics database in MongoDB also shows empty FormData collection.

Long story short after contacting Sitecore support, it seems that we need to add the following code

It’s mandatory to put that bit of code in your layout so that Sitecore analytics can work properly. It also reminds me of an issue that I had with a similar fix http://reyrahadian.com/2014/10/28/sitecore-analytics-stops-tracking/

While investigating this issue I also found a couple of things that can help you test your WFFM forms

  • Any submitted form data will be flushed after the session has expired which by default the session timeout is 20 minutes, for quick testing purposes we can change the value which is specified in the web.config to for example 2 minutes

  • In the Sitecore.Forms.Config file which you can find under the App_Config/Include folder, there’s a setting to limit how often can a user submits the form which by default it limit each user to submit 2 form submissions under 1 minute or 100 form submissions under 60 minute. For testing purposes you can for example change the setting to 20 form submissions under 1 minute.

  • If you want to test WFFM in production environment where setting the session to timeout in 2 minutes is not possible, you can clear the browser cookies to force browser session to end SC_ANALYTICS_GLOBAL_COOKIE , SC_ANALYTICS_SESSION_COOKIE and ASP.NET_SessionId. Or you can implement a custom functionality to force browser session end through URL query string such as what has been done here http://www.delphicdigital.com/home/blog.aspx/d=2517/title=web-forms-for-marketers-25-upgrade

$scw(…).eventTracking javascript error in Sitecore 8 MVC WFFM

If you had an issue where you’re using WFFM in Sitecore 8 where it keeps throwing javascript error about $scw(…).eventTracking you’re missing some file references

For Sitecore 7.5 this files would be the following (Taken from the Sitecore Web Forms for Marketers V2.5 Reference section 3.15.2):

 

For Sitecore 8 the reference path is a different, see the following:

 

Sitecore ECM the remote name could not be resolved: ‘example.host’

Got an error when a user submits a form built using the WFFM. After narrowing it down I found that it has something to do with having to set the save action to send an email.

Here’s the following error:

Exception: System.Net.WebException
Message: The remote name could not be resolved: ‘example.host’
Source: System
at System.Net.ServicePoint.GetConnection(PooledStream PooledStream, Object owner, Boolean async, IPAddress& address, Socket& abortSocket, Socket& abortSocket6)
at System.Net.PooledStream.Activate(Object owningObject, Boolean async, GeneralAsyncDelegate asyncCallback)
at System.Net.ConnectionPool.GetConnection(Object owningObject, GeneralAsyncDelegate asyncCallback, Int32 creationTimeout)
at System.Net.Mail.SmtpConnection.GetConnection(ServicePoint servicePoint)
at System.Net.Mail.SmtpClient.Send(MailMessage message)

If you got an error like that, go to /sitecore/system/Modules/Web Forms for Marketers/Settings/Actions/Save Actions/Send Email Message and change the parameters field to the correct SMTP details.

For me I need to changed it to the following to make it works

Also I found a useful tool to check email in your local machine, if you don’t need to setup an actual SMTP server and just need to test simple email functionality try using papercut.codeplex.com

Sitecore analytics stops tracking

Got a weird issue recently when Sitecore analytics stop tracking page visits on one of our client site. The site is running on Sitecore 7.2 rev 140228 and has been running for a few weeks, looking at the ItemUrls and Visits table the records doesn’t seem to be populated when the site pages was accessed a couple of times even with different browsers.

I quickly ran SQL Server Profiler against the analytics database and try to access a couple of the pages again and definitely see some activities against the analytics database but none of it which inserting data to ItemUrls and Visits table. I tried clearing out all the records in the analytics database, all tables except PageEventDefinitions, TrafficTypes, and VisitorClassifications because those tables have default values in it when we setup a Sitecore fresh install.

Still no avail, I then wondered if the config values for the analytics is set correctly. I access the /sitecore/admin/showconfig.aspx and was able to confirm that the configuration was already right. Trying to pinpoint the issue I was wondering what files are included when we install the analytics feature on Sitecore, opening the zip file I found the following:

  • Sitecore.Analytics.config
  • Sitecore.Analytics.ExcludeRobots.config
  • Sitecore.Analytics.RobotDetection.config
  • Sitecore.Analytics.ldf
  • Sitecore.Analytics.mdf

Wondering if the cause is because one of the following config file I then tried to disable the Sitecore.Analytics.RobotDetection.config file by renaming it to Sitecore.Analytics.RobotDetection.config.disabled (rename the extension to anything other than .config so Sitecore ignores it).

Aha!. The analytics start working again.

In the Sitecore.Analytics.RobotDetection.config file this two line pick my interest

It replaces the Robots tracker from the Sitecore.Analytics and use the one from Sitecore.Analytics.RobotDetection dll instead. When I take a look on the Sitecore.Analytics.RobotDetection assemby this is what I found.

Sitecore.Analytics.RobotDetection.Pipelines.AcceptChanges.Robots

Sitecore.Analytics.RobotDetection.Pipelines.InitializeTracker.Robots

On the first visit it will set the visitor classification to 925 which means this visitor is flagged as Bot – Auto Detected which then on the acceptChanges pipeline it will abort the pipeline causing it not to insert any data to the ItemUrls and Visits table.

The issue is on the Sitecore.Analytics.RobotDetection.Pipelines.InitializeTracker.Robots class, it always mark the visitor on first visit as Bot which caused the activity to be ignored by the Sitecore analytics. Compared to what Sitecore.Analytics.Pipelines.AcceptChanges.Robots is doing here

on Sitecore.Analytics.Pipelines.AcceptChanges.Robots it will mark the visitor as Bot on the first visit but then on the second visit it will mark the visitor as Unidentified and log the activitiy to ItemUrl and Visits table.

[update-7-November-2014]

My colleague Cahyadi Hendrawan contacted Sitecore support and confirmed there was a breaking changes in Sitecore 7.2 robot detection logic. The solution was to add

[/update-7-November-2014]