All posts by Rey Rahadian

About Rey Rahadian

This is a personal weblog. The opinions expressed here represent my own and not those of my employer

Sitecore logging integration with Graylog

With Sitecore infrastructure which expands beyond the basic 1 CM and 1 CD, having a centralized logging mechanism is crucial in order to understand what happens in the servers in a consolidated view.

Some of the popular APM out there in the market offers an automatic  records of exception/errors which occurred in the application, this proves really useful when investigating a problem down to the exact line of code which raised the exception. Some types of issues requires you to dig through the Sitecore logs in order to gain an understanding what happened in the server itself , for example when investigating publishing issues, security audits, index crawling, exm logs and fxm logs in the delivery servers.

For those types issues we typically look at different types of Sitecore logs in each servers in order to find out what exactly happened, if we have multiple servers in the picture we would need to go into each servers or pull down the log files from each servers and analyze the log files. This is where centralized logging for Sitecore logs will come in handy as we would have a centralized view of the log files which span multiple servers and we can perform a search query against the log information.

A couple of centralized Log provider in the market are:

  • https://azure.microsoft.com/en-us/services/application-insights/
  • https://www.splunk.com/
  • https://www.elastic.co/products/elasticsearch
  • https://www.graylog.org/

I stumble across Graylog in a recent project where we currently pushing windows event logs to it but not the Sitecore log information yet. So I played around with it a bit to get a feel *rolling sleeves*.

There’s a couple of steps that we need to figure out in order to integrate the Sitecore logs to Graylog

  1. Create a custom log appender
  2. Install Graylog server
  3. Configure Graylog server
  4. Send Sitecore log information to Graylog server

Create a custom log appender

Sitecore uses log4net as the logging framework which is extensible, by default it uses the LogFileAppender class which outputs the log information in text files.

As I want to send the Sitecore log information to Graylog server through http protocol, I would need to create a custom log appender. And as sending log information from Sitecore to Graylog server needs to be in a certain format, GELF format to be precise, we would need to format the log format to match GELF specification in order for Graylog server to understand and able to parse it.

I found the gelf4net library which was mentioned in the Graylog documentation which already did all the heavy lifting of formatting the data. When I installed the nuget package and configured the log4net section in Sitecore according to this library documentation it doesn’t work though.

One gotcha that I found is that when we want to create a custom log appender in Sitecore , we need to reference the AppenderSkeleton class in the Sitecore.Kernel assembly – previously I added log4net nuget package (comes with gelf4net as a dependency) and was hoping that would work, instead it failed miserably 🙁

In the end I created my own appender class which replicate gelf4net appender implementation. https://github.com/reyrahadian/sitecore-gelf-logappender/blob/master/ScGraylog/Appender/GelfHttpAppender.cs 

Install Graylog server

Having read through the Graylog documentation, the easiest and quickest way to setup a Graylog server for my POC is to download the VM. It’s all preconfigured, we just need to load the .voa file using VMware player or VirtualBox to get it up and running.

reference: http://docs.graylog.org/en/latest/pages/getting_started.html

Configure Graylog server

The next things that we need to do after we have our Graylog server up and running is to configure the input source. There’s multiple options that Graylog provides: http, tcp, udp or file dumps.

HTTP input source fits what I need for my POC, so I created a new HTTP input source and have it running.

Send Sitecore log information to Graylog server

Here’s where we put things together. With our custom log appender ready, now we only need to send the Sitecore log information to our Graylog server by using the following log4net configuration

Check if things works as expected

If everything works as expected then you should see some log information coming from Sitecore

 

source code: https://github.com/reyrahadian/sitecore-gelf-logappender

Sitecore Commerce Server Connect.10.0.70.update package installation hangs

I had a strange issue where I previously able to successfully setup a couple of Sitecore Commerce 8.2.1 initial release instances then suddenly creating a new instance keeps failing.

To be exact, it was failing when trying to install Sitecore Commerce Server Connect.10.0.70.update package through the Sitecore update installation wizard page, it just keeps loading indefinitely. Changing the Sitecore log files to DEBUG mode and checking the Windows event log file doesn’t reveal anything. I do notice that memory consumption for the W3WP process is absurdly high in my local machine and it seems like something went wrong.

True enough, after debugging the W3WP process it reveals that the process throws a StackOverflow exception which stems from NewRelic Agent process that I’ve installed yesterday. Uninstalling the NewRelic Agent seems to fix this issue and I’m able to continue with the packages installation per normal.

Extend Sitecore Engagement Plan UI to assign contacts to an engagement plan state

The default Sitecore Engagement Plan currently supports the following option to assign users/contacts to an engagement plan state

  • Add from CSV File
  • Add a Sitecore user
  • Add all Members of a Sitecore Role
  • Add a Segment

Out off all those options only the segment option have the capability of assigning existing contacts to an engagement plan state.

However, what if we would like to import contacts from a CSV file there’s no option to do so, the Add from CSV file will import the data as Sitecore users instead of contacts.

We would need to modify the UI in order to achieve the functionality that we’re aiming for. To do so first we would need to figure which file to modify.

As the UI is using Silverlight it’s a bit difficult to figure which file we need to modify, however searching for the “Welcome to the Import Contacts Wizard” text under the /sitecore  folder reveals that this file seems to be the correct one

Further inspecting the code inside that file and tinkering with it a bit confirms that this is the correct file to modify.

We will not modify the existing Sitecore file, instead we will use the Override folder in order to apply our modification to the existing XML control. To do this copy and paste the existing ImportVisitors.xml file to the override folder

You can put the file directly inside the Override folder, but it’s good practice to replicate the original folder paths so you know which file you’re overriding. After we now know which file to modify we can start modifying the default behaviour of that XML control.

The modified ImportVisitors.xml

 

The code beside file

 

The end result

Source Code is available in GitHub

 

 

Sitecore Server Role Checker Tool

When configuring Sitecore in a distributed environment, you typically have more than 1 server in the production environment configured as a different roles (CM, CD, Processing, Reporting Service, etc).

More often than not I’ve noticed that issues are raised due to misconfiguration rather than implementation itself. You then go to Sitecore documentation site and check the configuration for each of the server that you’ve setup to see if there’s anything that you miss.

In the 8.0 documentation you would need to read a long list of tables containing information which config files that you need to enable/disable.

Since the 8.1 release, these steps are simplified with Sitecore providing us an excel spreadsheet file as a guide to enable/disable the config files depending on the role(s) that you want to setup. These steps are manual though and highly likely that we will miss one or two config files and could cause some issues down the line.

With the number of projects that you need to review, this task will start eating up your time and should really be automated. Some might have already done so and create a little tool tucked away somewhere.

I’ve decided to create my own version of the tool called Sitecore Server Role Checker, can’t be more obvious than that 🙂

How does it work?

This tool would basically uses a converted csv format from the Sitecore official spreadsheet guide and read the configuration based on your selected roles. It currently supports the following Sitecore version

  • 8.1 update 3
  • 8.2 initial release
  • 8.2 update 1
  • 8.2 update 2

Only those version is supported as those are only the spreadsheet available for now.

How do I use it?

Follow this simple steps

  1. Choose your Sitecore version
  2. Choose your search engine provider
  3. Browse to your website folder
  4. Tick your intended role(s) for this particular Sitecore instance
  5. Click the Analyze button

It would then go through each of your configuration files and report if there’s any config files that should be enabled/disabled.

Through the tool you can also quickly disable/enable those config files.

What it doesn’t do

  • Check if your config files is updated accordingly, e.q:
    • Changing robot detection handler in CM
    • Configuring remote reporting service url
    • etc
  • Check the configuration files for WFFM, EXM.. yet
  • Make you coffee

Where can I get it?

The code is available in Github

Note that this tool is not deeply tested, if you have any issues or suggestions with the tool then raise a ticket in Github or do a PR

update: 22 February 2017

The tool is now available at Sitecore Marketplace

Sitecore SOLR index update strategies in a scaled environment

Sitecore by default ships with two main search provider which is Lucene or SOLR.

I will not delve too much on which one you should use as that’s been covered in https://doc.sitecore.net/sitecore_experience_platform/setting_up__maintaining/search_and_indexing/indexing/using_solr_or_lucene

I’d like to highlight some of the points mentioned in the article, where using SOLR is mandatory:

  1. You have a dedicated processing server
  2. You have multiple CM servers

The main thing here is that if sitecore_analytics_index exist in multiple servers then you would need to use SOLR. But since all the indexes are now in a centralized server, what’s the index update strategies for SOLR would be?

Typically in this setup you would set

  • CM as the indexing server
    • will perform index rebuild
    • Set the index update strategies as you fit (except manual)
  • CD only reads from index
    • Does not perform index rebuild
    • Set the index update strategies to manual

If you have multiple CM server then you can set one of the CM as the one that perform index rebuild while the other one only reads from the index.

https://doc.sitecore.net/sitecore_experience_platform/setting_up__maintaining/search_and_indexing/indexing/index_update_strategies

Rebuild sitecore_analytics_index contact data

There’s been an interesting question around how to rebuild the sitecore_analytics_index but specifically only for contact data.

As you’ve probably know that sitecore_analytics_index is not available to be indexed manually – go to the indexing manager in the control panel if you want to check. This is because the index is configured using an observer crawlers.

However there are ways to manually rebuild the sitecore_analytics_index. The example given in that seems to be specifically about rebuilding the interactions documents while what we want is to build the contacts.

I gained some hints from the provided example though, let’s have a look at the following code.

In particular this line

I then go to the handy showconfig.aspx page to find the following

Which shows the processingPools configured for the aggregation processes. From there I can spot the contact element definition which seems to be the one that I’m looking for and start building the code for it.

result:

Depending on your needs, you might want to only reindex known contacts for example. You can take the above example and adjust it yourself.

 

Sitecore 8 – Extending Experience Profile – Part 4 – Displaying Specific Interactions Data

How to display specific interaction data

For this example, we would like to have a custom list in the Experience Profile where it listed the triggered page event when the site visitor downloaded a PDF file.

End result:

end-result-downloadedpdffiles-tab

 

How to set this up

Source code link is available at the end of this post.

How to setup a new page event

First we need to create a page event through the content editor, we can do so by going to the /sitecore/system/Settings/Analytics/Page Events

In the below screenshot I’ve created a page event called “Download PDF File”

create-page-event-downloadpdffile

 

How to assign page event to a media item

There’s a tutorial on how to this on Sitecore documentation site

I’m using a page event rather than goal for this POC because this event “download a pdf” doesn’t necessarily lead to a conversion but more about the actions that the site visitors took that we want to be aware of.

In the screenshot below I’ve assigned the pdf files in the media library with my custom page event.

assign-downloadedpdffile-pageevent-to-mediaitem

Now if there’s a site visitors that download the specified files on the website, Sitecore will trigger the page event and store the information in MongoDB which later on we will extract and display it in the Experience Profile.

How to setup the custom tab in Experience Profile

The content tree

downloadedpdffilespanel-contenttree

We define three column to be displayed on the ListControl rendering.

 

The full list of the renderings

downloadedpdffilespanel-layout

The SubPageCode rendering links to a javascript file downloadedpdffiles.js which will be executed when this item is loaded into the tab.

We can control the page size -the amount of records returned by the server-  in the GenericDataProvider properties.

How to register a new processors for the Experience Profile

Jonathan Robbin’s blog post explain it nicely on what this processors is doing.

Schema.cs

 

ConstructDownloadedPdfFilesDataTable.cs

 

PopulateDownloadedPdfFilesWithXdbData.cs

 

Register our custom processors

TC.Sitecore.ExperienceProfile.Reporting.config

notice that the sorting and paging is already built in the framework.

Register our custom result transformer

TC.Sitecore.ExperienceProfile.Client.config

 

Wire things together

As with the previous post, when the DownloadedPdfFilesPanel item is loaded we will trigger an API call to request the data to be displayed in the list control

The javascript file that’s responsible to do this is the downloadedpdffiles.js

The difference here compared to the previous post when triggering the API call is that we specify the /intel/downloadedpdffiles in the url which Sitecore recognize to be a special route. This route will try to run donwnloadedpdffiles pipelines that we define earlier in the config file.

An example of the API response

http://habitat.dev.local/sitecore/api/ao/v1/contacts/02b2e6c4-d38a-4e30-a18e-b4a31b63907d/intel/downloadedpdffiles?&pageSize=3&pageNumber=1

downloadedpdffiles-webapi-call-request

 

Source code

Source code is available in Github