Category Archives: Sitecore

This category would contains all post related to Sitecore Experience Profile

How TDS Git Delta Deploy make my life easier

In one of recent project that I’ve encountered most of the team members raise the concern about having a such long production deployment time, which could span the entire normal working hours and even more if they encountered some issues.

One of the thing that I noticed can be optimized immediately is the TDS package installation process, where instead of installing the full TDS packages which can contains thousands of items (in my case it was more than 12,000 items) we can instead use the option of only deploying the delta items – which for every release cycle (2 weeks) normally amount to around 50 items or even less.

The TDS Git Delta Deploy itself is something that Sitecore MVP Sean Holmesby created to answer how to have a true “delta” of content items to be deployed to the target environment. Have a read of the full article here.

So, I know the solution of one the paint point area then I just need to install Sean’s nuget package and go on my merry way right? well.. not exactly.

The problem

In the current solution the team have multiple TDS projects per features (Helix anyone?), those TDS projects would then be bundled using TDS package bundling to create the final combined package.

To give a little bit of illustration, see below:

Given the following TDS projects

  • TDS Foundation Z
  • TDS Feature A
  • TDS Feature B
  • TDS Feature C
  • TDS Project X  (will bundle all content items from the above TDS projects)

So what’s the problem? the package bundling doesn’t respect the TDS Git Delta Deploy it will always include all the items instead of just the delta.

The solution

I reached out to the nice folks in the #tds slack channel in https://sitecorechat.slack.com/ for suggestions – btw if you haven’t joined then I suggest you do that first else you’re missing out 🙂

Not long after I raised the question, John Rappel replied that he has a fix for that in the latest nuget package – awesome!. I upgraded the nuget package and true enough that it now works. Read more about John’s bugfixes and feature updates here.

So now I got TDS delta packages working locally, it’s now time to integrate it in our Continuous Build process.

The setup in Team City

TDS Git Delta Deploy work by using “git diff” command in it’s core, which for this to work correctly you would need to ensure that:

  • TC is configured to use the agent side checkout
  • TC is doing a full “git fetch” instead of just retrieving a single branch
    • If you’re using TC version 10.0.4 and above you’re in luck as apparently you can set it using the following approach 
    • If you’re using TC version prior 10.0.4 then your other option to have this is to do a custom build step that perform a git fetch

A quick POC confirm that the approach is solid and I see those delta packages being generated!

TDS Delta packages generated.. what now?

With the TDS delta packages generated and ready to use, all we have to do now is to include it as part of our Continuous Deployment process. We’re using Octopus Deploy so it’s quite easy to setup.

You can use something like Sitecore Package Deployer to automate the package installation process and include it as part of Octopus deploy step. In our case, since we already have an existing custom web service that does the same job so we leverage that instead.

quick tip:

Enable TDS “publish package after deploy” to streamline the deployment process by automatically publishing only the deployed items

here’s the reference on how to do just that https://hedgehogdevelopment.github.io/tds/chapter4.html

What’s the outcome?

Having all this in place really helped streamlined the way we do a Sitecore deployment and cut massive time which in turns faster deployment => more time to work on something else => productivity++

Sitecore logging integration with Graylog

With Sitecore infrastructure which expands beyond the basic 1 CM and 1 CD, having a centralized logging mechanism is crucial in order to understand what happens in the servers in a consolidated view.

Some of the popular APM out there in the market offers an automatic  records of exception/errors which occurred in the application, this proves really useful when investigating a problem down to the exact line of code which raised the exception. Some types of issues requires you to dig through the Sitecore logs in order to gain an understanding what happened in the server itself , for example when investigating publishing issues, security audits, index crawling, exm logs and fxm logs in the delivery servers.

For those types issues we typically look at different types of Sitecore logs in each servers in order to find out what exactly happened, if we have multiple servers in the picture we would need to go into each servers or pull down the log files from each servers and analyze the log files. This is where centralized logging for Sitecore logs will come in handy as we would have a centralized view of the log files which span multiple servers and we can perform a search query against the log information.

A couple of centralized Log provider in the market are:

  • https://azure.microsoft.com/en-us/services/application-insights/
  • https://www.splunk.com/
  • https://www.elastic.co/products/elasticsearch
  • https://www.graylog.org/

I stumble across Graylog in a recent project where we currently pushing windows event logs to it but not the Sitecore log information yet. So I played around with it a bit to get a feel *rolling sleeves*.

There’s a couple of steps that we need to figure out in order to integrate the Sitecore logs to Graylog

  1. Create a custom log appender
  2. Install Graylog server
  3. Configure Graylog server
  4. Send Sitecore log information to Graylog server

Create a custom log appender

Sitecore uses log4net as the logging framework which is extensible, by default it uses the LogFileAppender class which outputs the log information in text files.

As I want to send the Sitecore log information to Graylog server through http protocol, I would need to create a custom log appender. And as sending log information from Sitecore to Graylog server needs to be in a certain format, GELF format to be precise, we would need to format the log format to match GELF specification in order for Graylog server to understand and able to parse it.

I found the gelf4net library which was mentioned in the Graylog documentation which already did all the heavy lifting of formatting the data. When I installed the nuget package and configured the log4net section in Sitecore according to this library documentation it doesn’t work though.

One gotcha that I found is that when we want to create a custom log appender in Sitecore , we need to reference the AppenderSkeleton class in the Sitecore.Logging assembly – previously I added log4net nuget package (comes with gelf4net as a dependency) and was hoping that would work, instead it failed miserably 🙁

In the end I created my own appender class which replicate gelf4net appender implementation. https://github.com/reyrahadian/sitecore-gelf-logappender/blob/master/ScGraylog/Appender/GelfHttpAppender.cs 

Install Graylog server

Having read through the Graylog documentation, the easiest and quickest way to setup a Graylog server for my POC is to download the VM. It’s all preconfigured, we just need to load the .voa file using VMware player or VirtualBox to get it up and running.

reference: http://docs.graylog.org/en/latest/pages/getting_started.html

Configure Graylog server

The next things that we need to do after we have our Graylog server up and running is to configure the input source. There’s multiple options that Graylog provides: http, tcp, udp or file dumps.

HTTP input source fits what I need for my POC, so I created a new HTTP input source and have it running.

Send Sitecore log information to Graylog server

Here’s where we put things together. With our custom log appender ready, now we only need to send the Sitecore log information to our Graylog server by using the following log4net configuration

Check if things works as expected

If everything works as expected then you should see some log information coming from Sitecore

 

source code: https://github.com/reyrahadian/sitecore-gelf-logappender

Sitecore Commerce Server Connect.10.0.70.update package installation hangs

I had a strange issue where I previously able to successfully setup a couple of Sitecore Commerce 8.2.1 initial release instances then suddenly creating a new instance keeps failing.

To be exact, it was failing when trying to install Sitecore Commerce Server Connect.10.0.70.update package through the Sitecore update installation wizard page, it just keeps loading indefinitely. Changing the Sitecore log files to DEBUG mode and checking the Windows event log file doesn’t reveal anything. I do notice that memory consumption for the W3WP process is absurdly high in my local machine and it seems like something went wrong.

True enough, after debugging the W3WP process it reveals that the process throws a StackOverflow exception which stems from NewRelic Agent process that I’ve installed yesterday. Uninstalling the NewRelic Agent seems to fix this issue and I’m able to continue with the packages installation per normal.

Extend Sitecore Engagement Plan UI to assign contacts to an engagement plan state

The default Sitecore Engagement Plan currently supports the following option to assign users/contacts to an engagement plan state

  • Add from CSV File
  • Add a Sitecore user
  • Add all Members of a Sitecore Role
  • Add a Segment

Out off all those options only the segment option have the capability of assigning existing contacts to an engagement plan state.

However, what if we would like to import contacts from a CSV file there’s no option to do so, the Add from CSV file will import the data as Sitecore users instead of contacts.

We would need to modify the UI in order to achieve the functionality that we’re aiming for. To do so first we would need to figure which file to modify.

As the UI is using Silverlight it’s a bit difficult to figure which file we need to modify, however searching for the “Welcome to the Import Contacts Wizard” text under the /sitecore  folder reveals that this file seems to be the correct one

Further inspecting the code inside that file and tinkering with it a bit confirms that this is the correct file to modify.

We will not modify the existing Sitecore file, instead we will use the Override folder in order to apply our modification to the existing XML control. To do this copy and paste the existing ImportVisitors.xml file to the override folder

You can put the file directly inside the Override folder, but it’s good practice to replicate the original folder paths so you know which file you’re overriding. After we now know which file to modify we can start modifying the default behaviour of that XML control.

The modified ImportVisitors.xml

 

The code beside file

 

The end result

Source Code is available in GitHub

 

 

Sitecore Server Role Checker Tool

When configuring Sitecore in a distributed environment, you typically have more than 1 server in the production environment configured as a different roles (CM, CD, Processing, Reporting Service, etc).

More often than not I’ve noticed that issues are raised due to misconfiguration rather than implementation itself. You then go to Sitecore documentation site and check the configuration for each of the server that you’ve setup to see if there’s anything that you miss.

In the 8.0 documentation you would need to read a long list of tables containing information which config files that you need to enable/disable.

Since the 8.1 release, these steps are simplified with Sitecore providing us an excel spreadsheet file as a guide to enable/disable the config files depending on the role(s) that you want to setup. These steps are manual though and highly likely that we will miss one or two config files and could cause some issues down the line.

With the number of projects that you need to review, this task will start eating up your time and should really be automated. Some might have already done so and create a little tool tucked away somewhere.

I’ve decided to create my own version of the tool called Sitecore Server Role Checker, can’t be more obvious than that 🙂

How does it work?

This tool would basically uses a converted csv format from the Sitecore official spreadsheet guide and read the configuration based on your selected roles. It currently supports the following Sitecore version

  • 8.1 update 3
  • 8.2 initial release
  • 8.2 update 1
  • 8.2 update 2

Only those version is supported as those are only the spreadsheet available for now.

How do I use it?

Follow this simple steps

  1. Choose your Sitecore version
  2. Choose your search engine provider
  3. Browse to your website folder
  4. Tick your intended role(s) for this particular Sitecore instance
  5. Click the Analyze button

It would then go through each of your configuration files and report if there’s any config files that should be enabled/disabled.

Through the tool you can also quickly disable/enable those config files.

What it doesn’t do

  • Check if your config files is updated accordingly, e.q:
    • Changing robot detection handler in CM
    • Configuring remote reporting service url
    • etc
  • Check the configuration files for WFFM, EXM.. yet
  • Make you coffee

Where can I get it?

The code is available in Github

Note that this tool is not deeply tested, if you have any issues or suggestions with the tool then raise a ticket in Github or do a PR

update: 22 February 2017

The tool is now available at Sitecore Marketplace

Sitecore SOLR index update strategies in a scaled environment

Sitecore by default ships with two main search provider which is Lucene or SOLR.

I will not delve too much on which one you should use as that’s been covered in https://doc.sitecore.net/sitecore_experience_platform/setting_up__maintaining/search_and_indexing/indexing/using_solr_or_lucene

I’d like to highlight some of the points mentioned in the article, where using SOLR is mandatory:

  1. You have a dedicated processing server
  2. You have multiple CM servers

The main thing here is that if sitecore_analytics_index exist in multiple servers then you would need to use SOLR. But since all the indexes are now in a centralized server, what’s the index update strategies for SOLR would be?

Typically in this setup you would set

  • CM as the indexing server
    • will perform index rebuild
    • Set the index update strategies as you fit (except manual)
  • CD only reads from index
    • Does not perform index rebuild
    • Set the index update strategies to manual

If you have multiple CM server then you can set one of the CM as the one that perform index rebuild while the other one only reads from the index.

https://doc.sitecore.net/sitecore_experience_platform/setting_up__maintaining/search_and_indexing/indexing/index_update_strategies

Rebuild sitecore_analytics_index contact data

There’s been an interesting question around how to rebuild the sitecore_analytics_index but specifically only for contact data.

As you’ve probably know that sitecore_analytics_index is not available to be indexed manually – go to the indexing manager in the control panel if you want to check. This is because the index is configured using an observer crawlers.

However there are ways to manually rebuild the sitecore_analytics_index. The example given in that seems to be specifically about rebuilding the interactions documents while what we want is to build the contacts.

I gained some hints from the provided example though, let’s have a look at the following code.

In particular this line

I then go to the handy showconfig.aspx page to find the following

Which shows the processingPools configured for the aggregation processes. From there I can spot the contact element definition which seems to be the one that I’m looking for and start building the code for it.

result:

Depending on your needs, you might want to only reindex known contacts for example. You can take the above example and adjust it yourself.

 

Sitecore 8 – Extending Experience Profile – Part 4 – Displaying Specific Interactions Data

How to display specific interaction data

For this example, we would like to have a custom list in the Experience Profile where it listed the triggered page event when the site visitor downloaded a PDF file.

End result:

end-result-downloadedpdffiles-tab

 

How to set this up

Source code link is available at the end of this post.

How to setup a new page event

First we need to create a page event through the content editor, we can do so by going to the /sitecore/system/Settings/Analytics/Page Events

In the below screenshot I’ve created a page event called “Download PDF File”

create-page-event-downloadpdffile

 

How to assign page event to a media item

There’s a tutorial on how to this on Sitecore documentation site

I’m using a page event rather than goal for this POC because this event “download a pdf” doesn’t necessarily lead to a conversion but more about the actions that the site visitors took that we want to be aware of.

In the screenshot below I’ve assigned the pdf files in the media library with my custom page event.

assign-downloadedpdffile-pageevent-to-mediaitem

Now if there’s a site visitors that download the specified files on the website, Sitecore will trigger the page event and store the information in MongoDB which later on we will extract and display it in the Experience Profile.

How to setup the custom tab in Experience Profile

The content tree

downloadedpdffilespanel-contenttree

We define three column to be displayed on the ListControl rendering.

 

The full list of the renderings

downloadedpdffilespanel-layout

The SubPageCode rendering links to a javascript file downloadedpdffiles.js which will be executed when this item is loaded into the tab.

We can control the page size -the amount of records returned by the server-  in the GenericDataProvider properties.

How to register a new processors for the Experience Profile

Jonathan Robbin’s blog post explain it nicely on what this processors is doing.

Schema.cs

 

ConstructDownloadedPdfFilesDataTable.cs

 

PopulateDownloadedPdfFilesWithXdbData.cs

 

Register our custom processors

TC.Sitecore.ExperienceProfile.Reporting.config

notice that the sorting and paging is already built in the framework.

Register our custom result transformer

TC.Sitecore.ExperienceProfile.Client.config

 

Wire things together

As with the previous post, when the DownloadedPdfFilesPanel item is loaded we will trigger an API call to request the data to be displayed in the list control

The javascript file that’s responsible to do this is the downloadedpdffiles.js

The difference here compared to the previous post when triggering the API call is that we specify the /intel/downloadedpdffiles in the url which Sitecore recognize to be a special route. This route will try to run donwnloadedpdffiles pipelines that we define earlier in the config file.

An example of the API response

http://habitat.dev.local/sitecore/api/ao/v1/contacts/02b2e6c4-d38a-4e30-a18e-b4a31b63907d/intel/downloadedpdffiles?&pageSize=3&pageNumber=1

downloadedpdffiles-webapi-call-request

 

Source code

Source code is available in Github

Sitecore 8 – Extending Experience Profile – Part 3 – Displaying Contact Facet Collection Property

How to display contact facet collection property

End result:

end-result-newsletter-tab

How to set this up

Source code link is available at the end of this post.

How to setup the custom contact facet

Similar to what we’ve done in the first post, the only difference here is that we are using collection property instead of simple properties.

 

The  updated TC.Sitecore.Analytics.Model.config file

note that we need to register our INewsletterElement as well in order for Sitecore to know which type to resolve during runtime.

An example of how the data is being stored in MongoDB

newslettersubscriptions-mongodb

 

How to setup the web API controller

The updated InitializeRoutes.cs

 

The updated CustomContactController.cs

 

The main reason why we are returning as a ResultSet object here is that the list control rendering expects the returned data in a specific JSON structure.

While going through this I thought of the performance implications if we have a lot of data where we should ideally implement pagination when querying against MongoDB. For this POC, this works fine as for newsletters typically the users won’t have thousands of records to be displayed .

An example on how to implement pagination will be provided in the next post.

 

How to setup a custom tab in Experience Profile

The content tree

newslettersubscriptionpanel-contenttree

Under the NewsletterSubscriptionPanel item you can that there’s NewsletterSubscriptions which based on ListControl template, under that we have the NewsletterName which is using the ColumnField template.

When we use the list control rendering, we can specify what columns are there in the list control and bind the value with a specific property name returned from the API

newslettername-columnfield

 

The full list of the renderings used

newslettersubscriptionpanel-layout

note that there’s “more” button assigned to the layout, but this didn’t actually being used. That’s for future improvements when we implement pagination.

 

Wire things together

As with the previous post, when the NewsletterSubscriptionPanel item is loaded we will trigger an API call to request the data to be displayed in the list control

The javascript file that’s responsible to do this is the newslettersubscriptions.js

 

An example of the API response

http://habitat.dev.local/sitecore/api/ao/v1/contacts/02b2e6c4-d38a-4e30-a18e-b4a31b63907d/newslettersubscriptions/?&pageNumber=1

newslettersubscriptions-webapi-call-request

note that it has to return the response in a specific JSON structure that the list control can consume.

 

Source code

Source code is available in Github

Sitecore 8 – Extending Experience Profile – Part 2 – Displaying Contact Facet Simple Properties

How to display contact facet simple properties

End result:

end-result

There’s a good reference on how to create a custom tab in Sitecore Experience Profile on Jonathan Robbins blog post,  I recommend you go and read it through.

How to set this up

Source code link is available at the end of this post.

How to setup the custom contact facet

There’s a good reference on Sitecore documentation site on how to do so.

In my custom facet I only have two properties, which is the HospitalName and the ProfessionName.

ICustomFieldsFacet.cs

 

CustomFieldsFacet.cs

 

And the config patch file to register this custom facet

TC.Sitecore.Analytics.Model.config

 

To fillout the data in the custom facet I’ve setup a page to do this

updatecontact-page

 

How to setup the web API controller

First we need to register our custom route to our custom controller

InitializeRoutes.cs

 

In our custom controller, we just return the custom facet that contains all the properties that we want to display in Experience Profile

CustomContactController.cs

 

Finally we add the InitializeRoutes class to the pipeline

TC.Sitecore.ExperienceProfile.config

 

How to setup a custom tab in Experience Profile

I will not go in detail on how I setup the content tree as the reference I’ve provided should give you an idea on how to set it up and I’ll be providing the source code at the end of this post.

This is the content tree in the core database that I’ve setup in order to display those 3 tabs

professionpanel-contenttree

The first tab which is the Profession tab will be displaying the custom field from the custom contact facet that I’ve setup.

The rendering items

 

Wire things together

To display this information, after our Profession tab is loaded in the Experience Profile we would need to call our CustomContactController class to retrieve the data.

The code that’s responsible to do this is the profession.js that is set in the SubPageCode rendering control in the ProfessionPanel layout. This javascript file is executed when the ProfessionPanel item is loaded into the Profession tab

professionpanel-layout-subpagecode

profession.js

 

The script basically calls an API method in the server which then gives back the response in a JSON format which we then assign to the controls defined in the ProfessionPanel item layout.

customfields-webapi-call-request

This is the full url it’s trying to access http://habitat.dev.local/sitecore/api/ao/v1/contacts/02b2e6c4-d38a-4e30-a18e-b4a31b63907d/customfields/

It’s passing the current contact id and a specific url ‘customfields’ which we route to our custom web API controller. It will then use the cintelUtil helper to bind the data to the text controls defined in our ProfessionPanel layout.

 

Source code

Source code is available in Github