Sitecore SOLR index update strategies in a scaled environment

Sitecore by default ships with two main search provider which is Lucene or SOLR.

I will not delve too much on which one you should use as that’s been covered in https://doc.sitecore.net/sitecore_experience_platform/setting_up__maintaining/search_and_indexing/indexing/using_solr_or_lucene

I’d like to highlight some of the points mentioned in the article, where using SOLR is mandatory:

  1. You have a dedicated processing server
  2. You have multiple CM servers

The main thing here is that if sitecore_analytics_index exist in multiple servers then you would need to use SOLR. But since all the indexes are now in a centralized server, what’s the index update strategies for SOLR would be?

Typically in this setup you would set

  • CM as the indexing server
    • will perform index rebuild
    • Set the index update strategies as you fit (except manual)
  • CD only reads from index
    • Does not perform index rebuild
    • Set the index update strategies to manual

If you have multiple CM server then you can set one of the CM as the one that perform index rebuild while the other one only reads from the index.

https://doc.sitecore.net/sitecore_experience_platform/setting_up__maintaining/search_and_indexing/indexing/index_update_strategies

Rebuild sitecore_analytics_index contact data

There’s been an interesting question around how to rebuild the sitecore_analytics_index but specifically only for contact data.

As you’ve probably know that sitecore_analytics_index is not available to be indexed manually – go to the indexing manager in the control panel if you want to check. This is because the index is configured using an observer crawlers.

However there are ways to manually rebuild the sitecore_analytics_index. The example given in that seems to be specifically about rebuilding the interactions documents while what we want is to build the contacts.

I gained some hints from the provided example though, let’s have a look at the following code.

In particular this line

I then go to the handy showconfig.aspx page to find the following

Which shows the processingPools configured for the aggregation processes. From there I can spot the contact element definition which seems to be the one that I’m looking for and start building the code for it.

result:

Depending on your needs, you might want to only reindex known contacts for example. You can take the above example and adjust it yourself.

 

Sitecore 8 – Extending Experience Profile – Part 4 – Displaying Specific Interactions Data

How to display specific interaction data

For this example, we would like to have a custom list in the Experience Profile where it listed the triggered page event when the site visitor downloaded a PDF file.

End result:

end-result-downloadedpdffiles-tab

 

How to set this up

Source code link is available at the end of this post.

How to setup a new page event

First we need to create a page event through the content editor, we can do so by going to the /sitecore/system/Settings/Analytics/Page Events

In the below screenshot I’ve created a page event called “Download PDF File”

create-page-event-downloadpdffile

 

How to assign page event to a media item

There’s a tutorial on how to this on Sitecore documentation site

I’m using a page event rather than goal for this POC because this event “download a pdf” doesn’t necessarily lead to a conversion but more about the actions that the site visitors took that we want to be aware of.

In the screenshot below I’ve assigned the pdf files in the media library with my custom page event.

assign-downloadedpdffile-pageevent-to-mediaitem

Now if there’s a site visitors that download the specified files on the website, Sitecore will trigger the page event and store the information in MongoDB which later on we will extract and display it in the Experience Profile.

How to setup the custom tab in Experience Profile

The content tree

downloadedpdffilespanel-contenttree

We define three column to be displayed on the ListControl rendering.

 

The full list of the renderings

downloadedpdffilespanel-layout

The SubPageCode rendering links to a javascript file downloadedpdffiles.js which will be executed when this item is loaded into the tab.

We can control the page size -the amount of records returned by the server-  in the GenericDataProvider properties.

How to register a new processors for the Experience Profile

Jonathan Robbin’s blog post explain it nicely on what this processors is doing.

Schema.cs

 

ConstructDownloadedPdfFilesDataTable.cs

 

PopulateDownloadedPdfFilesWithXdbData.cs

 

Register our custom processors

TC.Sitecore.ExperienceProfile.Reporting.config

notice that the sorting and paging is already built in the framework.

Register our custom result transformer

TC.Sitecore.ExperienceProfile.Client.config

 

Wire things together

As with the previous post, when the DownloadedPdfFilesPanel item is loaded we will trigger an API call to request the data to be displayed in the list control

The javascript file that’s responsible to do this is the downloadedpdffiles.js

The difference here compared to the previous post when triggering the API call is that we specify the /intel/downloadedpdffiles in the url which Sitecore recognize to be a special route. This route will try to run donwnloadedpdffiles pipelines that we define earlier in the config file.

An example of the API response

http://habitat.dev.local/sitecore/api/ao/v1/contacts/02b2e6c4-d38a-4e30-a18e-b4a31b63907d/intel/downloadedpdffiles?&pageSize=3&pageNumber=1

downloadedpdffiles-webapi-call-request

 

Source code

Source code is available in Github

Sitecore 8 – Extending Experience Profile – Part 3 – Displaying Contact Facet Collection Property

How to display contact facet collection property

End result:

end-result-newsletter-tab

How to set this up

Source code link is available at the end of this post.

How to setup the custom contact facet

Similar to what we’ve done in the first post, the only difference here is that we are using collection property instead of simple properties.

 

The  updated TC.Sitecore.Analytics.Model.config file

note that we need to register our INewsletterElement as well in order for Sitecore to know which type to resolve during runtime.

An example of how the data is being stored in MongoDB

newslettersubscriptions-mongodb

 

How to setup the web API controller

The updated InitializeRoutes.cs

 

The updated CustomContactController.cs

 

The main reason why we are returning as a ResultSet object here is that the list control rendering expects the returned data in a specific JSON structure.

While going through this I thought of the performance implications if we have a lot of data where we should ideally implement pagination when querying against MongoDB. For this POC, this works fine as for newsletters typically the users won’t have thousands of records to be displayed .

An example on how to implement pagination will be provided in the next post.

 

How to setup a custom tab in Experience Profile

The content tree

newslettersubscriptionpanel-contenttree

Under the NewsletterSubscriptionPanel item you can that there’s NewsletterSubscriptions which based on ListControl template, under that we have the NewsletterName which is using the ColumnField template.

When we use the list control rendering, we can specify what columns are there in the list control and bind the value with a specific property name returned from the API

newslettername-columnfield

 

The full list of the renderings used

newslettersubscriptionpanel-layout

note that there’s “more” button assigned to the layout, but this didn’t actually being used. That’s for future improvements when we implement pagination.

 

Wire things together

As with the previous post, when the NewsletterSubscriptionPanel item is loaded we will trigger an API call to request the data to be displayed in the list control

The javascript file that’s responsible to do this is the newslettersubscriptions.js

 

An example of the API response

http://habitat.dev.local/sitecore/api/ao/v1/contacts/02b2e6c4-d38a-4e30-a18e-b4a31b63907d/newslettersubscriptions/?&pageNumber=1

newslettersubscriptions-webapi-call-request

note that it has to return the response in a specific JSON structure that the list control can consume.

 

Source code

Source code is available in Github

Sitecore 8 – Extending Experience Profile – Part 2 – Displaying Contact Facet Simple Properties

How to display contact facet simple properties

End result:

end-result

There’s a good reference on how to create a custom tab in Sitecore Experience Profile on Jonathan Robbins blog post,  I recommend you go and read it through.

How to set this up

Source code link is available at the end of this post.

How to setup the custom contact facet

There’s a good reference on Sitecore documentation site on how to do so.

In my custom facet I only have two properties, which is the HospitalName and the ProfessionName.

ICustomFieldsFacet.cs

 

CustomFieldsFacet.cs

 

And the config patch file to register this custom facet

TC.Sitecore.Analytics.Model.config

 

To fillout the data in the custom facet I’ve setup a page to do this

updatecontact-page

 

How to setup the web API controller

First we need to register our custom route to our custom controller

InitializeRoutes.cs

 

In our custom controller, we just return the custom facet that contains all the properties that we want to display in Experience Profile

CustomContactController.cs

 

Finally we add the InitializeRoutes class to the pipeline

TC.Sitecore.ExperienceProfile.config

 

How to setup a custom tab in Experience Profile

I will not go in detail on how I setup the content tree as the reference I’ve provided should give you an idea on how to set it up and I’ll be providing the source code at the end of this post.

This is the content tree in the core database that I’ve setup in order to display those 3 tabs

professionpanel-contenttree

The first tab which is the Profession tab will be displaying the custom field from the custom contact facet that I’ve setup.

The rendering items

 

Wire things together

To display this information, after our Profession tab is loaded in the Experience Profile we would need to call our CustomContactController class to retrieve the data.

The code that’s responsible to do this is the profession.js that is set in the SubPageCode rendering control in the ProfessionPanel layout. This javascript file is executed when the ProfessionPanel item is loaded into the Profession tab

professionpanel-layout-subpagecode

profession.js

 

The script basically calls an API method in the server which then gives back the response in a JSON format which we then assign to the controls defined in the ProfessionPanel item layout.

customfields-webapi-call-request

This is the full url it’s trying to access http://habitat.dev.local/sitecore/api/ao/v1/contacts/02b2e6c4-d38a-4e30-a18e-b4a31b63907d/customfields/

It’s passing the current contact id and a specific url ‘customfields’ which we route to our custom web API controller. It will then use the cintelUtil helper to bind the data to the text controls defined in our ProfessionPanel layout.

 

Source code

Source code is available in Github

Sitecore 8 – Extending Experience Profile – Part 1 – Overview

I will be creating a series of blog posts on a recent POC that I’ve done on how to extend Sitecore 8 Experience Profile

This was done using Sitecore 8.1 update 1 at the time of writing.

There will be several things that this blog posts tries to cover

  1. How to display contact facet simple properties
  2. How to display contact facet collection property to a list control
  3. How to display specific interaction data to a list control

End results

Displaying contact facet simple properties

The below screenshot is an example on how to retrieve simple properties from the contact facet and then display them in our custom tab

end-result

 

Displaying contact facet collection property to a list control

The below screenshot is an example on how to retrieve collection property from the contact facet and then bind it to a list control

end-result-newsletter-tab

 

Displaying specific interaction data to a list control

The below screenshot is an example on how to retrieve specific interaction data, which in this case a Page Event that’s triggered whenever a site visitor downloaded a pdf file.

end-result-downloadedpdffiles-tab

 

Details on how to achieve this will be explained the next posts.

 

Sitecore 8 – how to update contact through FXM API

This post will talk through how to setup a custom actions to be executed to update Sitecore contact through FXM API. If you don’t know about FXM you can read a brief overview about it in the Sitecore site and the doc site

If you’ve played around with FXM and it’s javascript API then you would know that you can trigger client side call to send information back to Sitecore regarding the current visitor browsing behaviour. But what does one do if we want to update the Contact information instead? say the current visitor email address if they fill out a newsletter subscription form? or their unique user id if they have logged in through the external application?

Looking at the available javascript FXM API there’s these three methods available:

  1. SCBeacon.trackEvent
  2. SCBeacon.trackGoal
  3. SCBeacon.trackOutcome

and there’s no such things as SCBeacon.updateContact which we sought after. After some investigation there’s a couple of ways to do this.

1. A custom web service call to the Sitecore server

Setup a custom web service in Sitecore server and pass in the current visitor contact id for custom processing. Doable but I’m looking for something that can be done through the same javascript API if possible as it would mean greater coverage – can be reused by other external site. By doing it through FXM javascript API we don’t have to setup web service call in each external sites.

2. Modify the beacon script to include our custom function 

Modify the beacon script and include the custom SCBeacon.updateContact function. Figured that this won’t be ideal as we would muck around with the bundled javascript and won’t be compatible in Sitecore future upgrades.

3. Hook into one of the available methods to do some custom processing

This seems like the best place to start. Out of the available methods I choose to override the trackEvent tracking implementation because the action that I want to captured falls to “system” type action, doesn’t relate to conversion (Goal) or has monetary value (Outcome). Plus if I use page events then by default the events triggered will not show in the Experience Profile which might confused the marketers.

OK, let’s do this.

This is done with the latest Sitecore version at the time of writing which is 8.1 update 3.

First let’s open the Sitecore.FXM.config in /website/app_config/include/fxm folder

I opened up Sitecore.FXM.Pipelines.Tracking.TriggerPageEvent.RunRegisterPageEventProcessor using dotPeak to find out that it will run the tracking.registerpageevent pipeline

When I take a look at Sitecore.FXM.Pipelines.Tracking.RegisterPageEvent.RegisterPageEventProcessor and what it does it seems this would be the best place to put my custom logic

We can override the TriggerPageEvent method and implement our custom processing.

The next step is for us to create our custom Processor class which extends from Sitecore’s default RegisterPageEventProcessor

After that we need to register our custom processor class in the config file. For best practice use Sitecore config file patching to do.

In our custom config file we override the default processor class to process the triggered page events with our custom processor class. Put the file in /website/app_config/include/zzz.POC/ folder

Now that we have our custom processor class and have registered it through the custom config file, we need to create the actual page event items in Sitecore.

Here’s what the created page event items looks like in my setup.

registered-custom-page-events

And now to test it out, browse to the external site  and open the developer console to run these javascripts

We can check the end result in MongoDB after the session has timeout

updated-contact-through-FXM

And in the Experience Profile

updated-contact-through-FXM-xFile

That’s it folks.

 

Sitecore 8 – Create a custom personalization rule

Sitecore OOTB comes with tons of personalization rules that we can use, however there might be times where we need to create a custom rule condition that fits to the client business requirements.

Luckily with Sitecore framework that’s known easy to extend, we can easily achieve this.

Here’s the steps to create a custom personalization rule:

1.Register the tag

First of all we need to create a new tag item under /sitecore/system/Settings/Rules/Definitions/Tags

create-custom-personalization-rule-create-tag

2. Register a new rule element

Create a new element folder in /sitecore/system/Settings/Rules/Definitions/Elements

create-custom-personalization-rule-create-element-folder

3. Create a new personalization condition rule

create-custom-personalization-rule-create-personalization-rule

Here I’m creating a new rule and calling it “Specific Template Name” which will do what exactly that, checking if the current context item is based on a specific template name – not really useful in real world scenario but this post is about how to setup a custom personalization rule 🙂

In the newly created rule we need to set the Text and Type fields

Text field contains the text that we’re going to present to the content author, here’s the following text that I use

the [TemplateName, StringOperator,,compares to] represents the format of input that we want to get from the content author. It follows the following format

  • OperatorId, defines the public property of the class where we want to assign the value coming from the content author input
  • StringOperator, the built-in macro that we want to use to get the input from the user. In this case this will be a string comparation operation
  • blank, this parameter will depend on the type of macro that we use, this could be a default text value if we are using the default macro,  it could be a default start path if we’re using the Tree macro  – think of it like setting a field source when we’re building a template in Sitecore. A full list of macros can be found in /sitecore/system/Settings/Rules/Definitions/Macros
  • compares to, the last parameter is the text representation that we want to show to the content author, this value is clickable and when clicked Sitecore will display the appropriate input control based on the macro that we set on the second parameter

The Type field is the full assembly name of the custom class that we want to use to perform the logic behind this personalization rule

4. Assign our custom element default tag definition to our custom tag

create-custom-personalization-rule-assign-element-tag-definition-to-tag

Of course we  can assign our custom element tag definition to one of the existing tag under /sitecore/system/Settings/Rules/Definitions/Tags so that it will automatically appear on the content author personalization rule window however if you want to make it obvious which one is your custom ones then I recommend creating your own custom tag and assign it to that

5. Assign the tag to Conditional Rendering

The last step to make the rule visible for the content author to choose from is to assign our custom tag to one of default Rules Context folder

create-custom-personalization-rule-assign-tag

From the picture we’re setting the tag to the Conditional Rendering rules context which will appear when the user want to personalize a certain component, but you can also some other Rules Context folders such as FXM ones.

After we assign the tag, we can verify that the custom personalization rule is available for the content author to choose from

create-custom-personalization-rule-verifying-the-custom-rule-appears

Here’s the class that’s responsible to evaluate the custom rule condition

The class logic is simple and not doing much for the purposes of the tutorial. In a real world scenario you can make a rule that reads from a custom database, read a configuration file, call an external API (be careful with this as it will increase page load time).

This may sound like a lot of work at first compared to just creating a custom code in the rendering or similar approach. But when we consider that we’re enabling the content author/marketers to do it by themselves and removing/minimizing the dependency towards IT, it would lessen time to market and open up more possibilities for the marketers of what they can do using the platform.

 

Sitecore 8 – set facebook profile picture in Experience Profile

This post will talk about how to retrieve the Facebook profile picture using Sitecore Social Connect module which it seems Sitecore 8.1 update 2 does not do it on default. Of course however we can hook into the Sitecore Social User LoggedIn event or similar to perform this.

First thing first, if you don’t know how to setup a Facebook application with Sitecore then have a read through this article. Make sure that before going through the rest of the article you are able to sign-in using Facebook and verify that the user is created (check the Sitecore User Manager) and the contact is also created in MongoDB.

The first thing you would notice after the user is created is that the Facebook profile picture is not set by default in the Experience Profile after the user registered. This is what we’re going to address.

Reading through the Sitecore documentation on logging in using a Social Network credential it seems that the process is quite straightforward. Looking at the /App_config/include/social/Sitecore.Social.config, I found that there’s a couple of events that we can hook into to add additional logic

Looks like the perfect place to put the additional logic to get the Facebook profile picture. Which I did btw

note: This is by no means production ready code.

The class above will update the current contact profile picture which is coming from their Facebook account and also a couple of other information such as name, email address, gender.

That’s it.

A couple of tips that might be useful when you want to try this on your own:

  • Set Social.Logging.TraceToLog value to true to help debugging
  • You need to disable the interest field on the user’s profile as that field has been deprecated by Facebook – this is throwing an exception which I can see in the log file. You can do this by going to Sitecore.Social.ProfileMapping.Facebook.config and disable the fb_interests field.
  • If you’re trying to test the synchronization of user profile fields from Facebook or other social network then set Social.ProfileUpdating.DaysBeforeExpiration to 0. See Sitecore documentation for reference

Issues encountered with Sitecore Azure Module

I was playing around with Sitecore Azure Module last week and would like to share my experience to get the site up and running.

I am using Sitecore 8.0 version, WFFM and EXM modules. Obviously you would need the Sitecore Azure module to help you setup the content editing and content deliry farm.

  • Sitecore 8.0 rev. 160115 (8.0 Update-7)
  • Sitecore WFFM rev. 151127 (Update-6)
  • Sitecore EXM  3.1 rev. 151213 (3.1 Update-2)
  • Sitecore Azure 8.0 rev. 150522

The target Azure setup would be

  • 1CM and 1CD server in the same data center, in my case I create the farms in the Singapore data center as it’s the closest one for me
  • Sharing the same databases for CM and CD; core, master, web, reporting. In the case of CD it will not have a reference to master and reporting
  • xDB cloud will not be used, instead MongoDB databases will be created using the free 500MB databases from http://mongolab.com

Before proceeding you would need an Azure Subscription in order to be able to create the necessary VM/Cloud Service, if you’re like me and just want to try out the Sitecore Azure Module you can sign up for a free trial for one month.

Installation

In the installing Sitecore Azure Module article above, it also explain about how to deploy to Azure to setup a Editing Farm and Delivery Farm which seems to be easy and straightforward. However I encountered some issues while trying to deploy to Azure.

Issues when deploying

Error saying that the storage account already exist in Azure

Solution:

 

The storage account service name must be unique in Azure across subscriptions, meaning that it has to be unique not only for you and for other subscriptions in the Azure as well. To fix this you can go to /sitecore/system/Modules/Azure/[project_name]/[region_name]/Editing01/Storages/Default and update the Service Name field.

 

Error saying that the database server already exist

Solution:

In the /sitecore/system/Modules/Azure/CampaignManagement-Evaluation/Southeast Asia/Editing01/Sql01 “Server Name” field empty the value. And then clean the unused Azure SQL database server.

 

Error saying that the vmsize does not exist

Solution:

This is because the vm tier name has been changed by Microsoft, which originally was D3 is now has been updated to Standard_D3. To fix this you would need to update the /App_Config/AzureVendors/Microsoft.xml

 

With the free trial subscription you might encounter an issue when the Sitecore Azure Module tries to create the database in SQL Database S2 tier

Solution:

To fix this you can temporarily change the /sitecore/system/Modules/Azure/[project-Name]/[region_name]/Editing01/Sql01/Set01/[database_name] Tier field and change it to “Basic – Basic”. This could also be caused by the database size Sitecore Azure Module requested to be created is not supported by the free trial subscription, I haven’t tested this however if you would like to try it yourself then update the /App_Config/AzureVendors/Microsoft.xml change the

to something smaller, say 100GB or 10GB

 

Another issue when trying to create the Cloud Service

Solution:

This issue seems to be related to the siteStorageSize configuration in /App_Config/AzureVendors/Microsoft.xml. I was trying to create a D3 instance which has the default size of 180GB, reducing the value to 10GB or more seems to do the trick

 

Some reference links that might helps: