While refactoring the TFS and Continuous Integration struture of a couple projects, one thing toke my attention: Health Check builds were stealing most of the building times, which for me was representing long build queues to wait. The reason they were crowding the queue is because they were taking a long time to complete (about 10 to 15 minutes), and building too often (every 5 minutes when changes are pending).

Strength or kindness?

Since our CI tool has only two build agents, the obvious answer is to increase this number to properly serve the whole company. But does this tells the whole story? More build agents means more resources (disk space, licenses, etc) and unfortunately we still live in a world of limited resources. This also means we rarely will have resources enough to properly serve everyone at peak times.

Take as comparison the limited space most cities have to make their roads. Most of the time traffic can be ok, but when everyone goes out at same time, like in rushy times, there are no room for all and we have car jams.

Fortunately it’s easier to interfere and positively affect our Continuous Integration systems, so why not try a thing or two as a sign of civility, just like we do in a Drive Thru not taking too much to order your fast food?

Health Check builds, how big?

Depending on the Solution Architect who setup the project you can have different things, but most cases what I see is Health Checks doing everything but deployments. Since we use TDS in our projects, you can make it build a package of the whole deployment, along with some meta data. But is it really necessary? It’s a question I made myself, when decided to make it different.

My Health Check ended up being minimalist: just a compilation is executed, while TDS is kept totally off. No packages, nothing. Ended up with builds taking 45 to 90 secs to finish, much better!

Health Check builds: how often?

Being tiny means it can build more often. In my case I have it building after changes are detected, with a 5 minutes latency to avoid subsequent check-ins to causing multiple builds in sequence.

Full builds and Health Checks working together

No doubt the resulting Health Check builds are weaker, as it’s doing just a tiny part of the integration process. To fill the gaps, I also setup full builds working in conjunction with them. These are also set to run automatically when changes are pending, but with a much longer latency. Having it building each 4 or 8 hours will ensure a full integration is made once or twice in a normal day work.

Let’s also keep in mind these deploys can and must be manually triggered by developers, no matter the latency, as soon as they finish the user story they are working on, so they can test what they did at the integration servers before considering their work done.

In short: deployments will be made when developers finishes their work, or at minimum once or twice a day if pending changes are to be deployed.

Full builds with Asynchronous HTTP calls

Another improvement I made was replacing at full builds some Synchronous HTTP calls by Asynchronous. They were mainly used to wake up instances after deployments and publishing from CM to CD. Most cases, The build agent doesn’t really need to wait for these calls to respond before going to the next step, so we can save it’s precious time to another teams.

Your impressions?

What about your experiences, do you agree and disagree? What other factors are left behind at this analysis? Let me know your thoughts!

]]> https://blog.peplau.com.br/health-check-builds-with-continuous-integration-how-big-and-how-often-these-needs-to-be/feed/ 0 https://blog.peplau.com.br/sitecore-7-5-and-mongodb-through-different-environments/ https://blog.peplau.com.br/sitecore-7-5-and-mongodb-through-different-environments/#comments Mon, 09 Mar 2015 00:58:11 +0000 http://blog.peplau.com.br/?p=127 Jan Löwgren, another Sitecore Consultant, in response to my post “Sitecore Environments for Dev, QA, UAT and Production” said It would be interesting to elaborate on how to handle MongoDB though different environments. In the project I am currently working the team was about to upgrade from Sitecore 7.2 to 7.5, so the opportunity to engage into a MongoDB […]]]> Jan Löwgren, another Sitecore Consultant, in response to my post “Sitecore Environments for Dev, QA, UAT and Production” said It would be interesting to elaborate on how to handle MongoDB though different environments. In the project I am currently working the team was about to upgrade from Sitecore 7.2 to 7.5, so the opportunity to engage into a MongoDB experience was the excuse I wanted to volunteer myself to the upgrade task.

Trusting your instincts

My first thought before getting my hands on the issue was trying to handle mongo databases as we do with Sql server dbs: each Developer would need to install a MongoDB server at their machines.

Setup before Sitecore 7.5

For the context of the project, our environment’s setup was pretty similar to what is described in my first article:

1. Developers and front-end devs works at their local machines, with local copies of solution and databases – everyone has their set of “master”, “core” and “web” databases. TDS is used to syncronize Sitecore data with the team;
2. We have an integration server Sitecore instance, periodically updated by TeamCity on every build change;
3. We also have a QA Sitecore instance, updated by the trigger of the QA team;
4. The last instance is the UAT server, also updated by TeamCity by the trigger of someone with this privilege (usually the Project Owner by request of the client);

Solution

The final setup ended up being pretty similar to my first thought. It makes sense if you think that MongoDB is not much different than a normal Database Server. At Sitecore 7.5 MongoDB serves DMS Analytics and access data, so we can safely have separated databases for each Sitecore instance (Dev, CI, QA and UAT).

MongoDB Installation

Since every developer needs a separated MongoDB server, everyone needs to install at their own machines and access using their “localhost”. This very easy operation can be made by following the steps of this article (Sitecore and xDB – Setting up MongoDB on your developer machine).

Here is where the developer’s ConnectionStrings will end up:

<add name=”analytics” connectionString=”mongodb://localhost/dotz_analytics” />
<add name=”tracking.live” connectionString=”mongodb://localhost/dotz_tracking_live” />
<add name=”tracking.history” connectionString=”mongodb://localhost/dotz_tracking_history” />

MongoDB server initializing

MongoDB is not a service you can install on Windows – you must run it manually – so it’s recommended to have a batch pinned up at your Windows start button before you launch your development instance.

Windows Start Menu with a link to the batch that starts MongoDB

Your batch can simply contain the following:

“C:\Program Files\MongoDB\Server\3.0\bin\mongod.exe” –dbpath “D:\MongoDB”

• First path depends on where your MongoDB is installed
• Second path is where MongoDB creates the files to manage databases

What about the CI, QA and UAT servers?

At the Continuous Integration servers the installation is pretty much the same, the only difference at our case is that we opted to have a centralized MongoDB server, with separated sets of databases for each Sitecore instance (CI, QA and UAT).

Some time ago I found this Stack Overflow question discussing multiple aspects of serving a Sitecore solution during different phases of a project life cycle. The question was pretty old but still the only answer was not as good as I know it could be. Due of the relevance of this issue without a good response, I decided to write my version of the answer.

Premisses

So the question from Azeem Siddiqui starts by putting some premisses:

Our organization’s website is moving to Sitecore CMS but we are struggling with setting up the environments for Developers (4), Designers (4), QA persons (3), Authors (10-15) and Approvers (4-10) in a way where they can work independently, I know that there will be dependencies but idea is to minimize it.

Here are couple of rules:

1) Whoever is responsible for the change then they should do it everything until and unless there is any dependency.

2) If one team is working on one feature then it shouldn’t stop or effect other team’s work. For example, if QA is testing the feature then Derringers and Developers should continue their work on the same feature for new enhancements.

Common questions for those experienced with another CMS platforms and seeking to know how Sitecore works in the real life.

The Big Picture

I started my reply with a diagram illustrating a way to setup servers for a Sitecore project, allowing the team to work together without interfering each other.

How people work together?

1) Where the Designers will work? I mean where they will add their html, js and images? On which server? In Sitecore? In Source Control (TFS)?

2) How the Designers and Developers should work together? I know developers will work on their local machine’s in Sitecore. And will promote their work to Integration server but How they will get the Designers stuff? Let suppose the feature has gone into production successfully now only Graphics Design changes are required, let say font styles and some images then where Designers should make these changes? On which Server? And after that how that Sitecore instance will sync with other Sitecore instances. And for design changes I do not want developers for promoting any code or file.

Answer: Both Devs and Designers works at their local machines, using local Sitecore instances. They use TFS as the Source Control system so they can mutually integrate their work. Usually Designers works more in CSS, Javascrips, Images, Sublayouts (markups) and Developers at the code itself. We have a Continuous Integration server in place (Ex: TeamCity) deploying changes to 3 different environments – CI Server (for build healthcheck), QA Server (For QA) and Prod Server (for content edition and public access). When, for instance, a designer has to fix a layout issue, he will do that at his local machine, then commit changes to TFS. Next step, TeamCity will deploy changes to the CI server, if the build is Ok the QA person can trigger a build and test the fixes. If everything is working as expected, someone can trigger the build to the production server, and the fix goes live.

You have this second diagram showing details on how you can setup your Production server to separate Content Authoring and Content Delivery – Here is a search I found several blog posts on setting this up: sitecore content authoring delivery

How to deal with Database changes?

3) What is the safest way to sync the Sitecore environment/databases? Means whatever has been published into production website, we will need back in DEV, QA and UAT environments.

We do not want to do any manual promotion of code, html, js and image files. Is there any way to do these kind of things automatically via tool or Sitecore commands. Personally I do not like the Sitecore packages.

Answer: You need TDS (Team Development for Sitecore) as shown in the first diagram – Use this tool to serialize/deserialize items from one Sitecore instance to another. Then you can have the serialized files to TFS and share across the team and environments. The good thing is you can use TeamCity to automatically push items to CI/QA/Prod environments.

Where to find good documentation?

4) Do you know any good reference? Where I can find answers of similar questions? Any website, book, blog?

I know one document “Understanding Sitecore Deployments 6.2″ but designers part and how the different environments will be synchronized are not discussed over there.

Answer: The main source for info on Sitecore is their SDN – You can register free (or have an extended account if you own a sitecore license). Some other blogs were suggested, such as John West’s and Alex Shyba’s, as well as – why not – this humble blog I’m just starting.