Sunday, 29 July 2018

Implementing a Continuous Delivery Pipeline - The Basics

Continuous Delivery The Basics

Part 1 of my posts around Continuous Delivery.

This post will cover the basics of Continuous Delivery (CD) and also the tools/technologies I have used to build a CD pipeline.

Let's start with a nice definition:

What is Continuous Delivery?

As detailed on Wikipedia here, Continuous Delivery is an "approach in which teams produce software in short cycles, ensuring that the software can be reliably released at any time. It aims at building, testing, and releasing software with greater speed and frequency".

Sounds alright, doesn't it? But you may be all:

Why should I care?

If you want frustrating, big-bang, on-a-knife-edge release nights to be a thing of the past, you should strive to implement CD practices. The good thing about striving for this is that even though you may not get all the way, all the changes and good practices that you have implemented in trying to get there will see your team benefiting from more automation, quicker releases to your environments and confidence in that what you are preparing to release is actually fit.
From my experience I can't think of another approach that has produced so much benefit to a software delivery team as implementing good CD practices.

If you're interest is piqued, you may be thinking:

What are the basic tools/methods I will need?

In order to get closer to a CD approach, it is essential to have:
  • Source control - but you know that already, right? Preferably we want to use Git, the reasons for this we will cover later.
  • A "good" branching strategy - I am an advocate of feature branching, though this is not a one size fits all solution. This by Dave Farley is an excellent read on this and more - including how doing anything other than always checking into master negates true CI (I haven't worked in a team that has achieved this yet).
  • Continuous integration (CI) - when developers check in, this is our first "quality gate" in the entire release pipeline. We want to run unit tests to ensure that the developer's commit will not break the build of trunk/master.
  • Testing coverage - comprising of unit tests, automated functional and component tests plus manual exploratory testing.
  • The ability to compose release definitions, using something like TFS.
  • DevOps - you're going to need help with such joys as server access, AD, databases and other infrastructure related delights. You need a good relationship with your Ops representative, and DBA.
Let's go into more detail about quality gates.

Quality Gates

As I mentioned in the context of CI above, CD must have quality gates to work. Having these quality gates at the rights points ensures less wasted effort and confidence in your release artefact. I would set quality gates at these points:

  • Developer checks in code - unit tests ensure this new code won't break anything
  • Merge to trunk/master - run unit tests against the merged version of the master branch to ensure the new code doesn't break this branch. If it does, this becomes a priority for the developer, if not the team, to fix. See such approaches as this.
  • Automated tests - be it functional or component, a failed automated test must mean the artefact under test is not fit for release and does not go onto the next stage.
  • Manual testing - if your tester's find a bug, the artefact is not fit for release and does not go any further.
  • Post release testing - if your post release tests find an issue, you may either view this as worth investigating with the opportunity to fix forward, or you may be so mature that you automatically roll back through automation.
With these points in mind, let's look at how these gates fit into the release pipeline:

Continuous Delivery quality gates

See how this saves on wasted effort? Most obviously for the beloved manual testers - who won't have to waste time testing an artefact that is not fit for release as previous quality gates would have already flagged it as unfit.

Why Git?

Git helps to ensure that the above quality gate methods are being conducted against the same artefact every step of the way. Git is faster and easier when using a feature branching strategy as creating a new feature branch is completed via a simple command.
Additionally, we can set up various automation to happen only against the context of master. For example deploying code automatically to the development environment once code is merge to master, and the build succeeds.
Git encourages more frequent commits and merges. Why is that important?
 

Developer check in blast radius

To minimise the blast radius! 

We want developers to check in as frequently as possible. Use CI! It is meant to be continuous! And here we think of it as continuous feedback. Use the CI build definition to get fast feedback via unit tests (if you're doing TDD, you'll likely have a nice large suite of tests to rely on). Also, the quicker developers are committing, the less change there is to merge to master. and the less chance of conflicts. This is doubly important when working in a large development teams that can see sets of developers working on the same component at once.

Tangent alert - branching strategy experiences

I must mention a particular branching strategy I have worked with before. We were using Team Foundation Server version control, and for each component solution had a Dev and Main branch. The idea here was that devs would do their work against the Dev branch, and release to the development environments from there. Only when they had tested their changes in that environment would they merge to Main. The release candidate would then be built from that Main branch.
This is an antipattern - ensure you promote at a binary level, rather that at a source code level. Why? By promoting at a binary level, you are increasing the testing exposure to that release candidate right from the get go, and interrogating its fitness for release through the entire deployment pipeline. I see this as one of the pillar philosophies of CD. Promoting at source level, as we did with that Dev and Main model, means you are creating an entirely new release candidate after the dev has merged their code to Main, meaning this new release candidate will not have gone through the initial dev CI goodness, or been tested on the development environment. SAD!

So that's probably enough in a "The Basics" post around source control, branching strategies and quality gates. So let's move on.

Finally! So, you mentioned DevOps...

Yes. DevOps is essential to implementing CD. What is it? You can read about it here.
As you will be deploying to environments faster using automation, and will no doubt be looking to implement IaC, you are going to need a friendly Ops companion on your journey through permissions, access and related wonders. You simply won't be able to get very far without having an Ops resource to assist with this stuff.
The ideal working model here is to have your Ops resource sitting in the team (doesn't have to be permanently, but certainly encourage as much access as possible) pairing with devs as issues arise. If boundaries wish to be maintained, the Ops resource should be the one to approve pull requests for changes to environment configuration (e.g. changes to firewall or AD).
It is important to try to achieve buy in from your Ops resource, lest you face feeling like you are just irritating them whenever you need something doing. They should be in the trenches with you every step of the way, striving for the same goals. Try to avoid a pull model, but strive for collaboration and equality to such an extent that your Ops resource is coming up with solutions before they have become an issue for you - this is so beneficial, and a pleasure to work with.

...And the DBAs?

All what I have mentioned for the Ops resource should also apply to a DBA resource. In my experience, DBA teams or individuals are a bottleneck and strained, so it is that much harder to drive a DevOps culture here. But again, the benefits are massive. Imagine having a DBA attending your refining and planning sessions, sitting with and advising the team as database changes are coded, and being a part of your release. Contrast that with the typical model - fights for the permissions to release database changes, actually getting hold of someone to discuss it, short emails around permission of least privilege etc... it doesn't have to be this way!

Ok, that's enough for an introduction. Hopefully you can see the benefits of CD, and have a rough idea of what is involved to get started. Check back soon for more detailed posts on Powershell DSC, Pester tests, TFS release management, DevOps, database deployments and more.

Friday, 27 July 2018

Windows Server - Loopback Check

IIS LoopbackCheck

What is it, and what trouble does it cause?


I'm going to post this as an interlude to my Continuous Delivery series, as I have been wrestling with an IIS issue this week, and wanted to share the solution in case anyone else suffers the same frustrations!


After struggling for 2 days with sporadic 401s in our IIS logs, quadruple checking authentication settings, permissions and config it was a great relief to pin down the underlying issue: loopback check.

What is loopback check?

Introduced in Windows server 2003, and found in all subsequent versions (at the time of writing), the purpose of the loopback check is to protect a server against threats exploiting connections to the loop back adapter specifically, and this extra protection level applies to all incoming connections and protocols.
So, if you issue a call on the same node using the FQDN to browse a local web site hosted in IIS, you will get a 401 back. Not really the first thought that'd enter your head when seeing a 401...

How can I fix it?

There are two fixes, both of which require registry entries - the first is to add the relevant host name to a BackConnectionHostNames registry key here:
 HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa\MSV1_0

 The second, less appealing fix is to add a DisableLoopcheck key with a value of 1 to here:
 HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa
 ...but this will disable the check entirely for the node. Feels a bit to heavyweight, especially if you need to do this on a production server.

Here is a good article on Stack Overflow for more:
https://stackoverflow.com/questions/17466665/windows-authentication-not-working-on-local-iis-7-5-error-401-1 ‹ collapse

Can I automate this?

Yes! And to bring us back into the realms of continuous delivery, we can add your preferred fix from the above choices into a Powershell DSC configuration as described here.
For example, this will add your relevant host name to the "BackConnectionHostNames" registry key on your destination node:

Registry RegistryExample
    {
        Ensure      =
"Present"
        Key         =
"HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa\MSV1_0"
        ValueName   =
"BackConnectionHostNames"
        ValueData   =
"YourHostName" 
    }


When DSC runs, it will ensure this above registry value is present, and add it if not :)

May your irrational 401s be gone!


Sunday, 22 July 2018

Implementing a Continuous Delivery Pipeline

Continuous Delivery


We've all been there. Midnight (or later), eyes darting around the release team, the question hanging in the air like the odour of rotting fruit... You've changed configuration that you might have a record of but probably not, restarted some servers, triple checked the permissions, tried to think back to 2 months ago (if you're lucky) of what is actually contained in the release...

Finally the lead grasps the rotting fruit and sighs: "I think we'll need to roll back..."

It doesn't have to be like this. By establishing Continuous Delivery practices you can reduce the pain of release night to a mere button push, safe in the knowledge that you're pipeline is so robust and chock full with quality gates that you've already caught all bugs and dealt with the usual integration issues. Plus, as you've implemented configuration as code, you know your destination servers are in your desired state, and there can't be any permission or config creep lurking. Additionally, you have had the (at times painful) conversations to ensure TechOps and DBAs know how your release pipeline works; how it is auditable, repeatable and secure. Of course, once your candidate is released, you have a probing set of post deployment tests that validate the release and ensure the environment is good.

Hey, I've been in the first paragraph. I've had to roll back at 1 in the morning, rush home for a few hours sleep and have to try again the next night.

But, having recently achieved a lot of success in implementing an automated release pipeline right through from development to production, I am in the enviable position of being able to pass on the key points that can get you closer to continuous delivery and make those strained release nights a problem of the past. Interested?

I will detail my experiences in a number of blog posts. These posts will consist of:

Implementing a CD Pipeline - the basics 

This will be an overview of Continuous Delivery and the tools/technologies I have used to scaffold a CD pipeline. If you have any interest in this subject whatsoever, you must read the meticulous and edifying Continuous Delivery by Jez Humble and David Farley.

DevOps, and The Politics

The parties that you will need to have on your side, and how DevOps plays a vital role in this.

Powershell DSC

How Powershell DSC is used to configure components on a destination server. Configuration as code.

TFS Release Management

Good practices in building robust, repeatable definitions.

Pester Tests

Validate your deployment with Powershell Pester tests, and create checklists for your environments.

Database Deployments

How I used SQL Server projects with DACPACs to deploy database changes. 

What I need to Improve

Testing! How can you have confidence that what you are going to release to production is fit, if you're not thoroughly testing your candidate?


Check back soon to see this collection evolve!

Friday, 12 January 2018

Sprint Planning - Velocity Planning?

Towards the end of our sprint planning session today we were in a position of committing to just over our current velocity. Our Scrum Master rightly wanted to check if the team felt comfortable with doing that, and was met with a resounding No. We have recently lost a developer from the team, and therefore felt uneasy about committing to more than, or equal to, our current velocity.

The stance back from our Product Owner was however that we should be committing to velocity, as recent training courses they had attended stated this was the thing to do, and if we don't do this, what is the point in having that statistic.

This led to a tricky stand off between the PO and the development team.

Our PO was absolutely of the mind that we should be planning to velocity. Of course, if we hadn't lost a developer then I'm sure this would have been accepted, but the development team was stating their unease in doing this, and wanted to commit to slightly lower than velocity, given the reduced capacity and the high complexity of the user stories in scope. Plus, we had a 3 day time boxed spike and 2 bugs (not included in our velocity) to contend with.

This has made me want to learn more about velocity planning, and has led me to two points:

  1. Should you use velocity to plan a sprint if the team has decreased in size, and
  2. Velocity v capacity planning

So, point 1. From what I have read, velocity planning is an excellent gauge of what a team can achieve in a sprint, if that team is mature and has a constant team size. As Mountain Goat software states:

Velocity-driven sprint planning is based on the premise that the amount of work a team will do in the coming sprint is roughly equal to what they’ve done in prior sprints. This is a very valid premise.
This does assume, of course, things such as a constant team size, the team is working on similar work from sprint to sprint, consistent sprint lengths, and so on. 

And as this article on Scrum Alliance states:

Velocity relies on team consistency in order to be most valuable. If your agile team changes, use common sense in planning future iterations. If 20% of your team is unavailable for a couple iterations, then reduce planned velocity by 20% or so. 

So these posts indicate the development team was correct to push back on using velocity planning in this instance. I can certainly see the benefit of using velocity planning in the future, but only once the team has re-stabilised and gone through several sprints (4-5 is the recommended).

Onto point 2, velocity v capacity planning.

Let's get the easy bit out of the way: you can only implement velocity planning if you have a team that has been around long enough to do that with. A worry, though, with velocity planning is that the development team can become "story point accountants". Capacity planning makes sense as you can build your sprint to who is available, but doesn't factor in complexity as well.

This article on Software Engineering Stack Exchange tackles the velocity v capacity planning question. I like the takeaway:

My advice would be to use your story-point velocity that you have established to determine which stories to take on, and then build out the true capacity for the upcoming sprint based on resource availability.

That makes sense, right?

Aside from the two points above, one angle that I hadn't considered was the Sprint Goal. If this had been set up front in the planning session, perhaps this would have given our PO more buy in into over committing, as we may have included the story that we moved out of the sprint if it was an integral part of the goal?

Oh, and in case you wanted to know what happened in our case, the agreement was made to commit to just under velocity, with the next story to pull in agreed to be the one that was not committed to. So the development team's view was taken over the PO.

From my research as illustrated above, I'm pleased that it was.

Saturday, 2 December 2017

DevOps at Sprint Planning

Today marked a real success for the team as we start to make our way to a more DevOps way of working. Our "DevOps Resource" (read representative of ISS / Sys Admin), sat in with us at our sprint planning meeting.

This allows Shaun to gain the context of each user story that we were going to commit to and promotes buy in, involvement and feeling a part of what the development team is doing.

Importantly, as this is sprint planning, Shaun has an upfront view of these stories, so we will avoid the usual rigmarole of emailing / Skyping as and when we need ISS involvement and the inevitable frustration that arises when a) we can't get hold of them, or b) they do not action our request quickly enough.

As we embark on the latest sprint, we're seeing these ISS tasks either get completed by the first day, or soon after. This has resulted in an improved relationship between the two parties and I believe a sense of empowerment for Shaun.

DevOps


Long may that continue!

Wednesday, 29 November 2017

Post #1

First post

Here's the very first post to my new blog. I am a lead engineer, and aim to post my thoughts and adventures from my day to day working life.

Check back soon to see what's been going on!