Tuesday 23 December 2014

DevOps - what does "good" look like?

I've blogged a lot about my definition of DevOps and what it means to me.

I've always thought it much larger than "Dev" and "Ops" and that there isn't really an end state - it's an ever changing idea that will adapt to meet the needs of the business.

That said, I'm asked quite often what "good" looks like?  How do you know you are on the right path?

For me DevOps is a primarily a cultural idea.  If the culture is right then the technology and people will follow.  I talk to a lot of people who see this word "DevOps" and reach out straight to technology.  DevOps isn't another word for automating Cloud servers...

In order to an effective "DevOps" organisation I think you need 4 main pillars:

  1. Culture - does the culture of your company promote collaboration, self improvement etc?
  2. Process - you need to have processes that support different silo's working together to achieve a fast pace of change
  3. People - if you don't have the right people then it doesn't matter how great your technology is.  People and culture are very linked but I've tried to separate them here to demonstrate qualities that individuals hold vs values that the company holds
  4. Technology - what are the building blocks you need to be a mature DevOps enterprise?

The following is very much my view of what "utopia" looks like.  The truth of the matter is that DevOps means different things to different people depending on what you are trying to achieve.  For example, you absolutely don't need to use public Cloud to achieve DevOps.  However, I do think it's a good pattern to promote working on high value tasks (rather than undifferentiated heavy lifting**) and using a common change process between development and Ops teams.

This is where I want my company to get to...


Culture
  • Measure everything, always
  • Individuals have empathy for the rest of the team (i.e. they don't pass the buck)
  • Shared goals and incentives
  • Don't reward the fire fighter, reward the fire preventer
  • Reward innovation and challenging the status quo
  • Don't punish people when they try something new but fail
  • There is no IT and "business".  IT as much "the business" as the sales people.
  • Seeking to break down silo's
  • "It's not my job" doesn't exist
  • "Its my server/code/network/database" doesn't exist
  • Individuals are empowered to make decisions
  • Top-up management rather than top-down


People
  • Restlessnes (relentlessly looking to improve themselves, others, processes etc)
  • Good technical ability across a broad skill set (understand as much of other people's jobs as possible)
  • Everyone can code and use version control
  • Everyone understands the test triangle
  • Dev-test pairing
  • Organised in small product focused teams rather than technology silo's (align teams to the business not the technology)
  • Common incentive schemes
  • Favour automation and repeatability above anything else
  • CIO/CTO are DevOps biggest guardians & SME's and seek to destroy anything that affects that  culture
  • Natural face-to-face influencers rather than endless emailers
  • Natural sharers of information
  • Take an interest in their specialism outside of work (i.e. go to conferences and take part in the wider community)

Process
  • Agile
  • Continuous Integration
  • Continuous Delivery
  • Lean
  • Fail-early, fail often
  • Release management team are facilitators of change not guardians of change (i.e. they try and aid change rather than stopping/slowing it)
  • All change (I mean all) goes through the pipeline from left to right (dev, test, acceptance, production)
  • Knowledge sharing and "just enough" documentation is part of the process
  • Measuring success and failure is part of the process
  • Retrospectives are part of the process


Technology
  • TDD/BDD everything (including Puppet etc)
  • Everything is in version control (code, automated tests, server config etc)
  • Release automation tooling
  • Convergent desired state tooling
  • Cloud (private or public but the ability to spin servers up on demand)
  • The same trending, monitoring & alerting solutions available through nonprod & prod 
  • Application Performance Management
  • Service Virtualisation
  • Continuous Integration
  • Continuous Unit tests
  • Continuous Service level tests
  • Continuous GUI tests
  • Performance testing

If you can get half way to delivering all of that then I think you should be in a very good place indeed.

It's a long hard journey if you are a traditional Enterprise.  Nothing worth achieving was ever easy though!

I'm outta here for what proved a fantastically successful 2014 (personally and work).  2015 is a massive year too for me but good luck in whatever you have planned ahead.

Happy Christmas to you all!

J
xx



Wednesday 3 December 2014

DevOps Rugby



I spoke at IBM DeveloperConnect a while back talking about DevOps and how it relates to what I'm doing in my current company.

That isn't particularly interesting in itself.  The really interesting thing was at the end of the day where I was well and truly blown out of the water by the key note speaker.

The key note was held by Kyran Bracken MBE (http://www.kyran.co.uk/) the World cup winning, ex-England rugby scrum half.

Firstly its impossible to compete against someone who has won the rugby world cup and captained the national team in your favourite sport.  In fact, I came away thinking I've never accomplished anything in comparison.  Soul destroying!

Kryan talked at length about the role of a manager vs. a leader, the importance of challenging convention and taking risks.

It was an hour speech so I won't document it all here.  However I thought the 3 things above are relevant to the world of DevOps.

Manager vs Leader
In a well functioning DevOps function, the team is empowered to make decisions.  You don't need a manager telling people what to do and how to do it.

However, you do need a leader.  Someone that wants to creates the future and inspires others to join him/her on a journey.

The leader welcomes risks because no one ever accomplished anything great by playing it safe.

Both Martin Johnson and Clive Woodward were leaders.  Martin because he inspired people and Clive because he challenged so many conventions...

Challenging convention
The only way to improve things is to take the status quo and change it/improve it.  You can't get better standing still.

The Leader facilitates these challenges and encourages them.  Its very healthy to look at what you do and critique it - even if it works really well as the current convention.

Don't rest on any laurels.

Before the World Cup England had a good team but in order to get that extra 1% from somewhere they needed to challenge long held conventions.

Most teams before mainly worked on training to improve their attack.  However, through statistical analysis found that teams with the best defense tend to win games.  They then spent a much greater proportion of time working on defense.

Traditionally, rugby player wore heavy jerseys like this:




They are really heavy when wet, don't let the body breathe and are easy to get a grip on them. The 2003 World Cup team were the first to champion the skin tight Lycra ones that everyone wears nowadays.


Taking risks
Trying new things can be seen to be risky.  However if you want to separate yourself from the competition you need to take risks.

Clive Woodward had the team training their eyes and working on other new techniques that may have been derided if they hadn't won the World Cup.

If your team doesn't take risks or is punished for every failure then they will never move forward.  Taking risks should be encouraged and not penalised if it doesn't work.  Learn from those failures and move forward.









Wednesday 12 November 2014

DevOps results - part 1

Last week I was lucky to be copied in to an email to the exec board of my company.

The email was written by the CIO advertising some of the success of our recent deployment automation efforts.

On a very basic level it compares human effort vs. automated effort and therefore cost and time.

The results are pretty amazing (well for us anyway) - we did 39 releases in a week for 1 application saving us 17.5 man days of effort.  We've done 49 this week and it's only Wednesday...

The benefits of this are probably more profound but it's a useful start.

For example...
  • How much time is saved by having environments consistently built? 
  • How much more beneficial is it to have every part of the deployment audited?
These types of things are very difficult to calculate the benefits of.  Thus my point recently at IBM DeveloperConnect (UK) when I said to focus on the softer benefits (like time saved) rather than deep ROI calculations (like money saved).

 Metrics are super important...
  • How do we know if we are delivering value if the team isn't measuring it?
  • How do we know if we are working on the right area's if we aren't measuring it?
  • How do we influence other teams to do more "DevOpsy" things if we can demonstrate the benefit?

Saturday 1 November 2014

Cloud tie-in

In thinking about moving to the Cloud (I'm specifically talking about Paas and Iaas here) I've always had in the back of my mind that we should ensure that we weren't too tied down to a particular vendor (such as AWS or Azure).

Wouldn't it be great if you had your whole infrastructure defined in Puppet manifests and you could just pick it up and move it?

Why might this be useful?

  •     Move to whatever Cloud provider is the cheapest that month
  •     Move away from a provider if you are experiencing issues (outages, performance etc)
  •     Move to a provider that have introduced some sexy new features

I believe the reality is that whoever your first Cloud provider is then it's likely you are going to have to stick with them.  The reality is that the Cloud is basically just another data center (albeit a very big and clever one) and the challenges of moving between data centers remain.

If you have written automation scripts (which you should have) to operate in the Cloud (provisioning, monitoring, alerting etc) you have almost certainly hooked into that Cloud's API.  You will have probably created an eco-system around your Cloud provider.

Also bear in mind that although a Windows 2008 IaaS server might be the same thing between Azure and AWS that things like load balancers won't be.  You won't be able to export a Azure load balancer and import it into Azure.

Moving Cloud could therefore mean rewriting all of this (this is why Puppet & Chef need to up their game).

In addition, you have the more obvious considerations of how to move the mass of machines and data from one place to another.  How easy it to move Azure Blobs to Amazon Elastic Block Storage for example?  How do I prevent downtime?

I'm guessing it's not impossible to move from one Cloud provider to the next but it's certainly not trivial.

I love the idea of utility based computing (i.e. only paying for what you use and when you use it) and the commodisation of IT as a product (i.e. I don' really care about what is happening under the covers, I just press a button and get a server).

However, to be a true commodity Cloud needs to ultimately become more portable.

Competitors of your current Cloud provider will like this.  Your current provider won't.


OpenStack?
Perhaps this is one of the reasons OpenStack was invented although I have concerns over it's long term viability.  OpenStack Cloud providers seems to bolt on additions to these standards to integrate it with their own systems or to create competitive advantage over another provider.  They have taken a standard and added to it - meaning it's no longer a standard.

OpenStack functionality vs. the might of an Azure or an AWS is also lacking.



Not that you would pick a Cloud provider on a whim, just be very careful to think about the wider implications of your selection criteria including an exit strategy.

Thursday 30 October 2014

Spiderman and Azure


I've been really impressed by Azure at the TechEd conference in Barcelona this week.  We are starting our Cloud journey in 2015 so it's been great to see how far things have moved on in the last year.

Because I'm a geek I signed up to the free Azure trial (http://azure.microsoft.com/en-us/pricing/free-trial/) and had a play around for an hour.

In 1 hour I created a load balanced pair of web servers and a geo-located SQL cluster (i.e. one node in Europe and the other in the US).

1 hour.

That's ridiculous.

I'm not sure we even have the technical skills to do something like that little lone the time it would take (literally months).

1 hour from a cold start with no real experience of doing it before.

So that's all very good.

However, earlier in the week the excelled David Chappell (http://www.davidchappell.com/) talked about the Cloud and how with great power comes great responsibility (like Mr Spiderman).  Couldn't agree more.

If its that easy to do and if you aren't careful, your company will be spinning up machines all over the shape and things will quickly get out of control.

Just because you can spin infrastructure quickly doesn't necessarily mean you should do...


One of way of getting around this might to be to put some kind of dashboard over the top of the Azure capabilities (I think System Centre can do something like this) which automates builds and adds in things like approvals.  I really don't like having to nanny individuals but the reality is that if you let people build servers and environments manually (as well as install applications) things will get out of sync pretty quickly.

And your credit card bill will be substantial (ooops I forgot to turn off those 100 test servers I built last year).


Automating Azure


I'm not speaking from a point of great experience so my thoughts on Microsoft's automation approach might be entirely misguided.  Apologies Microsoft is this is not the case but  I'm not convinced their automation approach for Azure is right.

I'm a big fan of idempotence, infrastructure as code and desired state.

I can talk about the above 3 for quite some time so I won't bore you here (if you aren't already!).

Microsoft's approach is heavily reliant on PowerShell for Azure automation jobs.  These jobs are basically script and therefore contain logic and without a very skilled developer they are not idempotant.  Moreover, its difficult to version control these scripts as it seems you enter the PowerShell code directly on the Azure portal (probably wrong here!).

Being a procedural script its difficult to see what the impact on script v1.1 has over script v1.0.

I would turn to Puppet but it doesn't have the maturity to hook into the vast Azure API.

So I'm going to be a little stuck with PowerShell for the time being.

PowerShell is very powerful but really I just want to define the state of an environment/server and let the software worry about how to get there (a la Puppet).

In order to make the migration from on-premise to Cloud (or use Cloud for DR) we also aren't just talking about VM's.  What about VLAN's, F5 load balancers and all the other stuff that makes up an application stack?

Puppet can define these and if I could potentially use the same manifests for on-premise and the cloud this would be very exciting.  The ability to abstract away implementation into a XML manifest is the dream.

I'm not sure if that's a deficiency in Puppet or Azure (or both).  Either way there is another opportunity for Microsoft here.

So please Mr (or Mrs) Microsoft extend Desired State Configuration so it can cover more than just servers and allow those same manifests to cover private-cloud and public-cloud.  It needs to be agnostic of whether you are using VMware, Hyper-V, F5, Alteon etc.

Write once, run everywhere.
   

Tuesday 28 October 2014

GatesOps

Love-hate
It might be super-unfashionable to say but I have to admit to being somewhat of a Microsoft-phile over the years. 

I've made a decent living from consulting around their technologies over the last 16 years for starters (thanks for paying the mortgage Mr Gates!).

In any dealings with them I've had they have been nice individuals and they have gone out of their way to make me feel welcome.

Not to say I don't have any criticisms.  Quite the opposite.

Anything to do with licensing and Microsoft then just forget it. 

I also hate how far they are behind the curve nowadays - always picking up what others have already started;  GUI operating systems, Cloud computing and Tablets are just a couple of examples.

DevOps?
This brings me on nicely to DevOps.

I'm currently in Barcelona at their annual TechEd event and took part today in their DevOps focus group.  Surprisingly for such a "focus group" that I was really the only one around the table that had any kind of matured DevOps culture at work (and really we are still only beginners tbh). 

Whether this is representative of their customers I don't know. What it is says about their customers I also don't know...

Anyway that is a bit disappointing, I'm always keen to hear from other adopters especially ones that are further down the path than we are.

I digress.

It strikes me that Microsoft are in a fantastic position to sow the market place right up.  They have tools and technologies that range from cradle to grave, from requirements analysis, through to development, testing, provisioning, release, support etc...

Add in that they also cover on premise, hybrid and cloud solutions then you have the vast majority of the application life-cycle and deployment scenarios.

I can't think of any other companies that are in the same privileged position.

However, they aren't embracing this and they are missing a big opportunity.  If they want to be the premier Cloud player then they need to make it easy for people to get from idea in someone's head to the production space in a fast, repeatable and easy fashion.  Anyone can provide a load of compute power.  The real differentiator is being able to facilitate the whole pipeline; laptop to production.

There are a lot of different tools along their chain - the vast majority of which are very Microsoft-centric meaning its difficult (or at least harder) to integrate your favourite technologies if they aren't from Microsoft.

Tools that don't integrate become silos of information and reduce visibility and transparency across teams;  ultimately leading to reduced collaboration.

The sheer number of different tools and solutions makes it difficult to know where to start. 

Do I use TFS or GIT?  Or both? Should I put System Centre in before and get the Ops and provisioning bit right first?  Oh hang on I want to use the Azure pack but I use VMware?

Prompt head explosion.

My boss said something today and he's quite right.  If we knew were to begin with all this stuff and it arrived out of the box then we'd probably buy everything Microsoft have.  It's just too darn hard and too darn complex.  Microsoft seem to expect that you are Microsoft only and are on the very latest everything.  Not realistic.

I hate this term but if there was a "single pain of glass" across your SDLC with the ability to be agnostic of technologies then Microsoft would be onto a winner.

With a Microsoft "something" orchestrating the life-cycle we could integrate what we have right now and eek out the non-MS stuff over time rather than having to blow our data-center up and start again.

Credibility

If Microsoft were really embracing DevOps then they wouldn't be as late to the game as they are and so uninsipring in some of the their technologies. 

For example the release tools in Visual Studio 2013 are so behind the likes of UrbanCode Deploy it's a bit embarrassing.  PowerShell DSC (Desired State Configuration) is so far behind Puppet its redic (as they say in Essex).  Why would I just want to configure my Windows servers when Puppet can control network devices and a whole range of other stuff too?

I think its the same as a personal trainer. 

I wouldn't ask a fat personal trainer to train me.

If you don't live and breathe it then you can't be credible for trying to sell it.

P.S. yes Microsoft I am available to help you put this right in exchange for a small island or something like that (I hear the Bahamas are nice)  :)



Monday 6 October 2014

Service virtualisation

You may have heard of Service Virtualisation (SV) recently.  It's an idea that seems to getting some traction especially in the Continuous Delivery space.

Not to be confused with Operating System virtualisation technologies such as Hyper-V and VMware.

My personal definition of SV is the productionisation of stubbing (granted this is a bit simplistic).

Technically SV can mock too but I didn't really want to get into a discussion of what is a mock and what is a stub :)

Imagine you have an application that is a composite of multiple other systems you have in your company.  It retrieves data from a web service, reads a file from an FTP site and runs a SQL query against a database from a different application and imports the data.

That's potentially problematic as all these systems needs to be available.

What happens if the people running the web services want to take it down for maintenance?  Does that mean you can't develop and test your application?

What happens if you don't have all these interfaces all the way through your environment stack?

What happens if you are reliant on a change that isn't going to be ready for another 10 weeks in one of these interfaces?  Are you going to be blocked from development for 10 weeks?

These are all the kinds of questions that Service Virtualisation hopes to address.

It provides an abstraction of other systems (such as SOA platforms) or even resources such as flat files you might have not have access to.

It can record the interactions between your code and the other system so that it's easy and quick to create the stubbed interface.  Commonly SV applications also provide an interface so you can change the values in these recorded stubs easily through the GUI.

We are looking at implementing something over the next couple of months for one of our key systems.  In a complicated SOA environment with logic managed by different teams and in different geographic locations it's really killing their pace of change.  They are unable to test until a full stack is in place (system test) which can be days/weeks until after they have made the original change.

You really want very short feedback loops between dev and test.  Ideally none at all and the developer can do a lot of testing using automated regression tools on his/her local developer machine.

Service Virtualisation should help them accomplish this rather than the dev, deploy, test cycle that they are currently going through.  It should really increase the pace of change and expect some big cost and efficiency gains.






Monday 18 August 2014

Daddy DevOps

2350073q9huagsinx

I found out this week I'm going to become a Dad.

It's a strange mix of emotions from excited, to terrified, to panicked and happiness.

Not too dissimilar to undertaking a big new project, scoping out how complex it is, getting close to go-live and then delivering it :)

[if any one you say I should use Agile processes for childbirth go to the back of the class - having a baby is definitely Waterfall]

At the moment I have time on my hands, which is probably one of the reasons I blog.  Time also gives you the room to think and ponder.  And because I spend all day, every day talking about DevOps I tend to think about it even in my downtime (and sleep) which is fairly plentiful despite a busy day job.

With my partner away this weekend, bored of playing on the Xbox and all my friends busy/married/parents, I've done a fair bit of sitting around of late.

I got around to thinking about how DevOps might compare to parenthood and I think there are some good analogies there.

My goals of being a good Dad are:
  1. Ensure that the family is provided for
  2. Ensure the health of my child
  3. Ensure that the baby is happy
  4. Ensure my partner is happy and supported
  5. Ensure that the family are safe
My partners goals are probably very similar so we shouldn't have to worry about pulling in different directions too much.

So that's cool, we have 2 different parties (Mum and Dad) who's goals and incentives are aligned and will work together to accomplish them.  The missus might not know it, but she's soon to be living DevOps!

I want to be a good Dad (and partner) and I've been thinking about what I can do as an individual to contribute towards our shared common goals?  And how do I make it easier for my partner to achieve her goals?

Commonly, DevOps teams turn to automation in order to break down the silos between them and meet their common goals.

If I could automate the changing of a nappy that would help!

I'm not sure I can automate nappy changes (I'd be very rich if I could).  Does anyone know if Puppet have a Pampers module?
    
    nappy { '/baby/bottom/':
      ensure => clean,
      owner  => 'parents',
      type => 'huggies',
      source => 'crib',
    }

Alright so I'm going a little off topic here.  My point being that we have two different resources who goals are the same (i.e. ensure the health and happiness of little Johnny).

The stereotyped Dad in the 1920's that went to work and had little to do with bringing up the child is a good example of a silo.

I've gone to work, it's your responsibility to look after the child - it's not my problem


Apart from the fact I want to be involved and I want to help my other half, I'm incentivised to help as much as possible because if I contribute then ultimately the happier we'll all be.

That's DevOps.  Changing your behaviour in order to meet shared goals.

Super excited.  Lets just hope the delivery isn't Continuous.

Saturday 9 August 2014

Motivations of the techy

I spent an hour the other day with the team discussing what their personal motivations were for working at the company.

The team is a mix of contractors and permanent staff.

The idea was two fold. Firstly, by understanding their motivations I would be a better manager and could help incentivise them in the right way. Secondly, by revealing their ambitions they would be motivated to produce metrics which would help them achieve their goals.

These metrics are key to proving the business value of the team and will help prove its success and extend its influence (which are my personal goals).

I assumed that everyone (especially the contractors) would say money is a key driver for them. I was wrong.

Everyone and I mean everyone, talked about company culture as one of the prime reasons they work where they do.

Interesting.

It makes sense I guess. If it was all about $$$ then they might be doing more lucrative and money focused roles. Maybe that's too simplistic? I guess people (including myself) want it all but are more prepared to sacrifice money in the search for happiness.

I've only ever worked in an IT role but I know bankers, brokers and finance types and for the majority of them it's about the bucks.

No real end point to what I'm saying except to say you need to understand what motivates your team in order to incentivise them. That sounds super obvious but it was a little light bulb for me.

Thursday 7 August 2014

Finding the right consultancy is hard

There are lots of different types of IT consultancies; big, small, modern, old fashioned, expensive, even more expensive...

I have to admit a bit of a mistrust when it comes to most large consultancies and in fact I'm generally not a huge of fan of outsourcing come to think of it.

I've seen consultancies (charging astronomical rates), send 6-7 consultants to an engagement when 1-2 were required.  Why?  Ramp up the revenue of course.  How can you be expected to hire a company who aren't incentivised to do the best thing for you?

The aforementioned then tell you need to buy  software X from them in order to solve problem Y.   Again, self interest and a fundamental misalignment of incentives between you and them.

Lots of consultancies come in and tell you what you knew already, along with a Gartner magic quadrant to let you know what you should be doing in the future (its not hard to find out yourself).  OK, so some validation is nice once in a while but the majority of consultancy engagements I've worked on I've have wanted to solve real world issues and not death by SmartArt and PowerPoint.

Just tell me what buttons to press!

Perhaps I should point the finger back of blame back at myself and I haven't led the engagement effectively enough so that I achieved the results I wanted. However, at a squillion pounds a day you'd think they would get to the root of what you are trying to achieve pretty quickly.

I'm tarnishing the whole consultancy trade here and to be honest of course there are some good ones; they are just hard to find.  I tend to work better with the smaller consultancies, although I've had some really good engagements with ThoughtWorks.

Another pattern with the big consultancies is that they are staffed largely by consultants who are good at theory but haven't really ever lived the jobs they are talking about.  How can a consultant that's worked his way up the ladder from university ever tell me about how to structure my unit tests?  Or the best way to structure my automated deployment?  You need battle scars sometimes.

What has this got to with DevOps?  Well I guess if you are an Enterprise who wants to move to DevOps your natural reaction might be to find a correspondingly large consultancy.

Personally, I'd start really small and hire a really good focused team.  Don't engage on some 7 figure consultancy engagement.  Build a small team and grow success organically.

Wednesday 6 August 2014

Why Continuous Delivery doesn't go far enough


My team “owns” (ensures that they are supported, provides best practises) a number of different tools and technologies in the DevOps space including:

  • Puppet Enterprise
  • IBM Urbancode Deploy
  • Jenkins
  • Teamcity
  • TFS
  • SVN
  • Artifactory
  • Selenium
  • Octopus Deploy

...and numerous others.

That’s quite a long list for a Continuous Delivery tool chain (made worse by natural Java and .net divides).

Tools that don’t integrate become silos of their own and encourage a divide between different disciplines (such as Dev and Ops).

It’s important to see the aggregate efficiency of your delivery pipeline and having a thousand different tools involved makes this difficult.

  • How many regression tests do we do a day?
  • What is the ratio between bugs captured in CI against those in SysTest?
  • Is there a relationship between the speed of a build and the number of deployment we do in a sprint?
  • Etc etc

Only by capturing these metrics can we improve our efficiency by highlighting and attacking bottlenecks.

Continuous Delivery is a process to improve the delivery of software change into production. 

For me that doesn’t go far enough.  Are we saying we don’t care about it once our changes have gone live?

We have this nice, shiny Continuous Delivery pipeline to deliver value to the business – usually in the form of increasing revenues somehow.

So, ultimately it’s not really about builds, testing and deployments.  It’s about what impact our change team makes to the bottom line.

For example...
  • What is the relationship between regression tests carried out in Systest and the number of errors that appear in production?
  • What is the relationship between production errors and revenue collected?
  • What is the relationship between how many story points we put in a sprint and the sentiment of our customers?
For me, this is the ultimate expression of DevOps.  We are aligning our delivery capabilities and processes in order to meet shared goals (in this example increased revenue).

I’m currently designing a proof of concept with Splunk to collate all these different types of metrics from the many  environments and tools we have in the delivery pipeline.   We should then be able to understand the different relationships between the capabilities in the team and the decisions we make.

Isn’t it powerful to have data which says to the business “stop putting too much change in a sprint”?  

Or that prioritising new features over technical debt is counterproductive to the bottom line?

I don’t expect that this will be an easy thing to accomplish and we can only infer these types of relationships.  However, the potential benefits are huge.

Tuesday 29 July 2014

Hierarchy in DevOps?

If you are moving to DevOps in your organisation, then bringing together different people from different silos and then expecting them (under their own steam) to magically cooperate and agree on everything is a little unrealistic.

If you are dealing with historical teams and employees - which I suspect most of you are - then breaking down silos is likely to be a significant challenge.

Aligning incentives makes things a lot easier. If both your devs and ops guys are paid a bonus depending on how much "change" they do in a year is a simple way of getting them to agree on whether they should automate releases for example.  That's a little too simplistic but hopefully you get what I mean.

This may well come at the cost of something else though.  The first thing that springs to mind in the last example is quality.

Let's put everything straight into live!  We get paid for it!

Ok so that isn't going to work.

Sometimes, at least to begin with, you need to bring people together and ultimately empower someone to make decisions when one can't be reached through consensus.   I've seen a real danger in DevOps of death by committee.  The more silos you bring together then potentially the greater the possibility of them not making a decision.

You may have heard about this idea of DevOps rockstars.  Guys (and gals) that get DevOps inside and out.  People that understand all sides of the coin (development, DBA, infrastructure etc).

They try and influence through education and persuasion and point people in the right direction.  They help the silos reach the right decisions themselves rather than a top-down "do what I say".

All said and done sometimes this open inclusiveness doesn't always work and someone needs to just stand up and be counted.

So what's my point? 

If you are trying to move towards DevOps then hire a rockstar.  Someone that has done it before.  Someone with a very mature sense of what DevOps is and preferably someone that has done most of the jobs he/she is trying to influence.

Moreover, hire someone who's first instinct is to try and bring people around to their way of thinking.  And lastly, someone who isn't ultimately afraid to put their privates on the line and make a goddam decision.

Saturday 26 July 2014

DevOps isn't automation

Perhaps I'm getting older and more and more like my father.  More intolerant or, at least, less likely to hide discontent or my  view of the world.

Perhaps this is why it makes me an angry and frustrated bear when I constantly hear DevOps being used as a word for automation.

LinkedIn for example is full of DevOps "experts" and forums talking about "doing DevOps". 

Deployment Automation != DevOps

DevOps is a culture of shared goals and incentives.

Deployment Automation may help you to get there if your shared goals are to get things into production faster and in a more consistent state.  However there are lots of other things to consider which also affect this.  If your Release Manager only let's 1 deployment into prod a month then it doesn'tatter how much faster you've done your non production systems. As I've said before DevOps is much more than 'dev' and 'ops'.

I need a little lie down and to get out of the heat.

Sorry for shouting 



Tuesday 22 July 2014

DevOps change challenges

One of the challenges of DevOps in the setting of a traditional (i.e. siloed, waterfall etc) Enterprise is where to begin.

I'm a keen advocate that DevOps isn't just Dev and Ops (see http://enterprisedevops.blogspot.co.uk/2014/07/devbiztestreleasethingyops.html).

With that in mind, potentially you are talking about some fairly radical changes across a wide number of people of various skills, experience and personality types.  I guess it really depends on what your organisation is structured like, the skills of your team and many other factors.  For us however, it's meant some fundamental change.

That fundamental change has been sponsored by some early and small wins by implementing things like Continuous Integration and labelling it "DevOps".  Ok so it's not DevOps but if using that term starts to get visibility and movement within your organisation then I don't think that's a bad thing.

I always we believed we needed a banner for the modernisation of the department to fly behind.  If that banner reads "DevOps" then so be it.

That early small success was really vital to us. 

When we started out we had a look of push back and negativity....

Why are you telling me how to do my job?
You are using the wrong tools
I don't see the benefit
I know more about this than you do
What you are trying to do is easy
We don't have the right people
This is a hipster idea for the likes of Facebook
There are regulatory reasons this won't work


Basically everything under the sun.

That's stressful when you are trying to do the right thing and move the incumbent forward.

I don't think we helped ourselves too much to begin with.  There was too much of my team trying to pick up and change other teams technologies and processes.  Not enough leading the other team to water and helping them implement something themselves.  Lesson learnt.

Don't do DevOps to teams.  Help them do the doing.


More about what changes we've made a little later.



Monday 21 July 2014

Failing early

I'm sure you are familiar with the idea of failing early.

It's best to fail as close to the point of originating a change because it's much easier and cheaper to fix.

When people talk about 'failing early' it's often as an approach to testing code - most noticeably through TDD/unit testing.

However, you can really apply the idea to most things. 

I was thinking today about deployments. 

Wouldn't it be good if you knew that your deployment was going to be work before you even started it? How many times have you had issues because there wasn't enough disk space or you couldn't reach all the servers you needed to?

What we are going to do is a series of pre-deployment checks. I want to fail before a deployment not mid way through.

I think that's a pretty good practise. Automatically test everything before you start with a set of environment readiness scripts. Is the environment in a state that you expect? Does it have the version and configuration that you expect?

If it doesn't fail the deployment and investigate. It will save you all kinds of headaches.

It's also a good idea to kickoff a load of automated tests after a deployment. I'd rather know if there was an issue BEFORE the business/test team get their little mitts on it...

Failing is good. You learn, you fix.

Tuesday 15 July 2014

If DevOps isn't a job title then it can't be a team? Can it?

I blogged earlier how DevOps is cultural idea and not a job title.  Definitely not.  Unimaginable.

So how is it I run a "DevOps" team?   Surely that's a collective of "DevOps engineers"?!

Ok so I appreciate that this contradictory in the extreme.

Or is it......?

My team "Platform Services" supports the idea of the change platform at my company.  The change platform consists of tools such as Jenkins, Puppet, uDeploy, Teamcity, TFS and a few others.

Someone ultimately needs to "own" the tools making sure they are maintained, creating best practices and establishing a roadmap for the future.

My team goes beyond this and helps different change functions adapt to the change platform and are evangelists for things like DevOps and Continuous Delivery.

With DevOps being concerned with breaking down silos its contradictory to have a team that bottleneck of knowledge and skills.  To this end, Platform Services is the silo-buster - we help others pick up the tools & processes and implement them.  

For example, we don't do deployments ourselves but we do sell the benefits to other teams and help them to adopt the tools and processes.

I think this is a good model and helps spread the success of DevOps around the IT function.

It helps that consultancy from my guys is effectively free to other parts of the IT team (thanks to an enlightened CIO & CTO).  

Why wouldn't you take the teams advice?  Why wouldn't you follow the patterns of success established elsewhere?

Go create a DevOps team; just don't let them become a silo.


DevBizTestReleaseThingyOps

The traditional definition of DevOps relates to changing the culture between the development and operation teams so they can reach shared end goals.

Thoroughly bought in that. Great.


But what about everyone else? Dev's and Ops aren't the only people involved in making change.


In my eyes, the term "DevOps" isn't not really just about Dev vs. Ops.

 

It’s about all the functions that come together to achieve meaningful change.
 

You might as well call it DevBizTestReleaseThingyOps.
 

To that, I'm starting to evolve my definition and understanding of this "DevOps" thing.
 

The business and techies (inc testers, BA's etc) should have shared goals and have shared incentives (not necessarily exclusively). Where possible they should be sat around the same desk. The team should rise and fall collectively (there is no "us and them"). The team should manage change from cradle to grave (requirements, build, test, deploy, support).
 

Ok so there is a lot of where "possible" in that paragraph but hopefully you get my point...

Why DevOps isn't a job


Buzzwords.  Gotta hate them.  I love the idea of this thing called "DevOps" though but I come out in a cold sweat when  I hear this word being thrown about with abandon.

My official position is that really there is no such thing as DevOps – at least not as a technical discipline.  I also happen to lead the DevOps initiative at a large financial services enterprise.

So not much contradiction there then...!

The ever increasing popularity of the Cloud has really driven a lot of ideas about new how to create and deploy ‘change, how to collaborate and how to manage mega sized infrastructures.  The massive scale of companies like Facebook, Amazon and Netflix have caused companies to look for new ways to scale their teams, technology and business processes. These types of companies often evangelise this term “DevOps” as part of the solution to their woes.

What they are really talking about though is culture.  Not a collection of funky hipster tools and technologies such as Puppet, Chef and Vagrant.

So, here are my five reasons why you shouldn't ever be able to get a "DevOps" job.

1.    DevOps means producing good quality software and systems that are easy to maintain and quick to change is the responsibility of everyone

That’s not new – just how software should be adopted in the first place.

Its not a developer responsibility.  Or a DBA.  Or a BA.  Or the Operations guys.  Its everyone involved in producing and managing change.

It doesn't just apply to funky new web startups out of Silicon Valley.  It's as applicable to traditional enterprises (including financial services where I work) as it is to those cool Etsy-esque Cloud companies.

2. You can’t get rid of organizational barriers by creating a new organizational silo and calling it DevOps

As companies grow they tend to put people in technology focused "teams".  A database team, or a network team, a development team, a test team, a release management team and so on.

By doing this, these teams often become silos, incentivised in different ways – not focusing on the same end goals.  That’s why you can’t become a DevOps engineer.  Silo’s aren’t fixed by creating a new silo and calling it “DevOps”.

This is why the scale of Cloud has really driven the idea behind DevOps as a “new” discipline.  The idea is to get everyone to collaborate effectively to achieve common goals.  If you want to deploy code to 100,000 servers there are some pretty massive engineering challenges.  Do we update 10% first?  How do we ensure that the change hasn’t broken anything?  Has the login rate dropped?  How do we quickly roll it back if something goes wrong?

3.  One of the misconceptions about DevOps is that it involves automated releases into production.

DevOps is about bringing people together and having the culture in place to achieve shared goals.  Once you have the culture in place, processes and tools become a means rather than an end.  How do developers and infrastructure engineers both ensure that they can test their scripts?  Can they use common techniques such as version control and unit testing?  These are all important questions, but the inherent technical skill sets are not new per say.

DevOps is about breaking down whatever barriers are necessary in order to achieve your goals.  Happy with your pace of change but want to increase quality?  Break down the silos between Devs and Test to get them aligned in a collaborative environment.  Have them work within the same team and incentivised the same way. By sharing tools and techniques, you can avoid developers falling into the trap of throwing code over the wall and moving on.

4. You can’t create a new discipline simply by cobbling up a cool portmanteau

With that in mind, it's not really just about Dev vs. Ops. It’s about the functions that come together to achieve meaningful change.  You might as well call it DevBizTestReleaseThingyOps (more to come on this a later date!)

5. Business culture cannot be changed by creating a new position category nobody really understands in the first place

DevOps is whatever you want to make it.  It’s a culture that extends far beyond technology.