Agile transformation, its unanticipated impact on management and how to understand it

Many organisations are still developing software using lists of requirements with teams working in silos, and only producing occasional releases. However there is a better way simply called ‘Agile’, this is not a prescriptive approach and is certainly not just for developers.

Agile can significantly improve delivery of business value however it’s defined in your situation. And Agile is certainly a prerequisite of any digital transformation.

Adoption of Agile is now mainstream superseding traditional methods, however poor implementations of Agile due to a lack of understanding are very common. One particular aspect that is often misunderstood is the impact that Agile has on the organisation as a whole. Unfortunately Agile is often viewed as just another development method, however the transformation only really succeeds if management is transformed too: ‘Agile management’. Indeed this may be where the greatest value is added.

  • Higher management is aware that it needs to be Agile in order to be more responsive to customers and other stakeholders. They also want improved productivity and reliability
  • Developers understand that it’s more effective to develop quality software than spend time maintaining that resulting from poor quality. And working together rather than waiting for hand-overs from colleagues in other silos

It’s those in the middle who are responsible for managing development who need to be engaged. Not merely with Powerpoint presentations, and possibly Agile games, to explain how the developers will be working in future, but as a valuable part of the transformation otherwise the major benefits of the Agile transformation will be lost. They need to work differently in order to move from the ‘Iron triangle’ of cost, time & scope and upfront long term planning and budgeting to smaller batches, variable scope and frequent re-planning. Agile Portfolio management can add significant value to the business by becoming far more dynamic.

A new way of thinking is required that is appropriate to the complex world of software rather than the (merely) complicated world of construction and manufacturing. Unfortunately it has taken many years for this understanding to be accepted in the IT world. The traditional controls may seem obvious however in a complex situation they can often have the opposite effect to that intended. This  transformation may well entail roles being changed to match the future needs, not just for the development team but management too.

The introduction of Agile puts a floodlight on current practices revealing concerns that would otherwise only manifest themselves much later and at much greater cost. This will prove to be frustrating but these concerns plus the opportunities and the discontinuities between traditional management and newly Agile developers need to be accepted and dealt with as early as possible in order to reduce risks and unnecessary expenditure.

Therefore a culture change is required with teams looking at options to find the best ways forward. And, as each organisation is unique, metrics specific to the organisation are required. This will enable progress to be measured and the way forward to be validated regularly.

To summarise Agile is the most appropriate way to handle most software situations, culture change needs to be a significant part of the Agile transformation and it requires a methodical but flexible customised approach. The change to Agile is not a one-off change but a journey to where the customer will be.

Were you aware of the impact that an Agile transformation has on management? Leave a reply below, or contact me by email.

The Testing Pyramid

Agile has totally changed the focus of testing and introduced automation to support its development practices. I have been involved with software development teams for nearly thirty years. Over that time I have seen many different methods and practices come and go but testing has remained focused around manual testing. That is until Agile software development arrived.

An Agile iterative approach to software development means you have to test all your software all the time. Ensuring what you have just built has not broken a previous written piece of functionality. Agile automates testing at all layers of the application. This approach to testing is fast overtaking the traditional manual approach. Automated tests that previously existed were focused on testing the front end, the most volatile part of the application. They were also written after the application was built therefore not giving the short feedback loop we use in Agile software development. The Testing Pyramid puts a different testing emphasis on each of the application layers and focuses efforts at the beginning of the development cycle rather than at the end.

The Testing Pyramid

Looking around the web you will see various implementations of the Testing Pyramid with different names for the levels and implementing different types of tests. The one I use is the basic three level pyramid – Unit Tests, Service Tests and UI Tests. As with any pyramid, each level builds on the solidity of the level below it.

From technical view point one can look at these three levels as small, medium and large. Small tests are discreet unit tests that use mocks and stubs to collaborate with other objects and are typically written in one of the xUnit frameworks. Service level tests are the medium sized tests and interface with one other system, typically a database or a data bus. Large tests, UI tests collaborate with many sub-system parts they support the end-to-end scenarios.

Personally I look at the levels in a functional way:

  • UI Tests: Does the software work as expected by the end user?
  • Service Tests: Does the software meet the acceptance criteria on the user story?
  • Unit Tests: As a developer does the software work as I expect?

Unit Tests

Unit tests are the foundation of the Testing Pyramid. This is where the bulk of tests are written using one of the xUnit frameworks – for example, JUnit when developing in Java. They ask the question “Are we building the product right?”. When writing software I like to take a Test Driven Development (TDD) approach. TDD is a design approach to development where tests are written first and code written to support the test. There are a number of benefits to taking this approach:

  • High test coverage of the written code
  • Encourages the use of Object Orientated Analysis and Design (OOAD) techniques
  • Allows you to move forward with confidence knowing the functionality works
  • Debugging is minimised because a test takes you straight to the problem
  • Developers take more responsibility for the quality of their code
  • Because it is written to support specific tests, the code works by design not by coincidence or accident

Service Tests

I see Service Tests as supporting the acceptance criteria on the user story. They ask the question “Are we building the right product?”. When writing these tests, I like to take advantage of the Given/When/Then BDD format of the acceptance criteria and use one of the BDD test frameworks, typically Cucumber. I only adopt specification by example that is use real world examples in the acceptance criteria. This approach gives a number of benefits;

  • Assurance that all stakeholders and delivery team members understand what needs to be delivered through greater collaboration
  • The avoidance of re-work through precise specifications and therefore higher productivity

UI Tests

The User Interface is the most volatile part of the application and should not contain any business logic. For these reasons the least emphasis on automated testing should be here. That does not mean there is no testing, I like to automate key user journeys through the system using one of the UI testing frameworks e.g. WebDriver. UI testing demonstrates that all subsystems are talking to each other.

I use manual testing for the look and feel, checking the UI acts as expected and exploratory testing to find those hidden nuances.

Test Delivery

Unit tests and Service tests should be delivered in the iteration by the delivery team. As part of their development practices developers write Unit tests when building the functional code. Ideally a test first, TDD approach should be used.

My conclusion?

The Testing Pyramid inverts the traditional approach to testing. The focus is at the beginning of the development process with the developer taking responsibility for quality of their code. This is a very different way at looking at the problem from the traditional approach where code is handed over to the tester and they are assumed responsible for the code.

The early identification of defects gives two major business benefits;

  • Issues are discovered early in the development process reducing the cost of defect fixing associated with late discovery
  • Issues are identified and resolved early therefore negating the need to postpone the production release through late identification of issues

References

http://www.mountaingoatsoftware.com/blog/the-forgotten-layer-of-the-test-automation-pyramid

http://martinfowler.com/bliki/TestPyramid.html

http://www.agilecoachjournal.com/index.php/2014-01-28/testing-2/the-agile-testing-pyramid/

The #snow must go on – the story of Project: Barry

Recap

This is the tale of a bunch of graduates, their first forays into projects and the trials and tribulations found within. “Project: Barry” is a Graduate project created to map snow worldwide using data pulled from Twitter, and this is the end of it’s saga – catch up on the full story here.

When it snows it pours: late November, two weeks since presentation

During this time, GIS moved to complete the mapping section of the project. The server would comprehend the information being sent to it by Barry, store it in a database and then map it in real time. The viewpoint of the map would be updated to the location of the most recent tweet, creating an interactive experience.

Next we put together a buffer for notifications, making them more useful. This allowed us to stack up tweets over some time, then send one notification with a number of tweets and a selection of locations involved.

Things were moving smoothly and ahead of schedule when we received word that the final presentation would be on the upcoming Friday – not in several weeks like the original deadline.

Whiteout: beginning of December – presentation time

Due to several issues, the date had changed but thanks to the Agile methods, we had a working program ready to be put into production at the end of each week. This allowed us to simply cancel later features (sorry GUI and web crawling!) and tidy up.

This iteration of Barry (v0.1) was at the point of being run on a server, sending information to the database in real time to be mapped and sending buffered notifications to the group – and this was what was presented in the beginning of December. The presentation was open to the entire Edinburgh office and saw a full room ready to interact with Barry and the mapping system in real time.

Participants were able to tweet using the word ‘snow’ and have their name, (embarrassing) display picture and location appear on screen as the map bounced around showing the locations of tweets from around the world. See the example below for a first hand look at Barry in its natural habitat! (You may need to click on the image to start its animation)

Animated image showing snow-spots around the globe on a map of the world

Outcomes

It’s safe to say each Barrista (demonym for those of Project: Barry) was glad to have been a part of the project. Gaining project experience alongside book learning was a huge addition to being part of Sopra Steria’s Graduate programme. Working alongside other streams really makes you think for others and seriously improves your knowledge.

Expanding and developing the project has proven to be incredibly fruitful, both in terms of improving our standing within the company (recognised as being “Those guys who made that Twitter thing”!) and increasing our effectiveness when stepping into the Real World™. Experiencing real deadlines, shifts in scope and late notice changes really gave us a chance to prove our worth and flex our skills.

After the thaw

After the end of the project, the Barristas have since moved to other projects – but the legacy of Barry will live on…

Barry’s real time mapping component is to be utilised to provide visual representation of activity on a potentially company wide basis… so stay tuned!


This is just one example of the innovative projects Sopra Steria graduates get involved with. If you are, or someone you know is, graduating in 2016 and looking for exciting opportunities, why not take a look at this year’s Graduate Recruitment programme.

Scaling Agile the Spotify way

The first book that I recommend anyone read about Agile is “Scrum and XP from the Trenches” by Henrik Kniberg.

Kniberg has a relaxed, informal and straight-forward way of explaining how to do Scrum, which leaves you in no doubt that the author has actually been a Scrum Master himself.

So naturally, when I heard that Kniberg was working at Spotify and was experimenting with some ideas to scale Agile beyond the team level, I was very interested in hearing what he had to say.

(Image courtesy of Spotify)

Squad

The squad is the central unit of Spotify’s scaled Agile approach. Squads are multi-functional, self-organising teams that focus on a particular aspect product.

Tribes

Tribes are collections of squads which share a focus on a certain product area. They meet regularly to showcase what each of the squads within the tribe is doing – both product work and the result of hack days (see below).

Chapters

Chapters are groups which are formed across squads made up of people with similar skillsets (e.g. web developers, testers, designers, UI developers). Each chapter has a senior member who, as well as being a member of one of the squads, acts as line manager to all the other members of that chapter.

Guilds

Guilds may have membership across more than one tribe and are not necessarily restricted to one subject area. Examples of guilds given by Kniberg are web guilds – which may include web developers, designers and testers and an Agile coaches guild.

The non-incidentals

Squads, tribes, chapters and guilds might be the most obvious features of the Spotify approach to scaling Agile. But some of the others aspects of the scaled Agile set up – which might seem incidental – may well be the most important.

Tribes sit together and have good facilities

In Spotify, squads all sit together, and wherever possible, squads in the same tribe are together in the same building. The squads have control over their surroundings. They have kitchens and breakout rooms and most of the walls have whiteboards on them.

Hack days

Each squad takes one day in ten as a “hack day”. The squads themselves can decide whether they take these singly or group them together to have a whole week. The purpose of this time is to give members of the squads the opportunity to learn about new technologies and communicate them to other members of the squad. It is also an opportunity for innovation.

The release team supports the Squads

Rather than having a separate release team which handles the release of software, the goal of the release team is to allow the development teams to release their own software.

What about architects?

The architecture of Spotify is service-oriented, with over a hundred systems making up the whole site. Each service has a system owner, and the most important systems have a developer/devops pair as system owner.

The System Owner is not a bottleneck or ivory tower architect. He does not personally have to make all decisions, or write all code, or do all releases. He is typically a squad member or chapter lead who has other day-to-day responsibilities in addition to the system ownership.

What about product owners?

There is a product owner associated with each squad. As Kniberg says:

“The PO is the ‘entrepreneur’ or ‘product champion’, focusing on delivering a great product.”

Although this isn’t discussed explicitly in the articles on Spotify that I’ve read, it can be imagined that the product owners themselves would be organised into a chapter – with their own chapter lead.

What is your experience in scaling Agile? Leave a reply below, or contact me by email.

How’s our driving? Software development using Empirical Process Control

Empirical [Adjective]

  1. Pertaining to or based on experience.
  2. Pertaining to, derived from, or testable by observations made using the physical senses or using instruments which extend the senses.
  3. (philosophy of science) Verifiable by means of scientific experimentation.

(definition taken from en.wiktionary.org)

In engineering there are two kinds of process control. Defined process control deals with processes which are repeatable and fully understood in detail. Empirical process control deals with processes which are either not fully understood or are naturally very variable.

One of the key insights of an Agile approach to software development is that building software is complex and unpredictable business and so it is best handled using empirical process control.

So what is Empirical Process Control?

To achieve empirical process control you need three things: transparency, inspection and adaptation.

Transparency

In order to be able to control a complex process, you need transparency – you need to be able to see what’s going on.

Driving a car is a complex control process. When you’re driving you need to be able to see through the windscreen.

Inspection

When you’re controlling a complex process, like driving a car, it’s not enough to be able see what’s going on.  You have to actually look.  You can have a nice clear windscreen but if you’re not looking out of it, if you’re not paying attention to what’s going on, then you’re still not going to have a very effective control mechanism in your car.

Adaptation

You need to able to see out of the windscreen (transparency), you need to actually look out of the windscreen, but then you also need to modify what you do in relation to what you see.  See an obstacle in front? Slow down.  See an empty road? Speed up.  See a sharp bend? Slow down and turn the steering wheel.

When you talk about driving in this way, it’s obvious how the pillars of empirical process control – transparency, inspection and adaptation – apply. But how are they achieved in software development?

Let’s take a look at how empirical process control is realised in one of the most popular Agile frameworks: Scrum.

Transparency and inspection: stand-ups, showcases and retrospectives

The basis of transparency in Scrum is the ‘stand-up’. This is short meeting of the team every day. People involved in the project say what they’re going to do today and what they did yesterday. They also highlight any problems which are preventing them from doing what they intend to do.

In Scrum there is a role of product owner.  The product owner acts as a single representative for the organisation that wants the software being developed by the development team. At the end of each sprint (a short, fixed period of time for which work is planned, typically two weeks) there’s a meeting called the ‘showcase’ entirely for the benefit of the product owner. In the showcase the team demonstrates working software that its developed in the recent sprint. The showcase makes transparent to the product owner, how the team is progressing.

At the end of each sprint the team runs a ‘retrospective’ where they ask themselves, what went well, what didn’t go so well. They come up with ideas for things they can try to improve the team’s performance. In terms of empirical process, the team is inspecting their performance and another way to think about this is that at the end of each sprint, the team members asks themselves and their passenger – “how’s our driving?”

Adaptation: making changes in response to what you see

Planning in Scrum happens not just once, but frequently: at the beginning of each sprint. This means the product owner can adapt what the team is going to do having looked at working software. If the software is right, the team can carry on going in that direction. If there’s a problem, the product owner can modify what the team should do next. By running retrospectives and acting on their findings, the team is also adapting.

What are your thoughts? Leave a reply below or contact me by email.

There’s no business like #snow business – the story of Project: Barry

This is the tale of a bunch of graduates, their first forays into projects and the trials and tribulations found within. This is the first of two blog posts – which can, and will be, officially regarded as a saga – so keep an eye out!

Outline

“Barry” is an application developed to track snow using Twitter, alerting users through phone notifications and mapping the information worldwide in real time. Barry has been used to track weather movements and is accurate when compared to traditional reports.

And some stats about the project:

  • Five graduates from Sopra Steria’s October 2015 intake
  • Three Java, plus GIS and Project Management streams
  • Seven weeks to compile the project as a side task
  • Two presentation dates with open invites to everyone in our Edinburgh office

Story Time

Outlook optimistic: mid October, end of week two with the company

Post-induction, post-presentations and post-welcomes, the Java graduates were looking for something with a little more bite. During one of our first meetings with our stream lead, it was tentatively proposed that we create a programme to grab information from Twitter and send a notification to your phone. This would become “Project: Barry”.

Naturally, the Java grads were keen for the opportunity to put our book knowledge to the test and stick a figurative toe into the sea of development. We decided to follow the general idea and the topic of snow was chosen; the temporary name of “SnowStalkers” was toyed with and we began putting our heads together.

The notification system came first, starting with the software Pushbullet, which is used for pushing notifications between devices. We developed a cheap and cheerful prototype and with that in place, we set our sights a little higher.

Clouds building: start of November, one week of work on project

We decided to open the doors of the project to other streams and in a quick series of conversations, we simultaneously increased and slimmed down our workload. We brought in a GIS graduate (Geographic Information Systems – it’s OK, I had to ask too), to expand into mapping the data we were gathering. Alongside this we picked up a Project Management graduate (yes we have those, and yes it’s viable), to whip us into shape and bring more structure to our project.

This was a big step towards making this idea into a serious project, as it was originally Java only – bringing in others allowed them to get more experience and working with other knowledge bases only improves your own learning. This is when Project Barry got its name; with a slip of the tongue, our GIS grad Brian was dubbed ‘Barry’ for the day and, as they say, the rest is history. We began structured meetings with agendas and began putting together our own scope – putting down features we must, should and wanted to have implemented.

First flakes: early November, under two weeks since the project expansion

Twitter integration working smoothly, mapping prototype running, and notifications flying – it was time for the first presentation, six weeks since starting the job. At the time Barry (v0.0) would grab one tweet based on snow and send a notification to the group. The presentation was to a few members of the Java team at Edinburgh, with all major points covered in under seven minutes (Pecha Kucha style).

With the first presentation completed, we made the change to Agile Sprints. We put together a trimmed feature list which at the time included:

  • Automation (continuously running without human input)
  • Twitter streaming API (similar to automation, but for Twitter)
  • Mapping (do you have to ask?)
  • Web crawling (grabbing information from websites linked in tweets)
  • Graphical User Interface (an interface to enter data)
  • Notification buffering (collecting tweets to send fewer notifications)

The aim was to implement one feature per week – taking us up to our apparent week 12 (since starting with the company) deadline with a week to spare. Java graduates brought automation and Twitter streaming into fruition soon after the presentation – Barry was continually running, pulling down tweets in real time and sending (far too many) notifications.

Next time on Project: Barry…

Read about the fate of Barry – its actionromance improvements, the twist in the tale and lessons learnt.


 

This is just one example of the innovative projects Sopra Steria graduates get involved with. If you are, or someone you know is, graduating in 2016 and looking for exciting opportunities, why not take a look at this year’s Graduate Recruitment programme.

Four reasons why Scrum is so popular

A 2013 survey by the company Version 1 showed that over 50 percent of all software development projects which are being managed using an Agile method are using Scrum. A further 15 percent or so are using either a “Scrum Hybrid” or “Scrumban” (a Scrum Hybrid).  This means that around two thirds (65 percent) of all the Agile projects that were taking place in 2013 were using Scrum. In the 2014 survey, this figure had crept up to 72 percent.

Why is Scrum so popular? Many of the other methods (DSMD, XP, Kanban) seem appealing and strong claims can be found for their success.  So why, in reality, is one method dominating? Here’s why I think Scrum is so popular.

Reason #1 – Scrum has a role dedicated to making Scrum work

The main purpose of the ScrumMaster is to facilitate the Scrum process. Actually, the ScrumMaster has two main purposes, the second being to remove impediments to progress that are identified by the team.  But the best way to identify those impediments is to facilitate the Scrum process.

Reason #2 – Scrum tells you what to do

Scrum isn’t an analysis, Scrum is a practice, or rather a set of practices. You too can start doing Scrum right now.

The most popular practice is the “daily Scrum” – or “stand-up”. This is a short meeting where everyone in the team says what they did yesterday, what they’re doing today, and, crucially, what’s stopping them – their impediments. This is one of the main ways the ScrumMaster identifies impediments. And then it’s his or her job to get them removed.

It really is that easy to get started, but to do Scrum more completely, the team needs to plan work in short, fixed periods of time (referred to as “Sprints”). They reflect at the end of each Sprint, how well they did, and what they would change (in a meeting called a ‘retrospective’). Finally, the team demonstrates working software to a representative of the customer.

Reason #3 – Scrum explicitly involves the customer

In Scrum the interests of the customer are represented by the Product Owner (PO). The PO agrees with the team at the beginning of each sprint which work should be done in that sprint. At the end of the sprint, the PO reviews the resulting working software and provides feedback. If things have changed since the beginning of the last sprint (priorities, understanding, technology, politics…) these can be incorporated into what’s planned for the next sprint.

Reason #4 – That’s it

There are lots of other techniques that you can use with Scrum – user stories, continuous integration, burn-down charts, burn-up charts, estimation of tasks in hours, complexity points, etc. But NONE of these are required.

Simplicity and clarity make it easy for teams to get started using Scrum. And, in my opinion, that’s why it’s the most popular Agile method, by far.

Mastering Scrum (rather than calling yourself a ScrumMaster) is a much more difficult task, but the only way to get there is to have that first stand-up.

What is your experience in Mastering Scrum? Leave a reply below, or contact me by email.