Craig Strong

When first introducing scrum into the workplace some time ago, many people had strong
views on what agility meant and certainly had some strong opinions on how Scrum should
be implemented. Some of these views created divides between teams, people and
processes. After quite some time re-factoring processes, communicating and refining
techniques everything started to take shape. We started getting to grips with TDD, had a
continuous integration server set-up, introduced product backlog tools to manage groomed producted backlogs and really
progressed well with the quality in writing user stories. In hindsight it was clear that what
we started with was far from what we needed and ended up with.

Although we seemed to follow the standard practice well and could see progress all
around, the first and largest project we started which was the project we introduced scrum
to the company with, was late and far from agile. Management had little confidence in the
project and were concerned that every time a change was made, bugs would surface into
the product backlog. Not only was the management concerned that change was
problematic, but confidence was so low that there was a fear that introducing new features
would put the launch of existing features at risk due to the possibility of new bugs
surfacing. This was not a good place to be for anyone!

Looking at this project many things went wrong, but most had been fixed. After a few root
cause analysis exercises we looked at the main symptoms. It was quite obvious the main
reason it was late was because the project developed an incredibly long tail of bug fixes.
“How could this be?” we asked ourselves, we have TDD in place, we have peer reviewing, a
pretty decent “done done” list and a few other XP practices in place. We knew bugs would
always surface as this is the nature of software, but this was a case where you fix one bug
and introduce three more. I began investigating the problem further, this was clearly not
the way to do it! I regained my title as Scrum Master in the problematic project and started to observe the team.

After a few hours I started to see the root cause of the problem. When the team were
tackling the user story they would notice bugs. However the bugs were “not directly
related” to the user story being worked on. As a result of this, the team would look down
the backlog and see a user story which covered the area where the bug surfaced. Then
they would negotiate with the product owner not to fix the bug in the existing user story, but
to leave it there and fix it when that user story surfaced. Apparently this would be quicker and it was a way to by pass the failing tests and negotiate a way around the “Done Done” list. After seeing this the penny truly dropped.

The problem with the above was that we were using an agile process with a waterfall
approach to the product backlog. After each retrospective taking this approach meant the
product was not shippable and the sprint reviews would introduce new directions for the
product. What this meant was that the user stories in the backlog which would fix the accepted bugs, may be removed, de-prioritised or never tackled. Therefore this meant that additional user stories had to be created to solve the debt which should never had been
there or that velocity would continuously decrease due to an ever increase bug debt.. The value of these user stories was very low. This exposed the fragility of the process and  also enforced why it is so important to ensure each user story is shippable by adhering the a clear, well defined “Done Done” list.

Following the identification of the above, we came up with a strategy to resolve the issue.
The Scrum Master (Myself), had to enforce a strong bug stance in accordance with a more
detailed “Done Done” list and flush the bugs out. Initially velocity went down considerably, but it was a
true velocity. After a few sprints not only would the bugs be flushed out, but the product backlog was much healthier. Managementʼs confidence started to rise again and although less was being delivered, what was delivered was shippable. After just a few sprints velocity started to rise and the long tail of bugs added to the product backlog started to diminish.

To conclude, we learnt at our expense that quality is a metric that is controlled by the team
and expected by the client. Putting off today what can be done tomorrow is not the path to
agility especially when this is quality. Doing so will increase the likelihood of project failure and make the product fragile.  The product backlog is a living entity subject to
constant change. Unlike waterfall the path is not linear and if it is, the path is not agile.

It surprises me how many people don’t fully appreciate what cross-functional teams (CFT) are or what they aim to achieve.This ends up creating problems, increasing costs, lowering moral and hindering progress. However outside the business environment many people fundamentally know the principles of a CFT, particularly anyone who has an interest in sport. The same people are sometimes incredibly passionate team players and go over and above to help their local soccer team achieve, even if this means playing in different positions just like the generalised specialists of Agile. However in a working environment, working with others outside their department is a chore and often avoided by some. Therefore why is it difficult to solve the same problem in business environments. Before we look into more details of CFT’s lets look at an official definition of what a CFT is :

“A cross-functional team is a group of people with different functional expertise working toward a common goal. It may include people from finance, marketing, operations, and human resources departments. Typically, it includes employees from all levels of an organisation. Members may also come from outside an organisation (in particular, from suppliers, key customers, or consultants).” – http://en.wikipedia.org/wiki/Cross-functional_team

If you were to slightly reword the definition above, you would end up with a close definition of what a soccer team might be :

“A soccer team is a group of people with different skills working toward a common goal. It may include players in attack, creativity, defence and man management. Typically, it includes players from all levels of physical and mental ability. Peripheral members may also come from outside the team (in particular, physios, trainers, fans, or sponsors).”

Although that is a pretty limited comparison, its pretty obvious to see that they are both basic examples of a team. As simple as this sounds, in business the idea of a Cross-Functional team is complex and for some people unachievable and/or undesired.

Good Team Work = Good Results

When creating a simple product no one skill is above all others. It requires team work, collaboration and good communication. If you look at any product that inspires you, I am pretty confident it impresses you because the people behind the product worked well together. You only have to look at an iPhone to see that the designers, developers, engineers, marketing, sales and other areas of the business worked together to create a single product.
Creating cross-functional teams increase the potency of a product in the market place. Staying with the soccer analogy you can see the power of teams in almost every competition. If anyone remembers Greece won the UEFA Euro 2004, not by having the most expensive players, or the worlds best striker, but by having the mentality of a team and a desire to win as a team :
“The Greeks, dismissed as rank outsiders before the tournament with bookmakers giving odds of between 80–1 to 150–1 for them to win, defeated some of the favourites in the competition including defending champions France and hosts Portugal, who Greece beat in both the opening game of the tournament and the second in the final.” – http://en.wikipedia.org/wiki/Greece_national_football_team

Functional Units vs Cross-Functional Teams

One of the main arguments and a valid one which prevent cross-functional teams existing is the benefit of functional units. A functional unit is a group of people grouped by a similar function for example developers with developers, designers with designers etc. The benefits of which mean that as a professional unit, strengths and processes can develop well and they can communicate with strength and confidence on how to develop further. I have no doubt that grouping people by skill will have it’s advantages. The problem is that one skill doesn’t make a product.

The sweet spot of team structure is to have the teams arranged by a purpose (product), but to have the communication of functional units.To solve this problem you need to ensure communication is managed well and is very strong. That means that efforts have to be made in order for this to work. Again referring to the soccer analogy this happens quite nicely. Although you have goalkeepers and strikers in the same team, they don’t undertake exactly the same training all the time and come together at set times with purpose. Therefore strikers will train with strikers and develop the skills needed to be a striker, but play with all the other positions for the match.

In business it’s not easy to split people up like the training example above. People have computers, desks, commutes etc. Therefore how can this be achieved ? Well one thing we have tried in work with some success is to have scheduled “Round Table” meetings. These are set times where all developers, testers or designers from different projects meet and discuss their problems, achievements or proposals. This seems to have had a positive affect and are time boxed so they don’t interfere with production metrics. There are many other methods which could help achieve the above such as weekly, fortnightly meetings (limited time); where each team can tell other teams what it developed and what it plans to develop in and how (like a large scrum meeting), good documentation, set training etc. The point is communication processes and time need to be recognised and set up just as you would schedule training in a soccer team. Unless an effort is made to account for the problems with cross-functional structures, the negatives will be problematic and a successful team will not materialise.

Fielding A Cross-Functional Team

If I was a soccer coach and was dealt with the task of fielding a team for the next match how do you think the following tactic would prevail for 90 minutes :

Isolated Functional Units

Looking at the “Attacking Tactic”,I’m pretty confident although my team might score some goals, but the opposition would score more. Looking at the “Defensive Tactic”, I’m pretty confident that I can limit the amount of goals conceded, but my team probably won’t score many in the process. With both examples, I am also pretty confident the players, coach, directors and the fans would be a little concerned with my approach and probably question my capability as the teams manager. It’s pretty obvious the team in both examples is off balance and doesn’t contain structure, limiting the chances of success. However obvious in both these examples, businesses often adopt these structures. They have teams which consist of isolated skills and capabilities and then shut them off to the other skill sets, forcing them to work in isolation physically and discourage communication.

If I was however to field a team for match day which combined all the skill sets which had a balance between defence and attack, with everyone working with a common structure and goal, I would increase the teams chances of success. To put this in perspective, match day would be developing a product in the business sense. The following very simple example demonstrates a teams structure from a business point of view :

CFT Soccer Zones

Hopefully the example above clearly demonstrates why and where different skills are needed. (I know it’s very simplified).

If its that simple, why doesn’t everybody do it ?

The analogy in this post over-simplifies a sometimes very complex business structure. For instance football team is 11 players, where as a business can be thousands. That being said, the basic strategy is relative.

There are countless reasons why this doesn’t get implemented in many companies, some more valid than others. However in my opinion some of the most common reasons why this is difficult to implement include communication, responsibility, ignorance, confidence and trust. Let me explain :

Communication

Businesses are often structured by departments, not by products. Naturally the environment is restrictive which will reduce communication and if your are co-located or international you need to allocate time and buy tools to invest in communication strategies. Being structured this way also hiders incentive. Why should Sales communicate with QA for instance? They don’t need to, “we work over here, they work over there and we do different things”. Working in isolation reduces the visibility of the big picture which is that everyone is on the same team. Instead there is more loyalty and effort shown to the each of their own departments. Waterfall project models emphasise this in their structure, when phase 1 is finished move it to another department for phase 2 and so on. This approach is fine, but often as a project moves through the sections, the vision can be lost. In a lot of cases as the project moves on through the departments, the motivation, incentive and understanding can diminish as well, causing the business to be less responsive.

Responsibility

With skills/departments not communicating well with each other, responsibility and ownership sits comfortably within the departments. If engineering make a mistake, it’s not the responsibility of marketing right? When you group people by profession and isolate the skill sets coupled with poor communication of course, a blame culture is encouraged which furthers the separation between functional units. Unless there is a common responsibility between the functional units to communicate, they will drift further apart hindering the benefits of working as a team and therefore affect the product. Organisations who don’t encourage functional units to take responsibility help further the divide at their peril. However most organisations don’t identify, encourage or monitor this responsibility as such. Yes you do have project managers who work with many departments, however project managers can work with functional units in isolation which is no benefit in getting functional units working towards the big picture. In a soccer team, you have a captain, coach and manager all making sure the team is communicating, in business you don’t always have these people.

Ignorance

If your skills and interest are limited to your department, domain or comfort zone and you are doing what is asked, why bother helping or bother spending time working with others not related to your craft? Do people from IT really need to care what Marketing do or vice versa? Of course the answer is yes to the previous question. Ignorance is more than often a choice by an individual. It’s very easy to do other people’s jobs everyone will have you think. Breaking down the barrier of ignorance can have huge benefits not just for others, but also for yourself. If one understands how others work they might structure their work differently which could increase overall performance and efficiency. Taking the IT/Marketing differences as an example, for IT to receive a change request through from marketing to a preview environment it might benefit marketing to know that each chance to a static page of content takes 3 hours of testing and 2 hours of deployment for instance. Knowing this might ensure marketing provide copy in a better state before passing it to IT, which in affect could change 10+ hours into 5 hours making the company more efficient by saving time and money. I have no doubt that Ignorance like this is costing companies millions of pounds per day!

Confidence And Trust

Yes believe it or not this can play a big part in poor communication in business. Functional Units often have leaders. Some leaders like to lead and like to be seen leading. Where functional units isolate persons and teams away from the big picture which is the cross-functional team working towards the same goal, leaders with inflated ego’s will not want to communicate. If you think you are doing a great job you might not want to give other leaders from other departments any insight to how you work as it will expose your processes which may involve weaknesses as well as strengths. If a player a soccer pitch holds onto the ball all the time and doesn’t pass, you pull them onto the bench quickly even if they are very good individually. In business who does this and is it as easy to spot?

So Now What ?

Well this post explains a simple analogy of cross-functional team structures. Obviously businesses are complex, but the same approach can be applied. For cross-functional team structures to work, decisions need to come from the top and effort need to be made to encourage the structures/tactics to have a chance to work. Analysing operations and refining processes will go a long way on setting up cross-functional teams, but just like soccer teams, they need management, structure and direction. Unless the vision is shared, the functional units have motivation,ability to communicate and people see themselves as team members and not the team, it will be difficult if not impossible to implement. Creating cross-functional team structures is a culture change as much as it is a structural change. Such changes take time, effort and planning which deter most who don’t wish to undertake long term thinking. However implemented well just like Greece winning the European Championships, you can generate more output with less input and increase your chances of success. Not all companies can be structured like this, but I’m sure the one’s who do will not look back in most cases.

If you have any feedback on this post or your company structures and experiences, please send me your response.

I am fortunate to work for a company that is growing extremely fast, right? Although that sounds great, company growth can be a very destructive force indeed. With growth you risk losing control of your every days tasks which you take for granted, such as assuming everyone knows the best practices and procedures, direction of the team, where to go for help, quality, limited interruptions etc. Rapid growth in teams can be a very time consuming and a very uncomfortable experience which can be costly to the business. Brooks’s law rings ever true : “Adding manpower to a late software project makes it later”.

A very inspiring book I read a few months ago that I highly recommend to anyone interested in a development mindset is “Rework”. A passage within this book resonates as one being told by very wise men who have experienced the problems with rapid growth :

“The right time to hire is when there’s more work than you can handle for a sustained period of time. There should be things you can’t do any more. You should notice the quality slipping. That’s when your hurting.” – (Rework: Jason Fried and David Heinemeier Hanson)

That said when you only expand when absolutely necessary you should be aware that extra attention is needed in the right areas. Using Scrum, we advocate smaller teams with better communication. This helps people progress quicker and teams to be become more independent and responsible, which ultimately leads to increased productivity.

When we start new projects or reshuffle teams, we try to keep sprints short (1week) until the team settles with each other and then open it up to 2 weeks. The main advantages of this is that it puts pressure on the Product Owners to ensure that the tasks are small enough to tackle over a shorter time frame and therefore increases the quality of user stories, but it allows us to have weekly retrospectives which gives the team time to flush out problems quickly. The retrospectives quickly identify the bigger problems and these problems can be identified and removed quickly as a result.

To put this in context, recently a team I was managing went through a lengthy process of retrospectives and a small but obvious problem began to surface. The team although situated very close (split between 2 rooms divided by a glass wall with an open door which I sat between) was starting to highlight a divide. I believe this was partially based on skill set differences and partially based on new members joining the others. However one obvious problem was that the divide was being reinforced by using instant messaging tools rather than face to face communication. Divides like this must be identified and dealt with sooner rather than later as the problems can escalate very quickly.

To tackle this problem I thought of a principle of the Agile Manifesto which was :

“The most efficient and effective method of conveying information to and within a development team is face-to-face conversation”

I instantly made changes to factor this in. I moved my desk out of the way into another room and made sure the entire team sat next to each other with no exceptions. I explained to the team why this was done in line with reinforcing if an individual does a great job, but the team as a whole fail, everyone fails within the team and vice versa. I only look at the success and failure of the team not the individuals within (there are other processes involved with this outside the scope of this post).

Following a very productive retrospective and now with an environment which eliminated the need to communicate using anything other than face to face communication, there were instant benefits. I am not exaggerating when I say there has been an increase in productivity close to 200-250% seen within the first week and this velocity has been sustained! (There were other issues raised within the retrospective which attributed to this growth as well, but without doubt co-location was a major contributor).

This to me clearly demonstrates the impact of small often unseen problems within production environments. When teams expand, an effort should be made to ensure the teams are working close and communicating well. You don’t need to physically be involved with the teams as you will get in the way, you need to clear the path and facilitate the platform to highlight problems. I believe excellence is a living entity that you need to keep striving for and analysing. You should be aware of the impact of environmental changes and aware of the dangers/benefits when expanding or moving teams.

Product Management or Product Ownership, depending on how you title the role embraces a role which should be filled by a person who acts as a visionary. A visionary is not a person who acts as an oracle of all things wise, but someone who utilises all the knowledge around them and makes what is perceived to be the most valuable decision whilst balancing risk. Depending on the person, he/she will determine how the product will evolve to be the best it can be in the market place (sometimes doing less is more valuable). This involves communicating a great deal with ALL those involved with the product and/or market including, but not confined to engineers, designers, UX specialists, stakeholders, end users, competitors and so on. Gathering knowledge can be intentional such as creating user group sessions, analysing successful products within the market place, but it can also be unpredicted. Rapid release cycles and analysis of customer feedback can steer products and lead to rapid innovative techniques which embraces customer centric development. I believe this to be the way in which Google often approach their development and their success needs no introduction.

It’s human nature to want to categorise everything, but in my opinion it’s unwise to restrict oneself to a methodology such as “Product Centric Development” or “Customer Centric Development”. Two favourite quotes of mine highlight the balance of product management and the role of a visionary.

“If I asked my customers what they wanted they would have said faster horses – Henry Ford”.

“People don’t want to buy a quarter inch drill they want a quarter inch hole – Theodore Levitt”

I personally feel that a product manager/owner should find the balance between Product Centric Development and Customer Centric Development to create great products. Sometimes this means you will have to make decisions which solve the core problems of customers where the “how” is the key innovative feature. By involving all in the brain storming sessions and continuously analysing the market from all angles, the direction of the product should discover new ways of doing things whilst successfully meeting the customers needs. The engineers will highlight whats possible with current technology, the stakeholders will facilitate the resources required and customers will tell you what problems they want solving.

As I have been developing for the web for over a decade, I have become comfortable with tools and technologies which have helped me get the job done. Some of these technologies I have seen evolve and progress into what are now essential and very powerful solutions. I have established an affinity with many of these tools and without doubt the Apache web server qualifies as such a technology for me.

Using Apache over the years as my main web server, I have produced all types of software solutions for many diverse businesses within many different industries.  Apache has served me well and I am an advocate of the old saying ‘If it’s not broken, why fix it’ (within reason). Until recently that is….

Over the past few years in the distant background I have heard mention of Cherokee as being a viable alternative to Apache, with some claiming it offers benefits. These have been whispers registering in the distance, but didn’t sound interesting enough to pursue and invest my time as Apache was doing it’s job well. However recently these whispers became a little louder following a few conversations with a work colleague singing Cherokee’s praises. What caught my attention most from the conversations wasn’t simply that Cherokee was faster, but that Cherokee was much easier to use and quicker to configure. As I don’t like ‘shaving Yaks‘, my approach was that if I start hitting obstacles, I’ll leave it for another day which contains more than 24 hours; after all Apache is working well for me.

My local lightweight development environment consists of a Linux VM (Fedora) which I operate using VirtualBox. I deliberately reduce my VM’s resource allocation such as setting my RAM allocation to 512mb on my VM to encourage good code and to identify memory leaks. I have the other usual stuff setup such as Zend Framework, xdebug, MySQL etc. This was the machine which was going to carry out the test switch from my trusted Apache to Cherokee.

I disabled Apache and installed Cherokee using the supporting well written user guide. Within minutes Cherokee was up and running and there we NO problems! I was expecting some configuration hurdles as per usual, but nothing. My sites were running as if I was on Apache and there were no noticeable differences. I configured the logs to act as the same as Apache and all was set.

As I delved deeper into the setup guide I was pleasantly introduced to Cherokee’s administration interface. This was a pleasant surprise as I’m so used to hard coding configuration information into httpd.conf. This interface presents configuration options for all the usual server settings such as ‘Virtual Servers’, ‘Directory Sources’, ‘Logging’, ‘Security’ etc. The beauty of this is not that it’s simply a little prettier than the command prompt, but it’s quicker to use. You can configure your server settings simply and quickly which are two good properties to have on your side.

Although I was impressed so far, the remaining challenge that Cherokee must live up to for me is it’s speed advantages. In the spirit of keeping things simple and to get a loose overview on performance advantages I thought I’d simply use ‘Zend Controller’ which is bundled with ‘Zend Server CE‘ to test how many requests per second both Apache and Cherokee could handle in turn upon my humble local virtual machine. From these tests I obtained the following results :

Apache Web Server Results

Apache requests per second on local VM

Cherokee Web Server Results

Cherokee requests per second on local VM

As you can imagine I was quite impressed with what the results presented. The results roughly show that Cherokee could handle 2.5 times more requests per second that Apache! That is no small margin!

What does this mean for me going forward? Well the first things that went through my mind understandably were cost and time savings. Potentially this could reduce the need for more hardware. Less hardware means less purchase costs and less maintenance time. Obviously there are other factors to consider before jumping the gun, but Cherokee certainly has my attention now. I will definitely be including Cherokee in my future plans. An exercise well worth the time.

If anyone reading this has done the switch, please feel free to reply to this post with any feedback

Related Link : alobbs.com