The Devil of Distributed Development

There is gold in them thar hills!

Over the last six months, I have been working on a model for distributed, Agile Scrum development. This has been in response to on-going issues on software projects where development (front end, back-end and testing) is done in an offshore office. The issues have been around areas such as relationships, quality, communication, profitability and client satisfaction.

In addition to these common trouble hotspots, I work in a creative led Digital Agency which, due to the nature of the beast, presents even more challenges around establishing dedicated teams with the right skill-sets, consistent delivery methodologies, adhering to best practices and educating internal and external stakeholders into rationale behind them.  The last point around education is probably the biggest overarching challenge since if everyone knew what the right thing was to do, they would do it.

The best example of an education challenge is explaining the tax of distributed development. From a financial person's perspective distributed development is a no brainer.

"What? You mean one expensive onshore developer can fund almost five cheap offshore developers. These margins are massive! Well get on the phone to any country with a slightly f*cked economy and round up developers. Ideally it should be a country that has been oppressed by a communist regime guaranteeing optimal economic exploitation combined with a decent level of education. Yeehaaa! There is gold in them thar hills!" said fictitious Financial Person.

It is hard to rail against this jubilation with a reality check that the tax on distributed development can have a greater impact on the overall ROI of software projects than if you were going to deliver the project with an on-site, trusted development team. It is especially hard when this tax doesn't live as a line on a spread-sheet and can easily calculated into the project cost. And therein lies what I think is the biggest challenge of distributed development: educating the commercial reality of distributed development.

What makes presenting the additional overhead and ultimately cost required for distributed development even more challenging is that distributed development can work really well. If you have a good team who trust each other, they can do great work no matter which geographical location they inhabit. The internet can facilitate communication on top of an established, effective team. But if you mess with the formula problems will start. The challenge of being in different places and time zones will make fixing problems more difficult and left untended, a once well-oiled machine will start to combust and drive itself off a cliff.

Given the environment described above, the challenge is to create a distributed development model which everyone understands, accepts and follows.

The key criteria for this model are:

1. Takes into account overhead required for distributed development 
2. Ensures that the right level of consistency exists in the software development lifecycle to keep it on track
3. Is flexible enough to deal with the realities and flux of working in a digital agency which is sometimes like a magic roundabout. The model must be able to adapt to changing people, skills, projects and technologies and attitudes.

An effective way to approach this challenge is to define a set of key principles that must be adhered to in each area. These are categorised into Team, Communication, Scrum Delivery, Processes and Tools and Incentives and Motivation.

Team

A good team is the ultimate recipe for success. Below are a few core principles behind effective distributed teams.

Ensure you have a good technical lead. The technical team needs a good technical lead that takes responsibility for overall technical quality and can support the project manager on planning, client liaison and fire-fighting activities. This role is generally a mix of development and management oversight.

Allow time for lead oversight. Ensure that you have enough time for technical lead oversight in each project. The reality is that there is always noise, unexpected issues, on-going questions and technical developer support required. Additionally, the time to ensure quality in areas such as technical briefs, adherence code quality and efficient development workflow control can be significant on technically complex projects. You can have a lead doing zero development and oversight only.

Keep rhythm through consistency.  Ensuring you have a consistent delivery team is probably the best way of keeping a good rhythm of development. Changing team members disrupts the rhythm of development by having to invest time and effort to establish new relationships, learning the domain and the technology stack of the project. It can take a new developer at least a month to become an effective team member.

Manage and accommodate change. Ensure that there is a solid on-boarding process of new team members which takes into account ramp up time for the new team member and support time required for the technical lead. Expect the velocity of the team to decrease by somewhere between 20 – 50 per cent when there is a personnel change.

A hybrid leadership model can work. If your project and account management is on shore, having a lead working in the same location will make a big difference in reducing noise, supporting project management and help the technical team understand the business context of delivery. If your technical lead is permanently onshore, creating an additional development lead offshore ensures local leadership for the offshore development team.

Communication

Communication is probably the biggest challenge in distributed development. To ensure effective communication, the following is essential to ensure the right amount of communication happens in a project.

Reduce Noise.  Noisy communication is probably the biggest problem in technical delivery. The development team needs to be shielded from outside noise so they can focus on what they’ve been tasked to do. Reducing noise can be achieved by having clear and structured communication forums with defined objectives (sprint preparation, sprint planning, stand ups) and having a protective wall which deals with noise and adhoc communication. This wall can be comprised of the project manager and technical lead, both shielding developers and testers from interruption.

Face to Face is a must. Ensuring the team and key client stakeholders (e.g. the product owner) meet face to face is essential to building relationships. The investment in travel and accommodation costs is significantly smaller to not creating a motivated, empowered team.

Use technology to communicate.  Video and voice conferencing and IM are essential tools for communication in a development team. Investing in communication facilities is essential for high bandwidth communication between team members. Skype, Google Hangouts, Web Ex are all excellent options to facilitate remote communication.

Scrum Delivery

The main unit of work in Scrum delivery is the Sprint. To ensure Sprints are effective, the following are essential activities to do before, during and after each and every Sprint.

Prepare the Backlog.  The product owner (generally client side) must ensure that a backlog of tickets or user stories is prepared prior to the next sprint. Tickets in the backlog must have enough information to enable the technical team to discuss implementation, identify any questions or risks and provide initial estimates.

Internally review and estimate. The technical team should have an internal review of the tickets in the backlog that are expected to go into the next sprint. Discussing and estimating within the team is the best way to discuss implementation, estimate accurately and identify key questions or issues early on. It also helps prepare the whole team for the next sprint.

Prioritise tickets during Sprint Planning. With a reviewed and estimated backlog, sprint planning with Client stakeholders (most importantly, the Product Owner) should be a painless exercise where the client decides what goes into the sprint, is made aware of any risks and key questions are answered.

Sprint Execution Escalation.  Once a sprint is underway, the developers and testers should be shielded from noise. If there are factors which are affecting the Sprint such as changing scope, unexpected issues and noise, these need to be escalated to senior team members who can help. If issues are being escalated but not being acknowledged or dealt with, they are being escalated to the wrong person.

Sprint Retrospective. A retrospective at the end of each sprint is essential to allow for continuous improvement of the process, ensuring team voices are heard, ensuring the team are acknowledged for good work.

Technical Processes and Tools

Technical development has evolved with many strategies around automating testing and quality control. There are several must haves when working on distributed development, especially if development is in multiple locations.

Continuous Integration (CI). Ensuring you have CI setup to allow for both Build Validation Tests (BVT) and automated deployment is essential to keeping the house in check and not impeding development because of unexpected failures.

Automated Testing. Investing time in testing (unit, integration) is equally important in catching errors early on in the Software Lifecycle. The time taken to write and maintain tests must be included as part of development estimates.

Incentives and Motivation

Unhappiness in the work place is a cancer for any company, project or individual. The primary cause of unhappiness is trust and as soon as trust breaks down anywhere in the team, the outputs will be dramatically affected. A disenfranchised team with a trust break down can work at a reduced velocity of, I believe, between 30 – 60 per cent.

Combining good leadership and the right balance of incentives, motivations and rewards is the key to keeping the right spirit in place. The ultimate aim, no matter how you dress it up, is to deliver faster and better but it is a happy by-product that people’s lives aren’t being made a misery.

These principles are key to keeping up team morale.

Full team involvement in Sprint preparation and retrospective activities. Having the team involved in Sprint preparation starts the sprint with the full engagement and buy-in of the team.  Having the team involved in the sprint retrospective allows for the whole team’s voice to be heard, issues to be acknowledged and complements for good work to be given.

Don’t ignore escalations. If escalations are being ignored the team will quickly feel that they are not important and third class citizens on the project. As per the previous principle, ensure escalations go to the right person who can do something about it.

Build relationships while you build product. Working together can be a very rewarding experience and keeping an eye on how relationships are blossoming or not is important to keep the team dynamic from moving in the right direction. Sometimes, soft skills and relationship counselling is necessary to keep people working well as a team. If you project manager or team lead has these skills, they may find themselves playing therapist.

As a final thought, Martin Fowler wrote an excellent white paper on distributed, Agile development which touches on similar subjects and themes. His final note was that he wasn't sure whether outsourced development would become the norm,  like how the steel industry moved away from first world, western countries, whether it was a passing fad or there would always a fraught mix of both onshore and offshore development. Whatever the case, having strong principles in place which keep teams happy and producing good work is best defence against the greedy, gold diggers of the New West.