Agile and Scrum buts causing Technical Debt

I am sure you must have heard some of these Agile and scrum buts which contribute to Technical debt?

• Re-factoring: I am aware that particular section of code needs re-factoring, as it was just built as a prototype. But it will have to wait, as the feature development takes priority over re-factoring.
• 3^rd Party dependency: I know we should integrate with these newly released 3rd party libraries as it contains the fixes we need. But based on prior experiences it could take up to a week. It would be better if we do it in the next iteration.
• Tests: I wanted to write end to end automated test case for this feature, but if I develop that, I can’t test the other features under development. I will plan for automated tests in next iteration.
• Defects: We are done with our stories, but we have 10 open defects that we plan to target first thing in next iteration.

The above “buts” will reduce the team’s velocity and add to technical debt. It will hinder teams ability to self organize and move them from being truly agile. If above reasons are ignored, for the first few iterations the teams might show good results and get away by breaking scrum principles. However in the long run this is a lose-lose proposal. Development of new features will take longer as development has cut corners for re-factoring effort. Development team will also spend considerable amount of time fixing defects that were missed due to lack of unit tests.  Test team will spend more time running regression tests and verifying defects rather than doing exploratory testing and testing new features. The further the team pushes integration with 3^rd party library, the more risk it adds to the overall release due to last minute surprises.

Conquering technical debt in a timely manner is extremely important for scrum teams to be truly agile. There are some solutions that I can think of – Ignore it and pay the penalty everyday, take a break from developing new features and clean up the mess by adding additional people resources, re-design and start again…

Implementing agile takes a lot of discipline and buy-in from the management team, as it’s a big shift in how business in conducted. A disciplined set of agile and scrum best practices should be followed, such that it empowers teams to take right steps to deliver “world class” quality software.

Enterprise scrums - A Tiered Approach (Implementation)

I want to continue on tiered discussion in this blog and specifically talk about the  implementation aspect of the tiered approach. In the earlier blog, I discussed the guidance process and how we are ensuring that all scrum teams are aligned towards a common goal. Let me take this a step further and see how we plan to integrate and demo the value-driven functionality at monthly boundaries that showcases each team's contribution towards Tier1 vision.


The guidance process gives direction and generates prioritized iteration backlog with sufficient details for scrum teams. Next step is to glue deliveries from these individual scrum teams together to have a viable end-to-end demo at monthly boundaries. To achieve this, we have setup integration-level scrum development team. This scrum team by itself does not develop any functional software, but it develops a framework to build and test the deliveries generated by feature and component teams for Tier1 goal. This scrum team also owns release management activities for the product being developed.


To achieve integration at the monthly boundary, we broke down the monthly sprint schedule for all scrums into 3 weeks of development with the last week of each month reserved for hardening phase. With this approach all scrum teams plan for 3 weeks worth of work and are done with their feature deliveries by end of 3rd week. In the hardening week, each scrum team verifies that they have met the "Done" criteria for their planned user stories - "Done" means all automated end-to-end integration tests passed, acceptance tests passed, functional specs and design updated on wiki, etc. 


During the hardening phase each scrum team also makes their deliverables available to the integration-level scrum for overall integration. If there are any hiccups during the integration testing, scrum teams makes themselves available to fix those issues.  An additional point to note is, integration is not just happening at Integration-level scrum, it's also happening between component teams and feature teams. As and when code is ready for integration, feature teams consume deliveries from component teams. Integration-level scrum is where all scrums deliver and everything comes together for end-to-end value-driven feature that customers value. 


To further align and ensure all teams are working in unison and bring transparency, we have daily Scrum of Scrum at Integration-level scrum with participants from all scrum teams. At this forum, we help resolve any team impediments as well as dependency issues.


This is the strategy we are using to align scrums and develop value-driven feature for our customers in enterprise environment with global teams. Are you using a different approach that worked for you? if yes, I would be really glad to hear your experiences.

Enterprise scrums - A Tiered Approach

One might think large corporations like EMC would have nailed out Agile and Scrum process right to the core. Unfortunately, we have the same challenges that any enterprise organization would have, like global teams, a mix of component and feature teams, suite of products, demanding priorities between different products, etc...

Earlier, in Agile development with Scrum, we tried the bottom-up approach where each scrum team provided input and what they planned to do in their upcoming iteration. Since scrum teams were aligned at monthly boundaries, teams demonstrated their achievement at the end of each month. Very soon we realized that each team is working on stories they deemed important, but their deliverable did not converge on value added functionality we set out to achieve out for our customers. We had setup a guidance team (Architects, Product Managers and few leads) to keep tab on deliverables from each team and align their deliverable to a value add feature. But it was difficult to achieve the alignment across globally dispersed 12+ scrum teams.

Since then, we moved to top-down model similar to Dean Leffingwell's whitepaper "The big picture of enterprise agility" with some tweaks. We still have the guidance team in place, but to improve transparency and align deliverables from each team, we extended guidance invite to team leads/scrum masters for each scrums along with Architects, Product Managers and Engineering leads/Product Owners. We also requested product managers to define prioritized list of themes they would like to see in the first release of the product. In addition, we did one month of advance pre-planning i.e for September iteration, the guidance team met in the month of August.

Next, we modeled a 3-tiered approach. Tier1 starts with high level guidance - In the first week of each month, the guidance team looks at prioritized themes from product management's backlog. In a group setting, the guidance team would add more detail and color to these themes to set a common vision and goal to which all teams should contribute in the upcoming iteration.

In Tier2 guidance which happens in the 2nd and 3rd week of each month, the guidance team gets together and decomposes Tier1 stories further. We start with a brain dump of listing out high level stories along with teams that should work on it, to accomplish the vision set in Tier1. Then with product manager's and product owners help, we prioritise the list of high level stories that were created. Following this, there is negotiation with product managers on which stories are absolutely necessary for upcoming iteration and what can deferred to future iteration. The idea behind Tier2 is to ensure each team has enough details to decompose stories further into iteration level stories.

In the Tier3 guidance, that happens in the last week of each month, the guidance team reviews the overall backlog and ensures all teams are aligned correctly to achieve the goal guidance team has envisioned.

At each tier, we have also setup a feedback loop through each team's scrum masters and engineering leads. Correct expectation is set with these leads to keep the guidance team informed on what can be realistically achieved and which items need further negotiation.

So that's the approach we have setup and marching forward with. Have you tried a different approach that worked for you in your environment? If yes, please share your thoughts.

Would longer build time hamper Agile Development?

This is one of the biggest issue I have seen in enterprise environment for teams trying to be Agile.

Consider multiple scrum teams delivering features and bug fixes to a common stream. If nightly builds takes 8 to 10 hours for completion then it’s difficult for this group of engineers to be Agile. Build can take long times for various factors like

Hardware – Build is running on slow network and/or running on older hardware that is unable to keep up with the demands.

Software – Build has circular dependencies between modules; each module is being executed in serial order, etc…

Global teams - If multiple global scrum teams (Ex. India and US) are delivering to common stream, and dependencies are not managed correctly, the nightly build might fail. A “simple fix” in such situations can turn into 24-hour delay due to time zone changes.

In addition, isolation of build knowledge at single geographical site adds to the delay in generating timely build.

Here are some possible solutions…

In my opinion, enough investment should be made on hardware and the tools used for build environment. Future growth and needs of the product should be kept in mind while designing the build system to avoid future disruptions to the build system.

Sufficient investment should be made in people who understand build system so that issues like circular dependencies in the builds can be broken.

If need be, take hit on active development to split the serial execution of modules so that they can be run in parallel on separate hardware to speed up the overall build process.

Train and educate build personnel across each site to avoid single point of failures.

I have led some successful agile projects and have seen over and over again that by investing up front and taking control of build environment helps reduce overall build delay. By setting up build environment using continuous integration server like Hudson, and build tool like Maven, the development team gets continuous feedback on quality of their deliveries and helps scrum team self organize.

The question you should ask yourself is – should I spend time and money NOW to correct this problem, or am I willing to take a hit, and pay the price of not fixing it forever.

Have you ever faced such situations? How did you remedy it?

There is some good discussion on this thread at "ScrumDevelopment" forum.

http://groups.yahoo.com/group/scrumdevelopment/message/40744

Have you ever tried geographically distributed cross functional scrum team?

For one of the projects that was being built ground up, we formed a geographically dispersed cross-functional team. This team had a very good balance of developers, testers and a dedicated PO as well as Scrum Master, a role that I played. This team was distributed across US, Ireland, Cork and Bangalore, India. As a team, we lined out some really good development and Scrum Best practices and we started with a mission of forming a model scrum team that would start training other teams as they came on board.

The momentum of the team was great with excellent participation -- A point to note here is - this was a global team - so some folks in this team worked off hours (late at night). As time progressed reality started sinking in -- I just want to highlight few issues that became very obvious

1. Planning meetings, Retrospectives, Demos, Daily Scrum meetings were becoming extremely painful due to time zone differences. Morning meetings in US meant late night meetings for folks in Bangalore and vice versa. Put Cork from Ireland in this mix and you are left with couple of hours slot in the mornings. As time progressed, getting full participation from all team members became challenging and as it was affecting people's personal family time. This is where I believe site afffinity that Ken Schwaber swears on definitely helps to build team momentum.
2. The idea behind co-located, self-organized team is to ensure that all issues are resolved by standing in your cube and giving a holler to your team mate. In global scrum teams, this is a major impediment, as defect fixes, build failures, code reviews, etc might have to wait until the next day. Again this depends on how the expertise in your team is distributed. As a Scrum Master, I have lived this experience where teams had spent cycles waiting for updates/feedback from remote site that could easily have been avoided.

As a Scrum Master, I saw the above mentioned issues first hand and after a few iteration we just realized that forming Co-loated teams at each site would make more sense. In any case, I still take pride in forming this team, as the development and scrum best practices that came out of this team are used by the entire organization.

What are your thoughts?

Can you form a Scrum team with just one person?

Recently, one manager approached me for advice on scrum practices which I was happy to assist with. This manager had a request to build some new features and integrate the same with the existing product. She was also looking to develop these new features using Agile software development with Scrum. All was good with me so far. What I found a bit unusual was this manager only had one person @ 60%capacity to work on the scrum. The manager herself would play role of a scrum master as well as a product owner.

My first thought was does she even have a Scrum team, a team of just one? However, talking further with this person and understanding what she was looking for, I thought one person scrum might make perfect sense too. She was looking for predictability on how long it will take to get this project done with one person working at 60%. Secondly, she also wanted to make a case to management that this project might not be one person effort and would need multiple folks to pull this release off. So, scrum made perfect sense for both her concerns. For now, I have asked her to do capacity planning, list sprint goal, conduct planning meeting, and finally demo at end of the sprint to demonstrate what was achieved to management. Only time will tell how successful or not so successful she was. However, I am inclined towards saying one person Scrum team is a real scrum team and a lot can be accomplished by this one person army.

I have posted this blog in scrumdevelopment forum and it has generated some really good discussion.

How to roll out a process for large global teams?

My role as a program manager demands communication in various forms within organizations. I have seen and used various approaches, some effective and some not when a new process is rolled out. I would like to share my experiences and at the same time would like to learn from your experiences.

My definition of a process, is a systematic and disciplined way of doing things that creates a positive impact on how the overall business is conducted. Some examples are scrum best practices, transitioning to a new tool, generating org wide common metrics, etc. Now consider rolling this out to a large (150+) engineering team (development,QE, project managers, etc) spread around the globe (America, Europe, Asia,...). The impact of a new solution depends on what mode of communication is used.

Email communication: I've noticed email communication works out better for general announcements like org changes, something to do with HR activities. However it has not always been as effective for process communication. Again, it depends on how big the change is and what type of impact you are trying to make. Email guarantees that a new process lands in everybody's inbox, but does not guarantee that everyone will adhere to this new process. Why? Because as my new manager says, "every individual is wired differently, has different ways of doing things and has different attention span" -( I think I am going to take her offer to read a book on Human Dynamics). My experience has also shown that for email communication to work effectively, multiple reminders at constant intervals are needed for a process to be part of an organization's DNA.

Meetings: From my experience, this approach seems to be most effective. This can be implemented in various ways.

• One approach is to roll out a process in phases. Select couple of high performing teams and educate them on a new process. Monitor and measure their progress for a short duration (2 to 3 weeks). Once you feel comfortable, use their success stories and start rolling out the new process to all teams by conducting further meetings. This also gives you an advantage of making tweaks to the process based on working experience of these teams. The cons of this approach is, it takes a while until the entire organization comes up to speed.

• Another approach is to form a team of key players like managers, team leads, scrum masters for Agile development across the organization that can be trained and educated on a new process. The next step is to hold them accountable as well as responsible for rolling out this new process to their teams. The trick to making this approach successful is to ensure that these subset of players talk and have same understanding of the process. If not, its difficult to get consistency, as each team will be trained differently based on the version of the new process understood by these trainers.

All hands: This forum can be a very effective and can make a huge impact if senior management briefly discusses the new process and enforces the importance of implementing it successfully for the organization. Honestly, I have my reservations on implementing any new process this way. Why? One. just because getting every participant in a room or on a phone in global teams is almost impossible due to timezone differences. Secondly, sharing minute details of a process and opening the forum for general Q&A is not viable.

Individual Goals and MBOs: Making everyone in the organization take a shared goal of some healthy percentage i.e 10 to 20 % almost guarantees that the new process will be effective. By tying dollars to a new process, everyone has equal stake in making the process work. However this can be an extreme step, as it can make lots of folks unhappy if the shared goal has to be marked incomplete.

In summary, there are multiple ways to roll out a new process in a large global team. The best way to ensure that everyone within the organization is in unison, is to use a combination of the above methods.