The long contentious never ending debate!

Introduction
In my first article in this series, I discussed the Agile method (Manifesto for Agile Development) in general and then specifics about the daily meeting. In the third and final one, I look at the cases for and against Agile. In this second article, I compare Agile vs Waterfall and discuss pitfalls and misconceptions of each and where hybrid models could be used.

Growth of Computing and Adoption of Waterfall Method
In the information technology world, the advent of computing hardware necessarily brought with it the need for software, the set of instructions which tells machines what and how to compute, calculate, etc. As the ability of computing grew and became more compact relatively speaking, the application for its use in business environments grew and so did the scope, size, and complexity of the programs. The “Waterfall” method of developing software was adopted and adapted from the model used for the manufacturing and construction industries.

Advent of Agile and the Manifesto for Agile Development
Over time, given many issues arising from the challenge of delivering quality software which may or may not have to do with the methodology itself, programmers tried coming up with and working with new methods such as “Agile” with the Manifesto for Agile Software Development.

Agile was designed to address what many saw as the shortcomings of the Waterfall method. It also was necessary to come up with a way to reduce time to market as the internet and web technologies required much shorter and shorter times to implement new products and enhancements.

Agile seemed more adaptable for software development in terms of the shift to rapid innovation. While many like to compare the two methods to each other and talk about why one is better than the other, in truth they each have their place and use depending on specific environments and cultures. They each have advantages and disadvantages and the pitfalls and misconceptions of implementing them can often cause not only a disconnect between the development team and customers, but often a need for an integrated hybrid method.

What Method is used when?
The use or application of a particular method will depend on where you live and work, the size of the company and the type of software they are building for internal or external use. In areas where there tend to be numerous large corporations hiring IT development teams, the method may still largely be Waterfall. The caveat to that are large web based companies who grew up successfully and rapidly from startups who need to deliver releases not just on a weekly but even daily and hourly basis. In areas where there are smaller companies and many startups, it may seem that every company is or has adopted some form of Agile and “everyone is doing it”.

Challenges in adoption of Agile vs Waterfall
Some of the problems I have seen in the various companies and projects I have worked on are where the development team has adopted Agile, but the end users, the customers, are more comfortable with Waterfall and resist going along with the Agile way of working. Another is where a large company decides it wants to pursue adoption of Agile to try and address known issues in the current SDLC method, and brings in teams of consultants to train management and workers on Agile. Then as they go along attempting to adopt it, have instances where part of the process is Agile-like and others remain in Waterfall or some similar mode (kind of a hybrid model). The various steps in the process are implemented haphazardly, such as the daily meeting, and ends up not adding the value that was promised or hoped for.

Key Factor in Adoption of a Model
A key factor to take into consideration when deciding which method to use or whether to make a change is the corporate culture. (I don’t go into details about corporate culture itself here but plan to in other articles.) What I mean by this is that each method is a framework as a starting point (not the end result) for a group to use and tailor to their own business requirements. The culture is all the internal business and social aspects of the environment that plays a part in shaping how the framework is turned into a workable (or not) process and methodology for a given company. The end result being that an implementation of “Agile” may not for example be Agile according to what purists say is Agile but it works for the given group. On the other hand, not sticking to essentials somewhere along the way may have been why it’s not working and fails. I will discuss that later.

Why Waterfall?
There are questions regarding why a particular company is using Waterfall, and even why they continue to use it after perhaps evaluating other methods. Companies may continue to use it because that is what they have always done and that is how they will continue to do things; it’s not in their nature to change and to try to change would require more effort and culture change than they could manage.

There may be other reasons as well but according to this article its demise is premature and in fact it is still used they say on a third of all development projects:

And as stated above, a particular implementation of Waterfall will vary from place to place depending on requirements along with adaptation and integration with other methods forming a hybrid process.

Then there are companies, generally large traditional one like banks, where the nature of their business requires its continued use. For example I spoke with a large regional bank about a project to convert their entire overall banking system(s) into a new more modern one. The size and complexity of the project, the large number of stakeholders, the huge number of business rules many of which were tied into laws and regulations, and the very high level of quality required necessitated a Waterfall approach especially in the early stages with analyzing the requirements and ensuring the workflows and data requirements were thoroughly evaluated and documented before development started. They said that later, once development started, in limited groups for specific areas, development may take the form of an Agile like iterative development process but that remained to be seen.

Why Agile?
Once, some years ago, I asked a developer colleague what Agile was and he said it’s not a particular process but a toolbox from which you can pick and choose. To make another long story short, Agile evolved out of innovative practices from a number of different new methods such as Extreme Programming. Once the concept (values and principles) was codified in the Manfesto, people tried a number of different ways to implement it in processes. For example, Scrum was another set of Manufacturing processes that was adopted and adapted to use in a software development framework in an Agile model.

Companies who might want to switch to Agile are those who:

1. have identified numerous problems with delivering software with their current methods,
2. need to respond to changes and implement more quickly
3. and are able to enact cultural change.

Agile seems to work best either in small companies or small development groups where requirements can be more quickly and loosely documented or codified in use cases/user stories, small chunks of work can be implemented quickly to produce useable software which can then be evaluated and tested quickly by customers.

Pitfalls and Misconceptions
It seems to me that on some of the projects I have worked on, the development manager just decided that they were going to use a particular method such as Agile without evaluating whether or not it would be a good fit for the environment and customer base they were going to be working with. On a number of projects I have worked on where that happened, the developers discussed the Agile method with the customers, said this is what they wanted to use, discussed how the customers would interface into that method, and asked for their agreement. In both cases the customers may have had questions and expressed some reservations but then agreed to it.

What am I getting at?
Let’s review.
So for example you have the tenet from the manifesto that

Customer collaboration over contract negotiation

is valued and also

Business people and developers must work together daily throughout the project

Business people can refer to internal customer facing business analysts or customers themselves.

On both projects the customers  or business people and developers were not co-located but remote. On the first project what would end up happening is that the customer group only made one person available about once a week (instead of the more people and more frequent meetings that were initially agreed to). This meant that it would take longer for the development group to get feedback in order to adjust software and deliver. They would continue to develop but it meant that adjustments would take place further down in the cycle turning what would have been in iterative cycle into more of a sequential (Waterfall-like) cycle. In other words, even though the customers had given lip service to agreeing to trying an Agile development method, they were still, in practice, doing Waterfall. The development group is doing Agile, the customers are doing Waterfall. This would have been a good place to have developed a more efficient and productive hybrid method, but that did not happen.

Another pitfall here was the idea that with Agile you don’t need documentation. (This is not actually part of the manifesto but a misconception. The principal states: “Working software over comprehensive documentation (That is, while there is value in the items on the right, we value the items on the left more.)” ) Therefore, none of the business requirements were documented. The lead architect gave whatever instruction was needed to other developers at the time of development. This lead to two problems. One was that some requirements were left out and others could not be verified by review from the customers. This meant additional coding time down the road after a feature had already been developed. In addition, the test group had no concrete way to know what had been developed in order to be tested and getting the requirements meant taking more time from developers who needed to instead have spent that time programming. If a business analyst or analysts had been allocated to the project as had been requested, they would have known what requirements had been in place since the business had been running and were not going to change that could be reliably documented and which ones would be fluid and could be dealt with as user stories. Therefore while “comprehensive” documentation was not required, not having any documentation where it was needed in the proper form was detrimental to the project.

On another project which was a similar setup to the prior one just discussed, the development team said it would implement a release every three weeks and the customers agreed to test the new features and report issues. Over time the number of users testing each iteration went down and eventually they decided they would look at and test the new features at some point. Then the customers decided that all the specs they had written needed to be rewritten and in addition they wanted to reorganize which parts of the application would be developed in what order, and in an order that they wanted to review and test it. They would self organize their own teams (and testing process) to do beta testing when all the development was complete. Again, clearly this was a case of the development team doing Agile and the customers doing Waterfall.

While there may be a disconnect between these two, failure to find alignment and in what customers may believe to be unmet expectations, can ultimately doom the project!

Other examples of pitfalls would be where parts of an Agile framework like Scrum are implemented but they are not effective or as effective as they could be. In a prior article here (link) I wrote about the daily stand up meeting and how over inviting people can lead to meeting bloat and extended meeting times. Another example would be in a Scrum implementation and using the sprint retrospective as a blame session instead of seeing what worked, what could be improved and how.

The Hybrid Method

A hybrid method would be useful in an environment where the strict application of one particular method is not possible or practical. It might be more effective to put in place parts from multiple methods and adapt them as needed so as to work effectively in that environment and improve productivity.

One example of a hybrid model was discussed in my example of the bank where they talked about the reasons and need for using Waterfall but finding usefulness in Agile at a certain stage.

Another example has to do with what types of documentation to produce and when. When documentation is necessary and where it is not.

Also a hybrid process may be useful where most of the development process has been Waterfall, but once you enter the final phase, use Scrum to speed up bug fixing for a period of four weeks or less.

Which method has been more effective than another for you?

Where might you need and how would you implement hybrid models?

Comment down below with your thoughts!
Like and Share this article and Subscribe for future updates!