Engineering Practices and the Success of Agile Projects
Agile Methodologies

Agile is not, as often thought, a silver bullet to ensure the success of an IT project. However, agile "done right” - structured, guided and monitored using appropriate software engineering techniques - will reap the full rewards of employing Lean Software Development, Extreme Programming (XP), Scrum, or other agile methodologies.

The additional effort to employ Agile within a comprehensive IT engineering framework will be recovered through increased efficiency of the production process, improved quality and stability of software code, minimized cost of system implementation and use, and decreased maintenance costs thereby delivering maximum return on investment.

Absent applicable software engineering techniques, inefficiencies inherent across each sprint will diminish the rewards of implementing agile; diminished results emanate from:

  • inadequate unit testing necessary to discover logic errors and requirement omissions
  • implementation of suboptimal and poorly designed code
  • tasks left uncompleted being pushed to a subsequent sprint
  • lack of adequate version control necessary to support rollbacks
  • reduction in the use of shared code

Without regard to the specific methodology employed, maximum benefit of using the Agile model cannot be achieved except through implementation of sound software engineering principles.

Many Agile Teams Dismiss the Significance of Software Engineering

Wrongly construed, two tenants of the Agile Manifesto can be interpreted as downplaying the need for applying engineering principles:

  • Individuals and Interactions over Processes and Tools
  • Working Software over Comprehensive Documentation

An emphasis on Individuals & Interactions and Working Software does not obviate the requirement to use appropriate engineering processes and tools to ensure that each Sprint is properly managed so as to produce the highest quality software at the lowest cost.

Combining the two tenants, though verbose, expresses the essence of each:

“A maximum emphasis on Individuals & Interactions and on Working Software using the most cost-effective Processes & Tools while generating the minimum Documentation necessary to ensure the highest quality result in the least amount of time and at the lowest cost.”

Software Engineering Methods and Techniques in the Agile Arena

As delineated in Systems Engineering and Design, scientific methods and techniques applicable to every IT project  include:

  • Test-Driven Development Utilizing Automated Acceptance Criteria Verification
  • Good Coding Practices Including Peer Review
  • Refactoring
  • Embedded Security
  • Adequate Version Control
  • Well Defined Project Work Scope

The emphasis on creating code in the least amount of time during each sprint makes use of peer review, refactoring, and adequate version control of paramount importance.

Essential to the success of all agile projects, whether developed under the constructs of Lean, XP, Scrum, or another agile framework, are principle-driven engineering processes and techniques to include: use of paired programming, static code analysis, coding in short cycles, frequent check-ins of smaller batches, and continuous automated integration.

Timely Code Reviews. Code reviews are well-documented as an effective tool for reducing bugs and improving the quality of software produced. Given the ideal frequency with which code is checked in, however, a traditional formal code review is impractical. Emanating from XP are several techniques which provide critical review within the fast-paced cycle of each Sprint including:

  • Shared Ownership. The concept of shared ownership considers the codebase owned by the team as a whole; thus, each team member can be expected to review, fix, refactor, enhance, or otherwise work on, code they may not have written.

    Other than specialists with expertise in areas such as networking, threading, databases, etc., developers are generalists lacking in-depth knowledge of any given area of a product. By its nature, this aspect of shared ownership is its biggest flaw limiting the effectiveness of code review; in reality a generalist will likely not be familiar with the requirements a given block of code is meant to address thus rendering his/her review less effective.

  • Swarming. In practice, swarming involves team coders working on one or a few stories concurrently rather than each working on an assigned story in parallel. As with shared ownership, stories are considered owned by the team not the individual. Developers are encouraged to actively work on the same story from initiation through check-in. Typically, once the author completes coding, other developers will begin follow-on processes to include review, refactoring, and, to the extent necessary, extending the code.

    Swarming ensures timely review and overcoming a frequent criticism of code review - that the reviewer is not sufficiently familiar with details of the story on which the code segment is based.

  • Pair Programming. Pair programming, as the name implies, requires two developers to collaborate on the same code segment. Like swarming, pair programming provides real-time review.

    However, by its nature, pair programming can be difficult to implement and is arguably inefficient. Unlike swarming which permits each developer to work on a discrete tasks in parallel, pair programming requires each developer to work in a tethered fashion completing tasks in a serial fashion.

Static Code Analysis (SCA). Code analysis tools permit developers to test code for vulnerabilities outside of a run-time environment; examples of discoverable issues include security flaws (buffer overflows, format string errors, etc.), deadlocks, and functions that code be called with null or uninitialized parameters. Note that SCA tools cannot determine that code is robust enough to address all acceptance criteria; validation of nature must be addressed using automated unit testing or manual Peer Review.

Frequent Check-ins of Smaller Batches. Frequent check-ins are good for traditional teams but are a critical a pillar of the agile development process. A small piece of new functionality is considered qualified for check-in to a shared development platform when coding has been finished, a clean local build completed, appropriate unit tests have been coded, and automated comprehensive unit testing has been completed to ensure no regressions have surfaced.

Frequent check-ins uphold the principle of “continuous integration” enabling project managers good visibility into daily progress of individuals as well as of the team as a whole. Consistently followed, the practice can reinforce a team’s sense of accomplishment and, importantly, of momentum.

Importantly, frequent check-ins reduce risk by permitting suboptimal error prone code to be identified sooner thus enabling a timely rollback so as to limit potential loss.

Worth noting, absent use of one of the code review techniques discussed above, a formal pre-check-in code review could work against the practice by creating a bottleneck should a qualified reviewer be available when needed. A reasonable solution would be to enforce a timely post-check-in code review.

Continuous Automated Integration (CAI). A CAI process, typically used on larger projects, automatically pulls checked-in code onto a continuous integration server along with the related unit tests; it executes a verification build, and runs a comprehensive suite of automated acceptance & unit tests. Integration tests include those to validate APIs, other interfaces, and data formats.

CI encourages reuse and sharing of code by making it accessible to the Team upon its integration. It ensures Integration Engineers are promptly made aware of integration errors or conflicts. Repetitive labor-intensive and error prone aspects of the testing process are eliminated. Its efficiency reduces use of resources, increases code quality, and provides real-time visibility into the status of the system development process vis-à-vis the project schedule.

Agile teams under pressure to produce working software within short timeframes might be tempted to forego the investment required to implement sound engineering practices. To do so will increase risk and diminish returns. Failure to discover logic errors and to ensure nuances of complex acceptance criteria are addressed will increase the largest avoidable project expense – the cost to find and eradicate software bugs. Tasks left uncompleted and pushed to a subsequent sprint and the failure to properly share code will waste additional resources. The inability to rollback code in a timely manner when required will incur unknown financial and political costs.

Management of an agile project using the tools of a comprehensive software engineering framework is critical to its successful completion and essential to generating the maximum return on investment possible.