Author: Scott

I came across this article about the failure of the Virtual Case File project about a week ago. I read things like this in the hope of learning from the mistakes of others (instead of having to make them myself). What follows are some of the conclusions I drew from reading the article, and how they might apply to other projects.

Have the Right People in the Right Roles

The author of the article (Harry Goldstein) calls the appointment of Larry Depew to manage the VCF project “an auspicious start”. Since Depew had no IT project management experience, putting him in charge of a project so large with such high stakes struck me as a mistake. This error was compounded by having him play the role of customer advocate as well. In order to play of the role of project manager effectively, you can’t be on a particular side. Building consensus that serves the needs of all stakeholders as well as possible simply couldn’t happen with one person playing both roles.

Balance Ambition and Resources

The FBI wanted the VCF to be a one-stop shop for all things investigative. But they lacked both the necessary infrastructure and the people to make this a realistic goal. A better approach would have prioritized the most challenging of the individual existing systems to replace (or the one with the greatest potential to boost productivity of FBI agents), and focused the efforts there. The terrorist attacks of 9/11/2001 exposed how far behind the FBI was from a technology perspective, added a ton of political pressure to hit a home run with the new system, and probably created unrealistically high expectations as well.

Enterprise Architecture is Vital

This part of Goldstein’s article provided an excellent definition of enterprise architecture, which I’ve included in full below:

This blueprint describes at a high level an organization’s mission and operations, how it organizes and uses technology to accomplish its tasks, and how the IT system is structured and designed to achieve those objectives. Besides describing how an organization operates currently, the enterprise architecture also states how it wants to operate in the future, and includes a road map–a transition plan–for getting there.

Unfortunately, the FBI didn’t have an enterprise architecture. This meant there was nothing guiding the decisions on what hardware and software to buy.

Delivering Earlier Means Dropping Features

When you combine ambition beyond available resources with shorter deadlines, disaster is virtual certainty. When SAIC agreed to deliver six months earlier than initially agreed, that should have been contingent on dropping certain features. Instead, they tried to deliver everything by having eight teams work in parallel. This meant integration of the individual components would have to be nearly flawless–a dubious proposition at best.

Projects Fail in the Requirements Phase

When a project fails, execution is usually blamed. The truth is that failed projects fail much earlier than that–in requirements. Requirements failures can take many forms, including:

• No written requirements
• Constantly changing requirements
• Requirements that specify “how” instead of “what”

The last two items describes the VCF’s requirements failure. The 800+ page document described web pages, form button captions, and logos instead of what the system needed to do.

In addition, it appears that there wasn’t a requirements traceability matrix as part of the planning documents.  The VCF as delivered in December 2003 (and rejected by the FBI), did things that there weren’t requirements for.  Building what wasn’t specified certainly wasted money and man-hours that could have been better spent.  I also inferred from the article that comprehensive test scenarios weren’t created until after the completed system had been delivered.  That could have (and should have) happened earlier than it did.

Buy or Borrow Before You Build

Particularly in the face of deadline pressure, it is vital that development buy existing components (or use open source) and integrate them wherever practical instead of building everything from scratch.  While we may believe that the problem we’re trying to solve is so unique that no software exists to address it, the truth is that viable software solutions exist to subsets of many of the problems we face.  SAIC building an “an e-mail-like system” when the FBI was already using GroupWise for e-mail was a failure in two respects.  From an opportunity cost perspective, the time this team spent re-inventing the wheel couldn’t be spent working on other functionality that actually needed to be custom built.  They missed an opportunity to leverage existing functionality.

Prototype for Usability Before You Build

Teams that build successful web applications come up with usability prototypes before code gets written.  At previous employers (marchFIRST and Lockheed-Martin in particular), after “comps” of key pages in the site were done, usability testing would take place to make sure that using the system would be intuitive for the user.  Particularly in e-commerce, if a user can’t understand your site, they’ll go somewhere else to buy what they want.  I attribute much of Amazon’s success to just how easy they make it to buy things.

In the case of the VCF, the system was 25% complete before the FBI decided they wanted “bread crumbs”.  A usability prototype would have caught this.  What really surprises me is that this functionality was left out of the design in the first place.  I can’t think of any website, whether it’s one I’ve built or one I’ve used, that didn’t have bread crumbs.  That seemed like a gigantic oversight to me.

One thing I didn’t realize about CruiseControl.NET until recently was that it automatically labels your builds.  It uses a ... naming scheme.  The way this helps with deployments is that you can always look figure out what you’ve deployed to production in the past–as long as you only deploy labeled builds.

We still need to get our continuous integration setup working again, but in the interim, manual labeling of releases is still helpful.

This week, an application I work on started logging an exception that provided no help at all in debugging the problem. My usual practice of running the app in debug mode with production values in the config file failed to reproduce the error too. After a couple of days of checking a bunch of different areas of code (and still not solving the problem), Bob (a consultant from Intervention Technologies) gave me some code to get at all the InnerException values for a given Exception. Download the function from here.

StackTrace can be pretty large. Since we log to a database, I was worried about overrunning the column width. I also wasn’t keen on the idea of looking at so large a text block if an Exception was nesting four or five additional ones. So instead of implementing the code above, I changed the code to log at each level. Doing it this way adds a log entry for each InnerException. Because the log viewer I implemented displays the entries in reverse-chronological order, the root cause of a really gnarly exception displays at the top. The changes I made to global.asax looked like this.

The result of this work revealed that the app had been complaining about not being able to reach the SMTP server to send e-mail (which it needs to send users their passwords when they register or recover lost passwords).

Once we’d established that the change was working properly, it was time to refactor the code to make the functionality more broadly available. To accomplish this, I updated our Log4Net wrapper like this.

When I first started developing solutions on version 2.0 of the .NET Framework, I saw examples that had some logic in the App_Code folder. For things like base pages, I thought App_Code was perfect–and that’s how I use it today. When I started to see applications put their entire middle tiers in App_Code however, I thought that was a bad idea. Beyond not being able to unit test your components (and the consequences associated with a lack of unit testing, coverage, etc), it just seemed … wrong.

Fortunately, there are additional reasons to minimize the use of App_Code:

• It makes your builds slower.
• It makes your design experience slower.
• It makes the debugging process far more tedious.