Saturday, August 19, 2006

Anatomy of a fuckup.



And it all started with this cocksucker.

This is what happens when stupid people believe in magic (emphasis added).

The FBI computer upgrade that wasn’t
$170 million bought an unusable system

By Dan Eggen and Griff Witte
WASHINGTON - As far as Zalmai Azmi was concerned, the FBI's technological revolution was only weeks away.

It was late 2003, and a contractor, Science Applications International Corp. (SAIC), had spent months writing 730,000 lines of computer code for the Virtual Case File (VCF), a networked system for tracking criminal cases that was designed to replace the bureau's antiquated paper files and, finally, shove J. Edgar Hoover's FBI into the 21st century.

It appeared to work beautifully. Until Azmi, now the FBI's technology chief, asked about the error rate.

Software problem reports, or SPRs, numbered in the hundreds, Azmi recalled in an interview. The problems were multiplying as engineers continued to run tests. Scores of basic functions had yet to be analyzed.

"A month before delivery, you don't have SPRs," Azmi said. "You're making things pretty. . . . You're changing colors."

Within a few days, Azmi said, he warned FBI Director Robert S. Mueller III that the $170 million system was in serious trouble. A year later, it was dead. The nation's premier law enforcement and counterterrorism agency, burdened with one of the government's most archaic computer systems, would have to start from scratch.

Failures of almost every kind

The collapse of the attempt to remake the FBI's filing system stemmed from failures of almost every kind, including poor conception and muddled execution of the steps needed to make the system work, according to outside reviews and interviews with people involved in the project.

But the problems were not the FBI's alone. Because of an open-ended contract with few safeguards, SAIC reaped more than $100 million as the project became bigger and more complicated, even though its software never worked properly. The company continued to meet the bureau's requests, accepting payments despite clear signs that the FBI's approach to the project was badly flawed, according to people who were involved in the project or later reviewed it for the government.
...

"SAIC was at fault because of the usual contractor reluctance to tell the customer, 'You're screwed up. You don't know what you're doing. This project is going to fail because you're not managing your side of the equation,' " said Kay, who later became the chief U.S. weapons inspector in Iraq. "There was no one to tell the government that they were asking the impossible. And they weren't going to get the impossible."

'Not on a path to success'

More than two years after Sept. 11, when a team of researchers from the National Research Council showed up to review the status of Trilogy, FBI officials assured them that the bureau had made great strides. That was true in part: By early 2004, two of the three main pillars of the program -- thousands of new PCs and an integrated hardware network -- were well on the way to being delivered and installed.
But, as the researchers soon learned, the heart of the makeover, VCF, remained badly off track. In its final report, in May 2004, the NRC team warned that the program was "currently not on a path to success."

The review team from the NRC, which is affiliated with the National Academy of Sciences, was made up of more than a dozen scientists and engineers from top universities and leading technology companies, all of them independent of the FBI and its contractors.

The report observed that the rollout of the new case-management software had been poorly planned nearly from the beginning. Months after the program was supposed to be complete, it remained riddled with shortcomings:
· Agents would not be able to take copies of their cases into the field for reference.
· The program lacked common features, such as bookmarking or histories, that would help agents navigate through millions of files.
· The system could not properly sort data.
· Most important, the FBI planned to launch the new software all at once, with minimal testing beforehand. Doing so, the NRC team concluded, could cause "mission-disruptive failures" if the software did not work, because the FBI had no backup plan.


"That was a little bit horrifying," said Matt Blaze, a professor of computer science at the University of Pennsylvania and a member of the review team. "A bunch of us were planning on committing a crime spree the day they switched over. If the new system didn't work, it would have just put the FBI out of business."
...

Matthew Patton, a programmer who worked on the contract for SAIC, said the company seemed to make no attempts to control costs. It kept 200 programmers on staff doing "make work," he said, when a couple of dozen would have been enough. The company's attitude was that "it's other people's money, so they'll burn it every which way they want to," he said.

Patton, a specialist in IT security, became nervous at one point that the project did not have sufficient safeguards. But he said his bosses had little interest. "Would the product actually work? Would it help agents do their jobs? I don't think anyone on the SAIC side cared about that," said Patton, who was removed from the project after three months when he posted his concerns online.
...

One FBI manager estimated that the scope of the Trilogy project as a whole expanded by 80 percent since it began, according to a February 2005 report by Justice Department Inspector General Glenn A. Fine.

SAIC has consistently said that it was trying to meet the FBI's needs but that its efforts were undermined by the bureau's chronic indecision. Executive Vice President Arnold Punaro submitted testimony to Congress in February 2005 that cited 19 government personnel changes in three years that kept the program's direction in flux.
...

"We had information that could have stopped 9/11," said Sen. Patrick J. Leahy (Vt.), the ranking Democrat on the Senate Judiciary Committee. "It was sitting there and was not acted upon. . . . I haven't seen them correct the problems. . . . We might be in the 22nd century before we get the 21st-century technology."


If you have ever toiled in the vineyards of IT for more than a minute, you have either worked on or been within the blast radius of a project like this.

Designed by committee, whose every whim and bowel-movement is praised as wise and sagacious beyond the consultant’s poor power to describe with mere words.

Absolutely yes, sirrah, using cutting-edge this and using new database theory that breaks the Normal Form barrier and cutting edge yadda yadds or our proprietary Natural Language interface using the latest in third fourth fifth sixth generation code we can and will build you the elusive Magic Box with the Easy Button on top.

The Super Ooper Duper Database that will bind up every wound and dry every tear.

The website that will make Almighty God Herself weep.

That’s when those of us who have been running in circles screaming that you are about to drive the division/company/country off a fucking cliff are told to STFU. To “get onboard or get out.”

Are called traitor as we desperately try to warn the decision makers that they are shoving a loaded gun in down the division’s/company’s/country’s throat and the consequences are going to be really, really bad.

“No, no, no,” we are told. “Everything is different now. The laws of cause and effect. Of human nature. Of history. Of thermodynamics. These have all been repealed by this New Majyk Doctrine! The New Majyk Paradigm! Now whatever we wish for…happens!

"It’s a miracle.

"It’s Amazing.

"It’s Fucking Wonkavision!!

“Don’t you see how new it is? NewNewNewNew, and sooo shiny!”

No, asshat, it’s not New. It’s Old. God Alone knows how old, but this scam of convincing idiots that doing something radically stupid is really, actually brilliant was old when Adam and Eve were playing “Snakes...In a Garden!”

And as to the moderns, technological gloss on this ancient con, I am only being semi-facetious when I say that it all began with that stupid, fucking IBM PC commercial. Where Charlie Chaplin toddles merrily around as tech and a few, slim, user-friendly manuals alight gently on his kitchen table.

He taps a few buttons and, Voila: Instant productivity.

Since that dark day, the treaders of the digital grapes have been fighting a rear-guard action against Stupid Bosses and Douchebag Consultants who insist that, with a little moxie and a little imagination -- and your name on this wildly open-ended contract -- all of the unicorns and Lucy Lius and 30% profit margins of your dreams will come positively rocketing out of the business end of the cornucopia we will build for you.

That’s when the money starts rolling in. Huge piles of it



delivered on palettes using C-5 Galaxies if necessary, and from that point on the death-spiral begins.

And to see this pattern in all of its perverse, pernicious, pervasiveness just substitute:
PNAC for SAIC.

The White House for the FBI.

The Iraqi Project for the Virtual Case File Project.


Because the Stupid Men at the top cannot admit failure and, oh, say, fire Don Fucking Rumsfeld without loss of face. They can take no corrective action because to do so they would have to confront the fact that everything their critics and opponents said was true, and everything their allies and friends have told them has been a lie.

And the dread of copping to that level of incompetence and cupidity -- of having to own up to being every inch the dribbling, inept idiot that the rest of the world outside his bubble already knows he is -- is a thing no Stupid Man with Power will ever, ever do.

They would rather die first. Or, more accurately, they would let the world die first.

So instead they redouble the bet, pour more money down the rat-hole, and fire the last, few, honest men within the organization who still dare to speak the truth and who still might be able to salvage the situation if given half a chance.

I have seen, up close, whole companies go under because of this scam. And, in the end, the parasites survive and scuttle on to leech the life out of yet another fat, dumb CEO and leave behind bankrupt pensions and ruined lives.

But of course, there is no bigger Hunny Pot than the United States Government, which is why the Bill Kristols of the world are calling for ever wilder, more insanely reckless action: They're going All In, because they know that after they're burned off of the flesh of this nation like the tics that they are, there is no place left for them to go.

For a long time there will be no more rich, bloated dimwits Preznits on which they can safely prey after the subpoenas start flying and George Bush and Neoconservativism vanish under weight of the blood-dimmed tide they created. Cast down, down, down into eternal infamy as the twin pillars of the worst Administration in American history.

This is the dawning realization that is scaring the fucktard rank-and-file of the GOP into ever more hysterical public behavior.

This is the sinking ship from which George Will and Tom Friedman are desperately trying to flee.

This is the existential kill-box of their own making that the Republican Party is now frantically trying to weasel out of, because of fucked projects and politics I know these six things to be true:

First, a fucked project at a bank for 100K, at the DMV for $5 million, at the FBI for $170 million, and in Iraq for $2 trillion are all depressingly similar in every characteristic except for order of magnitude of the costs and the scale of the consequences of failure.

Second, the oiliest “process reengineering consultant” or “visioneer” is no different from Ann Coulter or Rush Limbaugh, because there will always be Big Cake to be made pouring comforting lies into the ears of stupid, hateful people.

Third, these disasters take wing because Stupid Men with Power will always listen to flatterers and liars who tell them what they wish were true, and will always scream “Traitor!” at those who try to tell then what is actually true and stop the division/company/country from being stampeded into the abyss.

Fourth, as the project collapses, the criminals who created it will strive mightily to blame their failure on those who tried to warn them that this was a bad idea from the start. On badthinkfulness and a lack of sufficient corporate piety and prayer. God knows if they’re actually stupid enough to believe that a deficit of Tinkerbell Clapping is what’s driving their organization into the grave, but other than immediately admitting their error and atoning, this is the only card they have left to play.

Fifth, eventually they fall. They always fall, and they take lives and treasure down with them in an arc that was as predictable as the sunrise from the first moment they decided they could bribe Reality into changing the rules just for them.

Sixth, the funding that will be available to clean up the GOPs rubble and ruin will be vastly less than the cost of the original misadventure.

And the same people who broke the bank and spared no expense on hooch and hookers for the Big Iraqi Cakewalk Party, will be the first to bitch and whine over pennies and taxes when it comes to cleaning up the ocean of blood and vomit, debt and tragedy their short and despicable reign will leave behind.

21 comments:

Anonymous said...

Drifty, I think you should be the ProseHat Laureate of the new, shiny, clean-'em-up administration. Love your vision, as always. Heard from Lucy yet?

Anonymous said...

PS. Is it drown-in-the-bathtub sized yet?

Anonymous said...

Yet another masterpiece...!

As an IT'er myself, I can testify that's exactly the way it works...

zombie rotten mcdonald said...

That last paragraph is such an accurate reflection of the reality of the coming cleanup that it's wildly depressing.

Thanks anyway drifty. Sometimes, the news IS bad, and as you note, ignoring it just allows the destruction to ratchet up.

jurassicpork said...

Yeah, I forgot that you were in IT, DG. I can see how that alone would piss you off even if it wasn't a corollary for the entire government's constant and ongoing fuckups.

Remember reading on DKos or any number of blogs a year or two ago about the raid on the NYC Federal Reserve when several US contractors demanded to be paid immediately? Two billion dollars stacked, literally, on pallettes that took up the space of 83 washers was hastily extracted in a manner that would've done Hans Gruber's brother proud.

That's what I thought of when I saw that picture of that pallette.

Anyway, on an unrelated note, you might be interested in what's going on within a Church in Chicago.

driftglass said...

This site neither endorses nor condones physical violence in the pursuit of political ends.

Period.

You say you want a Revolution?

Until the day comes when the jackboots can come down main street in the night and drag Joe Citizen away without due process to be murdered in the dark, when you start talking about killing, you can absolutely count me out.

Anonymous said...

I work at a Large Government Agency in Atlanta Frequently In The News (LGAAFITN). Thank you for putting my thoughts in pixels.

Frank said...

Drifty- I'm sure you made the right call there, but they have been calling for our deaths for decades.

Note that the first half of this: "Until the day comes when the jackboots can come down main street in the night and drag Joe Citizen away without due process to be murdered in the dark," has already come true. And that a little dumb luck and Jose Padilla would have met the last half, since the goons with the jackboots don't always calibrate their beatings just so. We may see you on the barricades sooner than you expect.

Anonymous said...

"So instead they redouble the bet, pour more money down the rat-hole, and fire the last, few, honest men within the organization who still dare to speak the truth and who still might be able to salvage the situation if given half a chance."

personally i don't belive it's ever been any different. This IS the cycle. I'm in BT, and it's the same e'FFn plan - to the T. Me Dah worked with SDI projects = same same; pour on cash, feed the egos, cash out, repeat. newton's laws be damned.

The beatings will continue until moral improves...

WereBear said...

Thanks for the latest insight into the reverse-Midas touch of the current administration. Having fished in those waters myself, I know what must have gone on.

To me, the core thought is how the Bushies HATE competence. Even something that should be straightforward, such as the creation of a database that works, gets mangled in the machine.

Dear heavens, they are ALL that employee we've known who works harder on the appearance of working that they would have to work to actually do the work.

Mister Roboto said...

The thing that's really galling is how Movemement Conservatives are always preaching about taking responsibility for the consequences of one's behavior, but the behavior of their power structure clearly demonstrates their belief that there's no such thing as consequences for the elite. "Taking responsibility" is for poor people who need to be thrown off welfare into minimum wage jobs with no benefits! (And if these suckholes had all of their way, there wouldn't even be a minimum wage. Or workplace safety laws, for that matter.)

joemax:

Very well put. I remember how back in the 80's when I was going to prep school, it was received wisdom among all the young Reaganites that the competitive viciousness that characterized the rise to the top of the corporate heirarchy was A Good Thing because that made sure only "the best" prospered and advanced. What they didn't realize was that the successful climbers were "the best" at being backstabbing conformist weasles whose entire career trajectory was driven by developing only the most superficial personal qualities.

Doug Bennett said...

We may have reached the point where additional complexity will no longer provide commensurate benefits, and will in many cases become the proverbial Wrench in the Works. Some may argue that productivity stats belie this, but I suspect that recent increases in productivity may be more the result of working more hours, rather than a benefit of technology. To wit: I certainly aspire to Lazy Hippie-hood, but yet find myself, more than 1/2 way through the year, having taken a grand total of..........2 days of vacation.

The natural foods coop from which my family bought recently decided to climb another rung of the tech ladder and institute online ordering. Said rung broke, and now said company now lies in a heap of broken bones on the ground; IOW, out of business.

At work, I'm basically playing Solitaire for the next few days until an IT nightmare gets resolved. I watch colleagues spend significant portions of the day trying, for example, to learn to write a note an their fucking Blackberrry, when writing it down on paper woudl have done the job.

Much like the Iraq debacle, what gets lost is the human element. Our company, which is tangentially in showbiz, apsires to play with the big boys in the 24/7/365 Bang! Shazaam! tech orgy. Those who make the decisions get major wood over gizmos. Meanwhile, our human interaction with those who buy what we sell continues it's decade long slide.

Out in the big world, the US talks to no one. Talk is for wimps. Once upon a time, we Negotiated With Terrorists; we sat down with Mao, with Chiang, with Stalin. No more; now, we just puch a button and see what happens.

cieran said...

Sir Driftglass:

Ahhh, the carnival hucksters of Information Technology and their Neoconmen partners-in-crime. Brilliant idea, connecting the dots on these disparate Ponzi schemes. How ingenious...

But the FBI/SAIC debacle is inexcusable in this day and age, because our federal government (and all serious IT developers) already learned these lessons long ago. Deities like Fred Brooks and Watts Humphrey learned from software debacles long-since-passed (e.g., Multics), and the Department of Defense begat the Software Engineering Institute at Carnegie-Mellon in large part to insure that DoD software failures would become a thing of the past.

Which for the most part, they have. Unfortunately for us, we now deal with new kinds of failures from DoD, e.g., ones with surnames like "Rumsfeld" or "Feith".

Apparently, the FBI prefers to learn these painful lessons lessons of software development first-hand. It's bad enough that the FBI won't share intelligence information with other federal agencies -- apparently, they refuse to learn from the past experience of other agencies as well.

Brilliant juxtaposition here, DG, one more time. Is there a Nobel Prize in Literature for Blogging? There oughta be...

Anonymous said...

AMEN, my brother!
Goddamnit! I spent much of yesterday, at work, cussing at the top of my lungs the dumbest computer system that I have ever had the misfortune to work on. I curse it every....fucking.....day.
It freezed up. It spits out random error codes. It objects to something that you entered in a field, but WON'T LET YOU ERASE THE FUCKING DATA! It will not let you execute a command---so you must log off, and reboot, then it's all happy.
Why do we have this piece of shit software, you may ask.
Our administer is a soul-less fuck who is boinking a dumbass bitch. So he "rewarded" her with this IT overseeing job. She ain't no degreed specialist. She hasn't even taking an intro class at the junior college. So, to make her look good, they had to hire an underling/helper for her----who is dumber than she is.
Then, the other three people that actually knew what the fuck was going on, had their IT administrive privileges revoked.
We are so fucked.

The Minstrel Boy said...

very well said drifty. i swear sometimes when i read the swoop and swerve of your stuff i can hear the brave dr. thompson cackling in the background while sorting through a bag of pills.

Anonymous said...

You are always good, driftglass, but that last paragraph is probably the most depressing (and on target) thing I have ever read. I can only echo so many others...we are so fucked.

tech98 said...

Dear heavens, they are ALL that employee we've known who works harder on the appearance of working that they would have to work to actually do the work.

And petty bullies and credit-grabbing backstabbers who jealously run down the people they know are better at doing the work than they are. I fled a well-known IT corporation that seemed to excel only at attracting, creating and promoting incompetent, mendacious butt-kissing asshole personalities like Bush, Cheney and Rumsfeld.

But the FBI/SAIC debacle is inexcusable in this day and age, because our federal government (and all serious IT developers) already learned these lessons long ago.

That's just the point - they're too arrogant to learn. Evolution and global warming were debates settled years ago by people who actually investigate evidence and learn from experience. But these clowns just *believe* things with cement-headed certainty, and nothing will budge them. And because they subconsciously know they're beliefs have no evidentiary basis they develop the neurotic defence of never admitting to any mistakes whatsoever.

If their beliefs turn into utter disasters like Iraq, it's because not everyone believed in the fantasy strongly enough. The dogma cannot fail, it can only be failed.

This is what happens when you elect a Hee Haw who can't think -- someone with contempt for science, a neurotic insistence on his own deity-ordained infallibility, simplistic black-white views and a mindset steeped in magical superstition.

With an authoritarian conformist culture, Brezhnev-style ideologically-correc, no-consequences incompetence and self-dealing corruption rots down through the entire federal buraeucracy, and you end up with arrogant party apparatchiks making disastrous, wasteful dumbass decisions at every level.

Mister Roboto said...

As depressed as this thread makes one about the social and political state of this country, it is nice to know that there is a thoughtful handful of people in this corner of cyberspace who "get it". An' I'm sure Uncle Drifty hisself would say the same durn ol' thang! :-)

Anonymous said...

Clearly, in the FBI case, the errorists won.

Anonymous said...

Wood or Vinyl Siding for Your Home..Great new Web Site on replacing damaged vinyl siding If you are looking for replacing damaged vinyl siding We can help....... Visit ud today at http://allhousesiding.com/home.pl?ceda=seamless&stucco=off

Anonymous said...

Bloggers

Looking for painting contractor in miami at http://doityourselfpro.com/. Well I recomend This Site. Maybe your asking who am I