Forgot your password?
typodupeerror
Politics

Whitehouse Emails Were Lost Due to "Upgrade" 482

Posted by CmdrTaco
from the upgrading-to-orwell dept.
I Don't Believe in Imaginary Property writes "We now know how the Whitehouse managed to lose about five million emails. It seems that they 'upgraded' their Lotus Notes system, which had an automatic retention and backup system, for Microsoft Exchange, which did not support the automatic system. So they changed it to a manual process, where aides would manually sort emails one by one into individual PST files, which they call a 'journaling' archive system. They're still building a replacement for the retention system. Right when they had one finished, the White House CIO complained that it made Microsoft Exchange too slow, so they hired yet another contractor to build another one, causing a senior IT official to quit in protest. So they still haven't completed the project after almost eight years, and rely on humans to sort millions of emails."
This discussion has been archived. No new comments can be posted.

Whitehouse Emails Were Lost Due to "Upgrade"

Comments Filter:
  • by FuzzyDaddy (584528) on Wednesday April 30, 2008 @09:50AM (#23250710) Journal
    "Strategic Incompetence"
    • by PawNtheSandman (1238854) on Wednesday April 30, 2008 @09:56AM (#23250814)
      "Mission Accomplished"
    • by _Sprocket_ (42527)

      "Strategic Incompetence"
      Ahh! Strategery!
    • by Jerf (17166) on Wednesday April 30, 2008 @10:40AM (#23251434) Journal
      So, today the Bush administration is brilliant, and they came up with a genius plan to make email go away, while appearing incompetent.

      Someone be sure to send me the talking points when we're back to "The Bush administration is staffed by morons", k?

      (Such amazing IQ swings we see. Genius! Moronic! Brilliant! Ape-like! Bing-bam-boom! Sometimes several flip-flops in one day! One would almost wonder if the problem lies in the observers, rather than the observed.)

      "Never attribute to malice what can be adequately explained by incompetence." I think "incompetence" covers it just fine; I'm sure this is hardly the first migration screwed up this way.
      • by Chris Burke (6130) on Wednesday April 30, 2008 @01:33PM (#23253734) Homepage
        (Such amazing IQ swings we see. Genius! Moronic! Brilliant! Ape-like! Bing-bam-boom! Sometimes several flip-flops in one day! One would almost wonder if the problem lies in the observers, rather than the observed.)

        The problem lies in this ridiculous line of thinking where someone can only ever have one adjective applied, and that adjective must apply to everything they do.

        Here's the dope: The Bush White House is quite adept at playing politics -- genius when Rove was involved -- including yes the ability to make apparent incompetence into a strength. They are skilled at making the organizations they control work for them, producing the information they want to hear, and failing to find or losing the information they don't want anyone to hear, to support their political goals. When it comes to actually executing policies outside of Washington, they're terrible failures because in reality you can't get rid of facts you don't like and keep only the ones you do.

        What's so contradictory about that? I'm "brilliant" with computers, I'm "moronic" with cars. To think that one precludes the other is idiotic. But then again, so is the whole "flip-flop" figure of speech.
      • by Thing 1 (178996) on Wednesday April 30, 2008 @02:07PM (#23254198) Journal

        So, today the Bush administration is brilliant, and they came up with a genius plan to make email go away, while appearing incompetent.

        Well, yeah, except: what happened to the final backup tapes of the first installation?

        Since it's the last backup of that system it should definitely be marked for retention. And surely, as they realized that they had a retention issue with the new system, they would have ensured to maintain those tapes due to the Presidential Records Act that Bush himself amended?

        Also, doesn't it concern anyone that he changed the law regarding what communications can be released and when on Nov 1 2001, [fas.org] just three weeks after 9/11? Coincidence and circumstantial, perhaps, but concerning...

  • by zappepcs (820751) on Wednesday April 30, 2008 @09:50AM (#23250714) Journal
    read the summary and understood the Whitehouse is blaming Microsoft? hahahahahahahahahahahahahahahaha

    There is nothing that will happen for the rest of the week that can make me more light hearted than this. HAHAHAHAHAHAHAHAHA
    • by snl2587 (1177409)

      Or that the aides are manually sorting emails. What a crappy job.

    • by TobyRush (957946)
      Actually, when I read the summary, I thought, "Wait, when did "whitehouse" become one word?"
    • That would make a great marketing slogan for MS. "We take the fall for you!"

      This is why people choose Microsoft products. They know they will fail, but you can't be fired for choosing Microsoft, so when it does screw up, you don't look like an idiot.
    • I didn't get that from the summary, but this is one case where it really isn't Microsoft's fault. There are plenty of archiving systems built for Exchange, and some of them are really simple to install. Even if it weren't an ideal system, they could have gotten some kind of system up and running in a couple of hours, and it would be better than having people move messages into PST files.

      • by LO0G (606364)
        Heck, they could have used the journaling system built into Exchange [microsoft.com] and achieved what they wanted.

        • Re: (Score:3, Informative)

          by nine-times (778537)

          I think that's what they did. They turned on journaling, and then archived the journaling account to PST files.

          Unfortunately, this meant that a person was manually copying to PST, which introduces an opportunity for either human error or tampering. In addition, PST files aren't very good for this sort of archive. They've long had a history of getting corrupt as they grow in size, they're hard to search, and they don't have much in the way of built-in security controls. It'd be better to dump the files i

  • These days... (Score:5, Insightful)

    by neokushan (932374) on Wednesday April 30, 2008 @09:52AM (#23250740)
    It's getting harder and harder to tell the difference between subterfuge and sheer incompetence.
  • So now you're telling me it wasn't Alzheimer's that wiped out Reagan's memory?
    • by pohl (872)
      The Alzheimer's gambit was already played-out. They had to come up with something new for this administration.
  • by utnapistim (931738) <dan...barbus@@@gmail...com> on Wednesday April 30, 2008 @09:53AM (#23250760) Homepage

    What was that quote about never ascribing to malice?

    It's a well put-together story (plausible enough) but I'm still skeptic though.

    Maybe we've just seen too many lies :)

    • Had this been the excuse given on day one, it would have been a plausible example of incompetence. Given the amount of time since the scandal broke, it now looks like it took some work to craft a lie that appears to be plausible incompetence.
  • How Much Exchange and Windows lowers total Cost of Ownership. Sure This isn't MSFT's fault that lies strictly with with IT department, but if MSFT worked better with others this wouldn't be so much of a problem.

    Once your locked into MSFT's system you can leave easily. I have 15 year old email boxes that load up just fine in thunderbird, Apple Mail, pine, etc.
    • Re: (Score:3, Funny)

      by rbanffy (584143)
      "Upgrading our Lotus Notes with Microsoft Exchange allowed White House staff to cut jail time by more than 83%."
    • Re: (Score:3, Funny)

      by TheRaven64 (641858)
      If your TCO without Microsoft products losing your data includes jail time then I think that's pretty compelling evidence that Microsoft does lower your TCO...
  • Six P's (Score:3, Interesting)

    by gregarican (694358) on Wednesday April 30, 2008 @09:55AM (#23250794) Homepage
    Prior Planning Prevents Piss Poor Performance. This shouldn't be taken as a Micr$oft bash as much as an example of poor planning. After having administered both Lotus Notes and Microsoft Exchange I can say that ditching Notes for Exchange isn't a bad move. But doing so without planning out the migration path is. Any large scale project should involve a considerable outside contracting firm that would have automated measures in place. You could even plug in a server appliance before your front end Exchange servers that would automatically archive off mail messages being sent to/from the White House staffers. Another example of US government being inept. Just look to how the US air traffic control centers still operate with equipment that is so outdated that some units are out of commission because they can't order vacuum tubes to service them...
    • Re:Six P's (Score:5, Insightful)

      by Jeremi (14640) on Wednesday April 30, 2008 @10:09AM (#23251000) Homepage
      This shouldn't be taken as a Micr$oft bash as much as an example of poor planning.


      Or perhaps an example of really good planning. If I was planning to make sure a few million potentially incriminating emails never found their way into the public eye, that is how I might do it. Certainly if I had spent a number of meetings discussing how and when Americans should torture people [washingtonpost.com] I would be motivated to do so.

    • by ivan256 (17499)
      Exchange supports a small fraction of the functionality of Notes. As long as you only use Notes for e-mail and scheduling, moving to Exchange is fine.

      It's really too bad that most organizations running Notes don't understand all of the other things they can do with it.
  • by roystgnr (4015) <roystgnr@nOspAm.ticam.utexas.edu> on Wednesday April 30, 2008 @09:55AM (#23250796) Homepage
    The White House's failure to follow records retention laws was due to deficiencies in Microsoft software?

    I predict this will lead to a civil, thoughtful Slashdot discussion which results in many useful recommendations for avoiding similar problems in the future.

    I recommend fire.
    • by LizardKing (5245)

      Well, if it's a fire at the Whitehouse you want, then the spirit of 1812 [wikipedia.org] still burns in our hearts!

    • I recommend fire.
      Seriously, how are you supposed to burn an email? Print it out first?
    • Re: (Score:3, Funny)

      by Otter (3800)
      The White House's failure to follow records retention laws was due to deficiencies in Microsoft software?

      And it was completely unnecessary as Notes will happily set emails to zero length, without warning, if you archive them to a drive with no remaining space. No Microsoft products needed!

    • Re: (Score:3, Informative)

      by DigitAl56K (805623)
      The part that I find difficult to swallow is that they decided to change to Exchange knowing that it didn't meet the retention requirements and knowing that they've have to have aides sorting through millions of e-mails. I can't even imagine the kind of thinking that allowed that to happen, other than to make a *cough* "plausible" *cough* case for e-mail going missing.

      Still, how did such volumes of e-mail actually disappear? Either aides were sorting all e-mails into individual PST files and thus all the e-
  • Computation and Turing machines and all of that science stuff are just theories. The so called "experts" tell us that these things can sort email, but it's better to trust the people to get about the business of the government. I wonder if Bush looked into the eyes of Bill Gates and saw his soul? Maybe if we allow enhanced interrogation methods, we can recover the email? Ahh, too easy, I could go on for hours but I'll stop now.
  • Well we meant to backup up all these terrible incriminating emails but wouldn't you know it, there was a technical glitch.
  • by owlnation (858981) on Wednesday April 30, 2008 @10:01AM (#23250894)
    Basically this comes down to either:

    The Government was Incompetent.

    or...

    The Government is lying and covering up.

    Hmmm.... Mr Rock, meet Mr Hardplace.
  • the incompetence permeates all levels of this White House.

    LoB
  • by Reality Master 201 (578873) on Wednesday April 30, 2008 @10:02AM (#23250914) Journal
    Seriously, this is the least bullshit excuse the could come up with? If ANY corporation in the US tried this kind of thing, the wrath of SARBOX would rain down on them like you wouldn't believe.

    Even given the staggering incompetence of the Bush administration in nearly all aspects, this just doesn't pass the laugh test.
  • by metoc (224422) on Wednesday April 30, 2008 @10:03AM (#23250926)
    1) To bad the Whitehouse isn't using an e-mail system like millions of other people. Wait they are. Like it or not MS Exchange is everywhere.
    2) To bad the requirement for e-mail archiving and retention is unique to government. Wait, most publicly traded companies have legal and compliance requirements to do so.
    3) To bad there is no market for software to archive and retain e-mail on one of the most common e-mail platforms. Wait, there is, and its huge.
    4) To bad nobody has nobody has developed technology for this market. Wait, there are dozens of solutions.

    To bad no one is getting fired, imprisoned or impeached over this one.
  • by damn_registrars (1103043) <damn.registrars@gmail.com> on Wednesday April 30, 2008 @10:06AM (#23250968) Homepage Journal
    ... to how the Bush white house oversees "job creation":

    rely on humans to sort millions of emails
  • Are the people involved in this process, from conception to the current state, being held liable for criminal negligence?

    They're being heavily fined and potentially imprisoned for a blatant disregard for government policy?

    Is there anybody in a position to make in-depth enquiries regarding the processes involved in this fiasco, who has the wherewithal and political clout to actually do something about it?

    I didn't think so. Now bend over and get ready for another "Oops, we did it again!" situation.
  • by OpenSourced (323149) on Wednesday April 30, 2008 @10:08AM (#23250990) Journal
    and rely on humans to sort millions of emails.

    No problem. They had the job outsourced to India.

  • At best given the explaination provided, the emails are not lost, they are simply unsorted.

    Also, what about backup tapes? You don't do a major upgrade without a backup. Even the most slackjawed IT yokel (like me) knows that.
  • bloody hell. (Score:4, Insightful)

    by apodyopsis (1048476) on Wednesday April 30, 2008 @10:14AM (#23251092)
    utter bollocks. just unbe-fucking-lievable.

    *every* backup system should result in a set a of data offsite or in a storage area never to be touched again.

    even if you use incremental backup every nth backup should be a complete archival read only copy re the previous sentence.

    the *very* worst case should be the last major backup is in a format that is not readable with the current system and some red faced admins need help to read read the data.

    5 million emails? jesus wept.

    add the conspiracy theory factor into the mix and you have something that, on the face of it, sounds unbelievable.

    as one of our politicians in the UK said to another a short while ago "you cannot have it both ways, you were either ignorant or incompetent - and neither is acceptable".

  • be held accountable for this debacle?
  • Criminal? (Score:4, Interesting)

    by Relic of the Future (118669) <dales@di[ ]alfreaks.org ['git' in gap]> on Wednesday April 30, 2008 @10:20AM (#23251174)
    So, assuming for a moment their story is true and it *is* just negligence, incompetence, and stupidity; it is still FEDERALLY CRIMINAL negligence, incompetence, and stupidity, Right? Books will be thrown at those responsible, yes?
  • "The Bureaucracy must expand to meet the needs of the expanding bureaucracy." I wonder how much those human email sorters get paid? I bet one of them sabotaged the Exchange server
  • If only someone could come up with a device that can do repetitive work without error and without getting bored. Some sort of electronic mechanism that could look at a certain field in an electronic document and then put the associated text into an electronic bucket labeled for an individual.

    Why hasn't someone invented something like this?
  • by ashitaka (27544) on Wednesday April 30, 2008 @10:23AM (#23251230) Homepage
    One of my first projects after moving to Vancouver was a couple of test installations of pre-release Exchange server back in 1996. Since then I've worked constantly with every version of Exchange in all kinds of backup situations. Early versions of Exchange were a bitch to restore but it's gotten better.

    However, there has *always* been a way to retain and archive emails automatically from Exchange and no shortage of migration utilities from notes to Exchange. The reasons stated in the article just don't wash. No one, not even the newest tech school grad could come up with a system like that currently in use.

    However, it may in fact not be intentional malice from the start but more likely an existing state of incompetence that was taken advantage of to hide traces or misdeeds or at least to make finding any evidence difficult.

    This still doesn't address the use of non-government email systems for official business by Rove and other Republican members. According to the laws of the United States this is all highly illegal. Don't you care at all about what your government is doing or do you think whatever you do won't make any difference?
  • I lost a lot of email using Outlook and PST files for backups. After reinstalling Windows from scratch and trying to import the PST files into the new Outlook, 9 times out of 10 it couldn't import the email. That is why I switched to Thunderbird which has better success of backing up email files and importing them after a RRR (Reboot Reformat Reinstall).
  • They should just switch to hosted E-mail services with Google Apps :-)

    Seriously, though: the White House should not be in control of White House IT services. An independent agency should be responsible for that. It's all part of checks and balances.
  • I work for a fortune 500 company. We migrated from Notes to Exchange about 5 years ago. I liked Notes more.
    As an aside, Notes had support for POP3 so you could use any mail client you wanted.
    Exchange has its problems. One time a few years ago our company "focal" (lead supporting 50K people) could not resolve a bug which crept into my profile without deleting me from the system and re-creating my account. Unfortunately when this happens, all of the group mail lists and recurring meeting
    • Re: (Score:3, Informative)

      by atamido (1020905)
      I call FUD. Exchange supports POP3, IMAP, etc. Enabling/disabling it is trivial.

      With versions of Exchange prior to 2007, it was trivial to export an entire mailbox directly from the Exchange store (and reimport it later). I don't know why it would be necessary to delete an Exchange mailbox like that to fix a problem, but at the very least you could have copied everything from within Outlook to a local .PST. This would have saved everything except your rules.

      There are, and have always been, many good way
  • You can't argue that using Exchange in this case did not lower the total cost of ownership for the Whitehouse, by all indications, it would have been much more expensive for them if the emails had been preserved.
  • We need to have a single gov wide IT department and we need to keep it mostly in house with little to no contractors in it as they are slowed down by red tape. The armed forces can keep there own system under this as well as long as it is not the Navy/Marine Corps Intranet tape set up they same must have the same contractor rules aka the gov has full control and can step in at any time and take over.
  • They are still lying (Score:4, Informative)

    by Ralph Spoilsport (673134) * on Wednesday April 30, 2008 @10:30AM (#23251318) Journal
    everything that goes through the WH is:

    a: saved to tape and sent to a vault on a daily basis
    b: recorded by the NSA, who also saves and backs up data

    So, it's all a load of bullshit - they're thinking that the public is stupid enough to buy it, or, simply kick it down the road another month or two until the ADHD press finds something shiny to get distracted by like Miley Cyrus Boobs or another blast from Trainwreck Spears.

    RS

  • I know what that policy is. it's the "fuck you, we're covering our tracks and blaming microsoft" policy. I'm sorry, that doesn't fly around here. Someone broke the law. The white house CIO seems like the prime suspect. the presidential records act was violated, for all those who say, "what law?". Lost in an upgrade is what I expect from bill's plumbing and computer fixin's. It's even what I expect from enron. It is not what I expect from the white house. It shows an unparalleled level of incompeten
  • by Tom (822)
    When you're thinking in economics, it's easy to do the math:

    If the penalty for deleting mails that you are by law required to archive is less then the penalty for whatever those mails document, then it's the better choice to delete them.

    It really is that simple.

    And the only solution around that is one that's got its own problems, namely when you are required to have/show records in a case, and you can't or don't, you are assumed guilty and the penalty for deleting or not keeping those records is in addition
  • by Sleepy (4551) on Wednesday April 30, 2008 @11:03AM (#23251766) Homepage
    The industry is full of stories like this, for years. Exchange by default can't handle it - it's still a workgroup server at heart, and subject to many OS and filesystem limits. Does Hotmail.com even rely 100% on Exchange, or is it still UNIX at the core?

    De-centralized email storage and PST files?? COME ON!

    It is almost CERTAIN to expect that they knew this would cause emails to be lost and take the system from bad to worse. Even a junior IT person fresh off the boat would say this was CRAZY to attempt, with FEWER benefits and increased risk. In the corporate world, this would be met by massive civil lawsuits and possibly criminal charges. Any "contractor" the WH employed would know this for a fact.

    So given that such warnings had to have been given and they went ahead anyways, you have to wonder if strategic "loss" of emails was perfect cover for an email purge. Given the shady nature of these characters, I'm sure this was a calculated "feature".

Two is not equal to three, even for large values of two.

Working...