thedrifter
07-11-06, 12:51 PM
July 17, 2006
Web of frustration
Are micromanagement and 2-hour help-desk waits worth NMCI’s unwieldy price tag?
By John Hoellwarth
Times staff writer
It was February, and Marine Aircraft Group 24 in Kaneohe Bay, Hawaii, had just switched over to the Navy-Marine Corps Intranet.
Anticipating the changeover, the MAG’s intel Marines stored their existing files offline with plans to load them into the new system when everything went online. But when the computers were fired up and the files had been loaded into the network, the Marines couldn’t get to them.
Enter Lance Cpl. Timothy Hudak, the shop’s self-described “low man on the totem pole.” It was his job to call the NMCI help desk and get it all sorted out.
His first call was dropped. He called again. This time, he waited two hours before he got someone on the phone who could hear his problem.
But that wasn’t the end. The voice on the phone told him that, even before he began holding, he must have pressed 1 on the phone when he was supposed to press 2.
He’d have to start over.
“After another hour of waiting and 10 minutes of explaining the problem, the new guy told me I had to call someone else,” Hudak said. “I almost broke my desk.”
Introduced in late 2000 as the world’s largest, most ambitious stride toward global connectivity since the advent of the Internet, the ultrasecure NMCI — overseen by Plano, Texas-based contractor Electronic Data Systems — has become a source of frustration for the expanding list of sailors and Marines who rely on it to do their jobs.
The network is huge. About 600,000 NMCI users make nearly 17 million transactions every minute — sending and receiving everything from official orders and secret documents to personal banking information. But ask around and it’s hard to find many Marines or sailors who don’t have at least one reason to rage against the machine.
“Without using expletives, I’ll just say there’s a lot of negative reaction when [NMCI] comes up in conversation,” said Lance Cpl. Scott Ellison, a company clerk with the headquarters battalion in Twentynine Palms, Calif.
In fact, Marines at public affairs offices throughout the Corps were unable to produce a rank-and-file Marine willing to speak in favor of NMCI.
Mike Coehler, EDS’ lead executive for NMCI, said both EDS and the government may have initially underestimated the scope of the project.
“Nobody has taken on a vision this large before. [EDS and the government] both met unanticipated challenges,” he said. “For everyone, the details were deeper and more complex than originally expected.”
One place sailors and Marines vent their frustrations with those “unanticipated challenges” is www.nmcisucks.com, an online bulletin board where 119 registered members, including EDS employees, offer advice on how to navigate the bureaucracy. Bulletin board topics at the Web site include “Printers Keep Disappearing,” “Fraud, Waste and Abuse? All Three!” and “How to Get Past the Bull$@#!”
Marines claim NMCI places their productivity at the mercy of over-the-phone technical support provided by corporate civilians who don’t share the work ethic and sense of urgency that Marines expect from each other. Calling the help desk is an ordeal in itself, they said, and persistent problems with e-mail, passwords and Internet connectivity require them to spend hours on hold to report problems Marines could otherwise fix in minutes themselves.
“The local IT guys are not able to help directly — you have to call the help desk,” said a Navy photographer’s mate second class deployed to the Middle East, who asked to remain anonymous for fear of repercussions over speaking critically. “And that takes time just to get through. ... [T]he help-desk guys are not enlisted — they are damned contractors working for a ton of cash.” Enlisted information technology specialists, he said, “are close by and plenty skilled.”
Lack of computing speed is another common complaint. A commander who works at Navy Personnel Command in Millington, Tenn., who also requested anonymity, said he keeps nonelectronic reading material close at hand so he can get things done at the office while waiting to open e-mail, which he said is a one- to two-minute process, or sending it, which can take two to five minutes. It typically takes two to three hours to go through between 20 and 25 e-mails, he said.
“I do a lot of my work at home and e-mail it to myself at work,” the commander said. “It’s like they’ve given us a Cadillac, but we’re all still putting regular gas in it.”
A sailor whose command still employs a “legacy” computer network linked to a server at San Diego Naval Medical Center said it’s a huge improvement over NMCI, which he used in March while serving with Surgical Team 8 in Norfolk, Va., when the team wasn’t deployed.
Using the Internet via NMCI, said Hospital Corpsman 1st Class Charles O’Dell, “reminded me of the days of 28.8 [kilobyte-per-second] modems — almost like the connection speeds on the ships we were on while crossing the Atlantic.”
O’Dell, now assigned to Navy Environmental and Preventive Medicine Unit 5 in San Diego, said the old system he now uses is faster across the board.
Growing pains
Rear Adm. James Godwin, program executive officer for the Navy’s Enterprise Information Systems, has heard the complaints but said lost e-mails, blocked attachments and Internet blackouts are growing pains in an unprecedented global network of computers connected to servers monitored by people who blocked roughly “10 million attempts to illegally get inside NMCI” last year. In short, some convenience may need to be sacrificed for security.
About 82 percent of the Marine Corps has been rolled over to NMCI. The commandant’s staff at Marine Corps headquarters made the switch in late June.
And there’s no going back now. On March 24, the Navy inked a deal with EDS to extend the $11.9 billion contract’s life span another three years, to 2010. At that point, the government can keep leasing the gear from EDS, go with a competitor, or buy it and put data communications sailors and Marines back to work.
The contract also puts teeth into the implementation plan, creating measures to make sure both sides — the government and EDS — meet their deadlines. And if something doesn’t work, the people involved need to explain why.
Still, Marines, sailors and observers say that the world’s largest global network, which came at a price roughly equal to Uzbekistan’s entire economy, may also be its biggest lemon.
Rep. Mark Kirk, R-Ill., said he stands by the comments he made during a March 8 meeting of the House Appropriations subcommittee on military quality of life and veterans’ affairs. At the meeting, which was attended by the heads of the Navy and Marine Corps, Kirk called NMCI a “waste of money.”
Kirk, also a naval reservist, said it’s his duty as “an appropriator for the military” to question spending $11.9 billion on a system that “is fundamentally hostile to people communicating.”
“There are a couple of ways to handle cybersecurity” that wouldn’t sap the defense budget as much, he said. “When you look at the Federal Reserve Bank and the way key financial institutions transfer billions of dollars, I think there are lessons to learn there.”
A short leash
Under the 2000 contract, EDS owns all of the hardware and software the government is leasing for NMCI. Under the agreement, the military can’t move a computer from one wall outlet to another without EDS’ consent.
MAG-24 data network specialist Sgt. Richard Weiss clarified the contract’s provision, saying, “In a four-wall room, we can move it, but we can’t take it through a door.”
Printer problems, password resets and login issues — which once were the province of each unit’s own data network Marines — are now the responsibility of civilian EDS employees at call centers in San Diego, Norfolk and Boise, Idaho.
Weiss said he has administrative rights over his unit’s computers only when they’re disconnected from NMCI for field exercises or deployments. In the field, Weiss can build a computer network from the dirt up and has enough pre-NMCI experience to troubleshoot any standard network problem. But he can’t send his new lance corporal to support exercises alone because he doesn’t have experience maintaining a network in the rear. That’s EDS’ job.
Under the NMCI contract, there is little that Marines can do for their unit in garrison except change the sergeant major’s screen saver and refer the computer problems of a command that “still expects things to get done as quickly as possible” into the black hole of the NMCI call center, Weiss said.
“I can’t install drivers for a printer or a scanner. I can’t change the time if the computer’s clock is wrong. I can’t change somebody’s rank in the global address list if they just got promoted,” Weiss said. “Jobs I could do in five minutes, it takes [NMCI] three days.”
Said the Navy commander in Millington: “If I was running a private business and I used NMCI for my network support, I would have already fired them. I’ve been in meetings with flag officers who say NMCI is the worst mistake that the Navy ever made.”
NMCI’s help desk has its supporters.
“I’m more than happy with customer service,” said Mass Communications Specialist 2nd Class Johnny Michael, who puts out a weekly newspaper at the Pacific Missile Range Facility in Kekaha, Hawaii. “They’re always more than willing to help.”
But, he added, “It can be a while, depending on the problem.”
His biggest gripe: waiting for NMCI to provide software upgrades, without which, he said, “We’re not able to produce as robust a product as our peers on the street.”
Waiting on NMCI for products and services isn’t just a garrison problem. Marines who deploy with EDS-owned hardware and plug their computers into the old-school tactical data networks maintained by Marines in the field must still rely on EDS to fix or replace something if it breaks.
“Shipments were lost. We would end up having to reorder parts directly from Dell because NMCI could not guarantee they would get the parts to us,” said Sgt. Michael Herrera, who returned in January from a seven-month deployment to Al Asad Air Base, Iraq, with Marine Wing Communications Squadron 28.
Capt. Wesley Woolf, 1st Marine Logistics Group communications officer, said broken and missing gear the Corps would have chalked up to war was held against the unit when it came time to deploy again. He said EDS tightened restrictions on the number of replacement computers and hard drives his unit was allowed to include in its “pack-up kit” for Iraq, citing financial losses from the last go-round.
Coehler, EDS’s lead executive for NMCI, said his company was initially flexible over the number of replacement parts it would give a unit for its pack-up kit, but that EDS later decided to standardize the kits across the Navy and Marine Corps because “doing it one way is a fundamental tenet of what we do.”
“Let’s pick the number, let’s enforce the number, and if we need to make adjustments, we will down the road,” he said.
Woolf said, “The problem I have with this is that we are in a combat zone and these people are telling war fighters what they can and can’t have.”
Security first
Godwin, the admiral, said that Marines and sailors who complain about NMCI need to learn what officials are up against.
“Those people are important. Those are customers. They have a voice. We need to address their issues, but we also need to educate,” he said. “It’s not just about whether you can attach a picture to your e-mail. It is bigger than that, guys and gals. You’ve really got to consider the global aspect of what [NMCI] represents.”
To get an idea of how big NMCI is, imagine taking 600,000 people and scattering them across the continental U.S., Hawaii and Japan, then linking them to a single computer network that handles one million e-mails and 720 billion Web pages each month. Now, keep hackers out.
The network is monitored by teams of sailors, Marines and civilians at the Naval Network Warfare Command in Norfolk and the Marine Corps Network Operations Security Command in Quantico, Va. When they notice suspicious activity on the network, such as repetitive login failures or an unusually high volume of data being transferred to and from a single workstation, “they’ll put a microscope on it and take a look,” Godwin said. “Then if they find a weakness, they try to shore up that area that’s weak.”
Godwin said NMCI means security for sailors and Marines to conduct business as usual even as their network is assailed by every ne’er-do-well from 14-year-old pranksters to spies and international terrorists “all day, every day.”
He used a hypothetical F/A-18 Hornet squadron as an example. He said if a hacker could penetrate one of the squadron’s computers and start poking around the unit’s shared files, that intruder might get ahold of unclassified yet sensitive information, such as maintenance data detailing how many aircraft were down and which parts were failing regularly.
“If you were just covert inside the network, you could start to draw a pretty clear picture on what our posture was across the board,” Godwin said.
Bad guys know this, Godwin said, adding that they are constantly probing the network for weaknesses, using password generators that can spend days and weeks trying every possible permutation until one of them works. The threat of “e-spionage” is greatest at Navy and Marine Corps units not yet a part of NMCI. Godwin said these units are where the Navy Department is most vulnerable.
“We’ve seen those kinds of attempts quite a few times,” Godwin said. “As soon as we can get everybody inside NMCI, we don’t have to worry about that issue anymore.”
Alan Paller, director of research for the Bethesda, Md.-based Sans Institute for computer security education and information, said hackers are too efficient these days for a noncentralized network to defend itself against attacks.
Each time Microsoft announces a patch — a supplement to current software that shores up identified vulnerabilities — hackers are engineering programs to exploit those same vulnerabilities within hours, Paller said.
It is only with a unified network like the one intended by NMCI that the Navy and Marine Corps can protect all of its workstations simultaneously and prevent hackers from exploiting weaknesses, he said.
“There is e-spionage going on, but you want to make sure your response doesn’t cripple your own organization’s ability to carry out its mission,” said Kirk, the Illinois congressman.
If the mission is to keep the Navy and Marine Corps communicating, then “you want to provide the secure environment to make sure hackers do not get in and disrupt your daily work,” Godwin said.
But from his perspective as a data network Marine in Hawaii, Weiss said the biggest threat to his daily work, as well as his career, is NMCI itself.
He said he might spend a few minutes each day explaining why the commanding officer’s printer won’t work and suggesting a call to the help desk. But other than that, Weiss said he does nothing related to his military occupational specialty.
As a sergeant, Weiss gets fitness reports that must include an evaluation of his job proficiency. But with his hands tied by the NMCI contract, he is prevented from making short work of the computer problems the Marines in his command still take to him first.
“My fitness reports don’t look that great because I don’t do anything,” Weiss said. “They’re not bad, but they’re not as good as they could be if I was doing my normal job.”
Ironing out the kinks
When NMCI was being pitched around the Pentagon in the late 1990s, one of its most vocal critics was Gen. Michael Williams, then the assistant commandant of the Marine Corps. He was never against NMCI, he said, but he felt the Corps was jumping in over its head and being swept downstream by an initiative the Navy needed more than the Corps.
“We had a rather heated debate, which obviously I came out on the losing end of, but my concerns were listened to at every level,” said Williams, now retired. “I was trying to get this to go slower, and I was unable to do that.”
He said the idea of building the largest network in the world “kind of makes you nervous” and that there were concerns about whether the services and EDS had the resources to pull off such an ambitious plan under time constraints.
Williams said he told the secretary of the Navy that he feared “we’d go backward capability-wise,” at least in the short term. He likened his position to the Hippocratic oath taken by doctors, which states, “First, do no harm.”
“I don’t feel bad that people didn’t do what I said. I feel bad we’ve wasted money,” Williams said. “We failed at due diligence before we jumped in.”
Godwin said EDS had experience administering large networks for the corporate world but didn’t fully understand the complexity of what it had promised the government by signing a contract now valued at $11.9 billion.
EDS, he said, headed into the program thinking, “Just because that’s the way it’s done at General Motors, we’ll just take that and plug it into the Navy or the Marine Corps.”
Godwin said the difference in complexity between a corporate data network and a secure, global “war-fighting tool” is like comparing $1,000 to a penny.
New wording in the recent contract extension adds what Godwin called “execution discipline” to the way EDS has done business so far.
Previously, EDS and the government didn’t have a standardized process for introducing the network to fleet units. Godwin said EDS representatives would show up at a command, count the number of workstations needed, deliver half on time and claim “that’s the best we could do.”
“You’ve got 2,000 people. One thousand are happy and 1,000 are not,” Godwin said. “It was just not taken under advisement that that was not a good thing to have happen.”
Coehler said EDS employees are under pressure to substitute government computers with EDS gear as quickly as possible, and in the past “went to the easiest locations first, versus focusing the effort on two, three or five at a time.”
Over the last 18 months, the Navy has been working to “put the rigor into the system,” holding EDS financially accountable for the implementation deadlines it fails to meet, Godwin said. Neither Godwin nor Coehler would give specific information on times and locations of failed deadlines, but both confirmed that deadlines had been missed.
The wording in the contract extension reflects the fact that the inefficiency that has plagued NMCI from the outset isn’t the fault of EDS alone. Under the extension, the government is subject to financial penalties “if there’s something we did on the government side” that prevents EDS from meeting its goals, according to Godwin.
NMCI hit a snag recently when it was learned that the Navy and Marine Corps were not informing EDS about users with permanent change-of-station orders. A Marine who moved from North Carolina to California would be able to log in at his new duty station, but NMCI would still consider North Carolina his home. E-mails to the Marine would still be sent and stored on an East Coast server, which slowed the performance of the Marine’s e-mail software, Godwin said.
“We took a good look, and we found that we moved so many times, that now we had 80,000 people that were not home to the right location,” he said.
Refinements to the NMCI network are ongoing at the Washington Navy Yard, where systematic testing and evaluation is taking place.
Godwin said, “We have some definitive findings, and we’re going to go through the entire network and look at the whole thing to resolve nagging customer satisfaction issues” that come with balancing information access with information security.
Paller, the director of research at the Sans Institute, said a single, unified network is a necessary evil.
“The underlying truth is that all federal computing needs to go to common configurations — must go, because you cannot secure systems quickly enough if everyone is on different networks with different configurations,” Paller said. “And so NMCI was this terribly problem-prone experiment that every other federal agency will learn from as they follow in almost identical lock step, trying to avoid the same problems that the NMCI people created.
“Somebody needed to make the mistakes to show everyone else the right path. It’s just kind of unfair to the Marine Corps and the Navy that they had to be the ones to experience the pain.”
William H. McMichael contributed to this story.
Ellie
Web of frustration
Are micromanagement and 2-hour help-desk waits worth NMCI’s unwieldy price tag?
By John Hoellwarth
Times staff writer
It was February, and Marine Aircraft Group 24 in Kaneohe Bay, Hawaii, had just switched over to the Navy-Marine Corps Intranet.
Anticipating the changeover, the MAG’s intel Marines stored their existing files offline with plans to load them into the new system when everything went online. But when the computers were fired up and the files had been loaded into the network, the Marines couldn’t get to them.
Enter Lance Cpl. Timothy Hudak, the shop’s self-described “low man on the totem pole.” It was his job to call the NMCI help desk and get it all sorted out.
His first call was dropped. He called again. This time, he waited two hours before he got someone on the phone who could hear his problem.
But that wasn’t the end. The voice on the phone told him that, even before he began holding, he must have pressed 1 on the phone when he was supposed to press 2.
He’d have to start over.
“After another hour of waiting and 10 minutes of explaining the problem, the new guy told me I had to call someone else,” Hudak said. “I almost broke my desk.”
Introduced in late 2000 as the world’s largest, most ambitious stride toward global connectivity since the advent of the Internet, the ultrasecure NMCI — overseen by Plano, Texas-based contractor Electronic Data Systems — has become a source of frustration for the expanding list of sailors and Marines who rely on it to do their jobs.
The network is huge. About 600,000 NMCI users make nearly 17 million transactions every minute — sending and receiving everything from official orders and secret documents to personal banking information. But ask around and it’s hard to find many Marines or sailors who don’t have at least one reason to rage against the machine.
“Without using expletives, I’ll just say there’s a lot of negative reaction when [NMCI] comes up in conversation,” said Lance Cpl. Scott Ellison, a company clerk with the headquarters battalion in Twentynine Palms, Calif.
In fact, Marines at public affairs offices throughout the Corps were unable to produce a rank-and-file Marine willing to speak in favor of NMCI.
Mike Coehler, EDS’ lead executive for NMCI, said both EDS and the government may have initially underestimated the scope of the project.
“Nobody has taken on a vision this large before. [EDS and the government] both met unanticipated challenges,” he said. “For everyone, the details were deeper and more complex than originally expected.”
One place sailors and Marines vent their frustrations with those “unanticipated challenges” is www.nmcisucks.com, an online bulletin board where 119 registered members, including EDS employees, offer advice on how to navigate the bureaucracy. Bulletin board topics at the Web site include “Printers Keep Disappearing,” “Fraud, Waste and Abuse? All Three!” and “How to Get Past the Bull$@#!”
Marines claim NMCI places their productivity at the mercy of over-the-phone technical support provided by corporate civilians who don’t share the work ethic and sense of urgency that Marines expect from each other. Calling the help desk is an ordeal in itself, they said, and persistent problems with e-mail, passwords and Internet connectivity require them to spend hours on hold to report problems Marines could otherwise fix in minutes themselves.
“The local IT guys are not able to help directly — you have to call the help desk,” said a Navy photographer’s mate second class deployed to the Middle East, who asked to remain anonymous for fear of repercussions over speaking critically. “And that takes time just to get through. ... [T]he help-desk guys are not enlisted — they are damned contractors working for a ton of cash.” Enlisted information technology specialists, he said, “are close by and plenty skilled.”
Lack of computing speed is another common complaint. A commander who works at Navy Personnel Command in Millington, Tenn., who also requested anonymity, said he keeps nonelectronic reading material close at hand so he can get things done at the office while waiting to open e-mail, which he said is a one- to two-minute process, or sending it, which can take two to five minutes. It typically takes two to three hours to go through between 20 and 25 e-mails, he said.
“I do a lot of my work at home and e-mail it to myself at work,” the commander said. “It’s like they’ve given us a Cadillac, but we’re all still putting regular gas in it.”
A sailor whose command still employs a “legacy” computer network linked to a server at San Diego Naval Medical Center said it’s a huge improvement over NMCI, which he used in March while serving with Surgical Team 8 in Norfolk, Va., when the team wasn’t deployed.
Using the Internet via NMCI, said Hospital Corpsman 1st Class Charles O’Dell, “reminded me of the days of 28.8 [kilobyte-per-second] modems — almost like the connection speeds on the ships we were on while crossing the Atlantic.”
O’Dell, now assigned to Navy Environmental and Preventive Medicine Unit 5 in San Diego, said the old system he now uses is faster across the board.
Growing pains
Rear Adm. James Godwin, program executive officer for the Navy’s Enterprise Information Systems, has heard the complaints but said lost e-mails, blocked attachments and Internet blackouts are growing pains in an unprecedented global network of computers connected to servers monitored by people who blocked roughly “10 million attempts to illegally get inside NMCI” last year. In short, some convenience may need to be sacrificed for security.
About 82 percent of the Marine Corps has been rolled over to NMCI. The commandant’s staff at Marine Corps headquarters made the switch in late June.
And there’s no going back now. On March 24, the Navy inked a deal with EDS to extend the $11.9 billion contract’s life span another three years, to 2010. At that point, the government can keep leasing the gear from EDS, go with a competitor, or buy it and put data communications sailors and Marines back to work.
The contract also puts teeth into the implementation plan, creating measures to make sure both sides — the government and EDS — meet their deadlines. And if something doesn’t work, the people involved need to explain why.
Still, Marines, sailors and observers say that the world’s largest global network, which came at a price roughly equal to Uzbekistan’s entire economy, may also be its biggest lemon.
Rep. Mark Kirk, R-Ill., said he stands by the comments he made during a March 8 meeting of the House Appropriations subcommittee on military quality of life and veterans’ affairs. At the meeting, which was attended by the heads of the Navy and Marine Corps, Kirk called NMCI a “waste of money.”
Kirk, also a naval reservist, said it’s his duty as “an appropriator for the military” to question spending $11.9 billion on a system that “is fundamentally hostile to people communicating.”
“There are a couple of ways to handle cybersecurity” that wouldn’t sap the defense budget as much, he said. “When you look at the Federal Reserve Bank and the way key financial institutions transfer billions of dollars, I think there are lessons to learn there.”
A short leash
Under the 2000 contract, EDS owns all of the hardware and software the government is leasing for NMCI. Under the agreement, the military can’t move a computer from one wall outlet to another without EDS’ consent.
MAG-24 data network specialist Sgt. Richard Weiss clarified the contract’s provision, saying, “In a four-wall room, we can move it, but we can’t take it through a door.”
Printer problems, password resets and login issues — which once were the province of each unit’s own data network Marines — are now the responsibility of civilian EDS employees at call centers in San Diego, Norfolk and Boise, Idaho.
Weiss said he has administrative rights over his unit’s computers only when they’re disconnected from NMCI for field exercises or deployments. In the field, Weiss can build a computer network from the dirt up and has enough pre-NMCI experience to troubleshoot any standard network problem. But he can’t send his new lance corporal to support exercises alone because he doesn’t have experience maintaining a network in the rear. That’s EDS’ job.
Under the NMCI contract, there is little that Marines can do for their unit in garrison except change the sergeant major’s screen saver and refer the computer problems of a command that “still expects things to get done as quickly as possible” into the black hole of the NMCI call center, Weiss said.
“I can’t install drivers for a printer or a scanner. I can’t change the time if the computer’s clock is wrong. I can’t change somebody’s rank in the global address list if they just got promoted,” Weiss said. “Jobs I could do in five minutes, it takes [NMCI] three days.”
Said the Navy commander in Millington: “If I was running a private business and I used NMCI for my network support, I would have already fired them. I’ve been in meetings with flag officers who say NMCI is the worst mistake that the Navy ever made.”
NMCI’s help desk has its supporters.
“I’m more than happy with customer service,” said Mass Communications Specialist 2nd Class Johnny Michael, who puts out a weekly newspaper at the Pacific Missile Range Facility in Kekaha, Hawaii. “They’re always more than willing to help.”
But, he added, “It can be a while, depending on the problem.”
His biggest gripe: waiting for NMCI to provide software upgrades, without which, he said, “We’re not able to produce as robust a product as our peers on the street.”
Waiting on NMCI for products and services isn’t just a garrison problem. Marines who deploy with EDS-owned hardware and plug their computers into the old-school tactical data networks maintained by Marines in the field must still rely on EDS to fix or replace something if it breaks.
“Shipments were lost. We would end up having to reorder parts directly from Dell because NMCI could not guarantee they would get the parts to us,” said Sgt. Michael Herrera, who returned in January from a seven-month deployment to Al Asad Air Base, Iraq, with Marine Wing Communications Squadron 28.
Capt. Wesley Woolf, 1st Marine Logistics Group communications officer, said broken and missing gear the Corps would have chalked up to war was held against the unit when it came time to deploy again. He said EDS tightened restrictions on the number of replacement computers and hard drives his unit was allowed to include in its “pack-up kit” for Iraq, citing financial losses from the last go-round.
Coehler, EDS’s lead executive for NMCI, said his company was initially flexible over the number of replacement parts it would give a unit for its pack-up kit, but that EDS later decided to standardize the kits across the Navy and Marine Corps because “doing it one way is a fundamental tenet of what we do.”
“Let’s pick the number, let’s enforce the number, and if we need to make adjustments, we will down the road,” he said.
Woolf said, “The problem I have with this is that we are in a combat zone and these people are telling war fighters what they can and can’t have.”
Security first
Godwin, the admiral, said that Marines and sailors who complain about NMCI need to learn what officials are up against.
“Those people are important. Those are customers. They have a voice. We need to address their issues, but we also need to educate,” he said. “It’s not just about whether you can attach a picture to your e-mail. It is bigger than that, guys and gals. You’ve really got to consider the global aspect of what [NMCI] represents.”
To get an idea of how big NMCI is, imagine taking 600,000 people and scattering them across the continental U.S., Hawaii and Japan, then linking them to a single computer network that handles one million e-mails and 720 billion Web pages each month. Now, keep hackers out.
The network is monitored by teams of sailors, Marines and civilians at the Naval Network Warfare Command in Norfolk and the Marine Corps Network Operations Security Command in Quantico, Va. When they notice suspicious activity on the network, such as repetitive login failures or an unusually high volume of data being transferred to and from a single workstation, “they’ll put a microscope on it and take a look,” Godwin said. “Then if they find a weakness, they try to shore up that area that’s weak.”
Godwin said NMCI means security for sailors and Marines to conduct business as usual even as their network is assailed by every ne’er-do-well from 14-year-old pranksters to spies and international terrorists “all day, every day.”
He used a hypothetical F/A-18 Hornet squadron as an example. He said if a hacker could penetrate one of the squadron’s computers and start poking around the unit’s shared files, that intruder might get ahold of unclassified yet sensitive information, such as maintenance data detailing how many aircraft were down and which parts were failing regularly.
“If you were just covert inside the network, you could start to draw a pretty clear picture on what our posture was across the board,” Godwin said.
Bad guys know this, Godwin said, adding that they are constantly probing the network for weaknesses, using password generators that can spend days and weeks trying every possible permutation until one of them works. The threat of “e-spionage” is greatest at Navy and Marine Corps units not yet a part of NMCI. Godwin said these units are where the Navy Department is most vulnerable.
“We’ve seen those kinds of attempts quite a few times,” Godwin said. “As soon as we can get everybody inside NMCI, we don’t have to worry about that issue anymore.”
Alan Paller, director of research for the Bethesda, Md.-based Sans Institute for computer security education and information, said hackers are too efficient these days for a noncentralized network to defend itself against attacks.
Each time Microsoft announces a patch — a supplement to current software that shores up identified vulnerabilities — hackers are engineering programs to exploit those same vulnerabilities within hours, Paller said.
It is only with a unified network like the one intended by NMCI that the Navy and Marine Corps can protect all of its workstations simultaneously and prevent hackers from exploiting weaknesses, he said.
“There is e-spionage going on, but you want to make sure your response doesn’t cripple your own organization’s ability to carry out its mission,” said Kirk, the Illinois congressman.
If the mission is to keep the Navy and Marine Corps communicating, then “you want to provide the secure environment to make sure hackers do not get in and disrupt your daily work,” Godwin said.
But from his perspective as a data network Marine in Hawaii, Weiss said the biggest threat to his daily work, as well as his career, is NMCI itself.
He said he might spend a few minutes each day explaining why the commanding officer’s printer won’t work and suggesting a call to the help desk. But other than that, Weiss said he does nothing related to his military occupational specialty.
As a sergeant, Weiss gets fitness reports that must include an evaluation of his job proficiency. But with his hands tied by the NMCI contract, he is prevented from making short work of the computer problems the Marines in his command still take to him first.
“My fitness reports don’t look that great because I don’t do anything,” Weiss said. “They’re not bad, but they’re not as good as they could be if I was doing my normal job.”
Ironing out the kinks
When NMCI was being pitched around the Pentagon in the late 1990s, one of its most vocal critics was Gen. Michael Williams, then the assistant commandant of the Marine Corps. He was never against NMCI, he said, but he felt the Corps was jumping in over its head and being swept downstream by an initiative the Navy needed more than the Corps.
“We had a rather heated debate, which obviously I came out on the losing end of, but my concerns were listened to at every level,” said Williams, now retired. “I was trying to get this to go slower, and I was unable to do that.”
He said the idea of building the largest network in the world “kind of makes you nervous” and that there were concerns about whether the services and EDS had the resources to pull off such an ambitious plan under time constraints.
Williams said he told the secretary of the Navy that he feared “we’d go backward capability-wise,” at least in the short term. He likened his position to the Hippocratic oath taken by doctors, which states, “First, do no harm.”
“I don’t feel bad that people didn’t do what I said. I feel bad we’ve wasted money,” Williams said. “We failed at due diligence before we jumped in.”
Godwin said EDS had experience administering large networks for the corporate world but didn’t fully understand the complexity of what it had promised the government by signing a contract now valued at $11.9 billion.
EDS, he said, headed into the program thinking, “Just because that’s the way it’s done at General Motors, we’ll just take that and plug it into the Navy or the Marine Corps.”
Godwin said the difference in complexity between a corporate data network and a secure, global “war-fighting tool” is like comparing $1,000 to a penny.
New wording in the recent contract extension adds what Godwin called “execution discipline” to the way EDS has done business so far.
Previously, EDS and the government didn’t have a standardized process for introducing the network to fleet units. Godwin said EDS representatives would show up at a command, count the number of workstations needed, deliver half on time and claim “that’s the best we could do.”
“You’ve got 2,000 people. One thousand are happy and 1,000 are not,” Godwin said. “It was just not taken under advisement that that was not a good thing to have happen.”
Coehler said EDS employees are under pressure to substitute government computers with EDS gear as quickly as possible, and in the past “went to the easiest locations first, versus focusing the effort on two, three or five at a time.”
Over the last 18 months, the Navy has been working to “put the rigor into the system,” holding EDS financially accountable for the implementation deadlines it fails to meet, Godwin said. Neither Godwin nor Coehler would give specific information on times and locations of failed deadlines, but both confirmed that deadlines had been missed.
The wording in the contract extension reflects the fact that the inefficiency that has plagued NMCI from the outset isn’t the fault of EDS alone. Under the extension, the government is subject to financial penalties “if there’s something we did on the government side” that prevents EDS from meeting its goals, according to Godwin.
NMCI hit a snag recently when it was learned that the Navy and Marine Corps were not informing EDS about users with permanent change-of-station orders. A Marine who moved from North Carolina to California would be able to log in at his new duty station, but NMCI would still consider North Carolina his home. E-mails to the Marine would still be sent and stored on an East Coast server, which slowed the performance of the Marine’s e-mail software, Godwin said.
“We took a good look, and we found that we moved so many times, that now we had 80,000 people that were not home to the right location,” he said.
Refinements to the NMCI network are ongoing at the Washington Navy Yard, where systematic testing and evaluation is taking place.
Godwin said, “We have some definitive findings, and we’re going to go through the entire network and look at the whole thing to resolve nagging customer satisfaction issues” that come with balancing information access with information security.
Paller, the director of research at the Sans Institute, said a single, unified network is a necessary evil.
“The underlying truth is that all federal computing needs to go to common configurations — must go, because you cannot secure systems quickly enough if everyone is on different networks with different configurations,” Paller said. “And so NMCI was this terribly problem-prone experiment that every other federal agency will learn from as they follow in almost identical lock step, trying to avoid the same problems that the NMCI people created.
“Somebody needed to make the mistakes to show everyone else the right path. It’s just kind of unfair to the Marine Corps and the Navy that they had to be the ones to experience the pain.”
William H. McMichael contributed to this story.
Ellie