A gut-wrenching mishap surprised NASA days before reaching Pluto and nearly killed the mission

In humanity's search to closely probe our solar system and beyond, NASA just made one of its greatest achievements yet: On July 14 they flew their New Horizons spacecraft over the distant dwarf planet Pluto.

Of the original nine planets to visit, Pluto was last on NASA's check list. But the wait was well worth it: Their spacecraft, called New Horizons, has on board some of the most sophisticated instruments ever built. Without them, we could never have studied Pluto to the extent that we are.

But on July 4 - just 10 days before the scheduled historic flyby - that state-of-the-art technology worked just a little too well. It gave the New Horizons team the biggest scare they'd seen yet.

"OUT OF LOCK"

The gut-wrenching moment happened when a series of ones and zeros flashed across a computer screen in Mission Operations Center (MOC) that spelled out a disturbing message in binary code which read:

"OUT OF LOCK."

For the team, these were some of the worst three words they could ever hear from the instrument. It means that all communications have been lost.

But for the New Horizons team, this message was especially disconcerting for two reasons:

1. The spacecraft was just 10 days out from its closest approach to Pluto. If it was still out of lock with Earth by the time of the flyby, the team wouldn't be able to communicate with the instruments on board to collect the data they had been waiting nearly a decade for.
2. Since the spacecraft was so close to Pluto, that meant it was very far from Earth - 4.5 light hours to be exact. That meant that whatever made it terminate communications with the crew had taken place 4.5 hours ago. A lot can happen in that amount of time.

After receiving the message, Missions Operations manager Alice Bowman assembled the team's project manager, Glen Fountain, and mission leader, Alan Stern, to MOC in Johns Hopkins University's Applied Physics Laboratory (APL).

They wouldn't be going home that night.

"I stayed on the floor, and it was probably one of the best 15- or 20-minute sleeps I've ever had," Bowman told Joel Achenbach of the Washington Post for his inside story on the crisis and how it was resolved.

A major crisis

Achenbach wrote:

The official story from NASA and APL officials over the next two days was this was an "anomaly," and the team had resolved the issues and gotten the spacecraft back in shape for the Pluto flyby. But this was no mere glitch. This was almost a disaster. This was, as Stern would later admit, "our Apollo 13."

Shortly after NASA announced the "anomaly," scientist and senior editor for The Planetary Society, Emily Caldwell, wrote in her blog article, "This is scary. It's not what the team wanted to be dealing with right now."

With only days to resolve the issue on their spacecraft that was rapidly closing in on Pluto at 32,000 mph, covering 1 million miles every 31 hours, the New Horizons team had no room for error.

So, they quickly went to work.

Hours later, at 3:11 pm on July 4, they re-locked with the spacecraft by connecting with the back-up computer that had taken over the spacecraft's communications.

But the crisis wasn't over. In order for the backup computer to kick in, something had to have gone amiss with the spacecraft. The team had designed the spacecraft's sophisticated computer with an "autonomy" system that could decide what to do on its own if it detected a problem.

This autonomy system gives the spacecraft some independence on how to handle a situation, which is something you want when it takes you hours to communicate with your instrument. It also means that the team had limited control over what it would do if something did go amiss.

What's more, to further protect itself, New Horizons had kicked into safe mode. So even though the team could talk to it again, all of the scientific instruments were frozen, incapable of operating.

No instruments meant no science.

The sophisticated instruments on board could only function in the spacecraft's operation mode. But to get it back into that original mode would mean a lot of different steps and commands, a lot of reconfiguring and rebooting. And each command would to take 4.5 hours to reach the spacecraft.

At 10:21 a.m. Tuesday - three days after "OUT OF LOCK" flashed across their screens - MOC received another message from New Horizons saying it was "healthy, on track, [and] wide awake," Achenbach described.

The message came just in time: Fourteen hours later, the team programmed the spacecraft for its encounter with Pluto.

What NASA didn't tell us

NASA announced on Tuesday that they had resolved the "anomaly" and the mission was still on track to fly over Pluto on July 14. What they didn't say was what exactly had caused the anomaly. It turns out that the spacecraft became overloaded.

In other words, the spacecraft's main computer got overwhelmed.

It was compressing newly-gathered scientific data while, at the exact same time, it was supposed to be completing other commands that the team had programmed it to execute. The two processes were not supposed to coincide - the computer just decided to compress the scientific data at a very unfortunate time.

As a result, the computer got overloaded, its system recognized something amiss, and it shut down all noncritical systems and kicked into safe mode.

The mishap challenged and ultimately proved the team's remarkable ability to perform under pressure, and in the end, very little data was lost.

There were no photos the spacecraft was planned to take on July 4, only a few scheduled for July 5, and one for July 6, according to Lakadawalla. Although those few photos were never taken, the spacecraft made it to Pluto and the images it took during its fly over are far more impressive.

The New Horizons team has put the mishap far behind them and are now busy drooling over the data that the spacecraft is transmitting to Earth and that will continue to come in for the next 16 months.