In the late 1990s, at Pixar, a few technicians were looking at the folder that held the files for Woody, the cowboy of Toy Story 2. Every time they refreshed, there were fewer files. Then the folder stopped existing. They went up one level and saw Hamm, Mr. Potato Head and Rex; a moment later only Hamm was left, then nothing. Someone, somewhere, had run a Unix command on the film's drive that deletes everything beneath it as fast as it can: rm -r -f *.
Oren Jacob, one of the associate technical directors, phoned the machine room and asked them to pull the plug on the main server, something you simply do not do in an environment with hundreds of connected workstations. When the machine came back up and the film's folder was measured, it was a tenth of the size it should have been. About ninety per cent of the work was gone.
The copy that was not a copy
The first reaction was the one any of us would have had: there's the backup. Ed Catmull, then running the studio, tells it this way in his book Creativity, Inc.: we would lose half a day of work. Jacob's account, written years later in the first person, is more precise and more instructive. The film was restored from backups within forty-eight hours, some tests were run, frames were rendered, the pictures looked good and the crew was called back to work. It took several days of everyone working before they understood that the restore was incomplete and corrupt. The backup software was not reporting errors properly, masked perhaps by a full disk; the validation software, written in a hurry, did not check some important cases. All along, the machine had kept answering that everything was fine.
At that point the last full copy that might be trusted was about two months old. In the meeting where this sank in, Galyn Susman, the film's supervising technical director, said she might have one at home. After the birth of her second child she often worked from there, and she had set up an automatic copy of the entire film database to her home computer, once a week. She and Jacob drove to San Anselmo to fetch it. They wrapped it in blankets, laid it on the back seat and drove back in the slow lane. The computer, Jacob recalls, was carried into Pixar like an Egyptian pharaoh.
Restoring as a habit
Even that copy was not enough on its own. There were well over a hundred thousand files. The ones identical between Susman's copy and the older one were taken straight away; the doubtful ones, about thirty thousand, were checked by hand, one by one, against three partial archives. The film came out anyway.
What interests me is the object more than the rescue: a grey box wrapped in a blanket, carried like a patient. For years it had been nothing but a domestic habit of no importance. It became precious precisely because it did not depend on the system that had failed. In the essay The Builder Operating System I describe the work of building as a set of routines and checks; the lesson of that week is that a check is worth something only if it tests the thing that matters. A backup that has never been restored is not a copy, it is a hope. Like the checklist born from the 1935 flight, a test restore is a dull gesture made beforehand, so that courage is not needed afterwards.