Skip to main content

A Very Bad Mistake

Posted by Aaron Price on March 30, 2011 - 12:16pm

In 1998 I was hired as the AAVSO's "IT person". At the time the AAVSO had a 56K frame relay connection to the Internet (your current smartphone probably has faster access) and as much copper coax cable as ethernet. We were a MSDOS/Windows95/Windows NT network with one CDROM in the entire building that staff had to share. I was a former webmaster with extensive Windows NT experience and almost no UNIX experience. The AAVSO had a Linux box that ran the web, FTP, DNS and e-mail servers. Managing it was contracted out. I was 23-years old with a grand total of 2 years of sysadmin experience under my belt.

A few months into the job I decided to teach myself Perl so I could code something I called at the time "SuperStar" (it would eventually become our light curve generator). One day Elizabeth noted that our FTP directory of star charts had a mismatch of upper and lower case letters in their filenames. She recommended making them consistent. I decided such a tedious task could be made more palatable with perl.

Did I mention I was 23? Yes? Well, remember that.

So I wrote a script that would recursively go down the directly structure and rename everything to all caps. Except I made two mistakes. First, I recursively went in the wrong direction. That is, instead of going deeper into the file structure the program went up and away, all the way to the root directory. Secondly, I ran it as... root.

23.

My friendly UNIX-aware readers are now gasping because they know what happened next. For the rest of you: The program renamed every directory and file in the root directory to the correct name, but uppercase. So "/home" became "/HOME". In UNIX, I may as well have translated everything into piglatin - or Klingon. The system could no longer find any file that it needed.

E-mail, web, FTP and staff access to the Internet went kaput. We called our contractor who raced over in a couple of hours. He was flummoxed. At one point he asked me "what's an IP address?". It was that point that I realized our contractor had no clue what he was doing and it was up to me to fix this mess.

Next up was a weekend crash course in Linux. It was love at first sight. I was completely blown away about how logical and sensible it was. Most of all, I was amazed to see that there were reasons why things broke and that meant there were ways to fix it! I was coming from the world of Windows NT, where things just broke randomly and the only way to fix anything was to reboot endlessly and pray.

Slowly over the years we moved away from Windows and now the network is all UNIX-based (OS X or Linux) with one Windows laptop and one Windows workstation - both used for specific and narrow tasks.

That was an embarrassing mistake. But I am reminded of the quote by Neils Bohr:  "An expert is someone who has made all the mistakes that can be made in a very narrow field." I'm no systems expert, more mistakes have since been made and far, far more are yet to be made. But the quote used to hang over my desk and made me feel better.

Categories:

Comments

Loved you story! For years I managed networks for a large corporation. When I was interviewing possible team members one of my last questions was - "What was the biggest mistake you ever made and how did you fix it?" If they told me they had never made a big mistake they were either dodging the question or had never done anything important.

I remember standing in a server room,  alone, flourescent lights humming, 36 hours into an upgrade I had botched, Sunday morning 3 am, four hours before the first employees  would walk thru the door and I had no idea how I was going to get the system back up. I remember calculating how far I could drive in those four hours! I got out of the mess 5 mins before the division VP walked in and said. "morning george, you look like ----!" . All I could do was smile, my hands were still shaking.

This is one of the best stories I have read in a while. I especially like that part about how the folder names might as well been written in Klingon.


Qapla! well, maybe not...

AAVSO 49 Bay State Rd. Cambridge, MA 02138 aavso@aavso.org 617-354-0484