A General AAVSO Data Infrastructure Report
"Space is big. You just won't believe how vastly, hugely, mind-bogglingly big it is. I mean, you may think it's a long way down the road to the chemist's, but that's just peanuts to space."
-Douglas Adams, The Hitchhiker's Guide to the Galaxy
And the above quote goes for its data as well. Space data is big, and its getting bigger all the time. We've ended up spending the last few months keeping up with its growth.
A couple of years ago with Geir Klingenberg we took our first small steps into what was then the up -and-coming paradigm of "cloud computing" (although if you ask some of us "cloud computing" is nothing more than an evolution of the client/server model we had back in he 60s.) We created a virtual Windows server - the first connected with the AAVSO in a long time - on the Amazon cloud to host Geir's program that would ultimately be called VPhot. Over time VPhot, deservedly, has become a big hit and we were literally the victims of our own success. The server, originally equivalent to a 1GHz 2007 Opteron 64-bit processor with 1.7GB of RAM, could not keep up with what it was being given either in terms of storage space or processing power. It needed to be upgraded. The server was a "small" Windows instance on the Amazon cloud. Mike Simonsen found the money and the donors to finance a "medium" instance that would have 2 1GHz 2007 Opteron 64-bit processors and 3.75GB of RAM, and also some more disk space.
Of course, Windows being Windows, upgrading this instance was not as easy or straightforward as a Linux instance. It has now passed into AAVSO legend that literally as one of our wonderful donors was writing a check to Mike for the upgrade, I was busy…erasing the VPhot server! Yep, true story. Fortunately, backups were available and we were able to recover most of what had been lost. Geir worked with me during a flu he had to get us back online. When I finally meet him again, I at least owe him dinner. Geir typifies the quality of member and volunteer we are blessed with at the AAVSO. More than any other organization I've ever been involved in, the AAVSO would not be what it is without our member/volunteers!
We traditionally had three main computers here at the AAVSO when I arrived. Mira was our web server. Occam was our database and file server. Blackhole was our firewall computer. This year we went through a couple of related moving/upgrading projects.
The speed of the website has always been a concern for us. Originally, Mira ran the website and Occam ran the database. Mira would talk to Occam when needed in order to access the database. In our original HTML and PHP-based website, this seemed OK since Mira would talk to Occam when someone accessed the AAVSO International Database, the Light Curve Generator, or WebObs. That was it. The web itself didn't need to talk to Occam. That changed last year when went to a Drupal-based website. Now the database was an integral part of the website and it was felt that having the two parts of the site on separate computers was slowing us down. So, we moved the database from Occam to Mira. This left Occam as a data processing server for both the AAVSONet and for APASS, and, as we found out, this was certainly a big enough job for it. At the beginning of this year Occam was running 12TB - 12,000GB - of disk space mainly for AAVSONet and APASS Data.
At the beginning of the summer Arne let me know that he was quickly running out of space. We needed to figure out how to get more. More than 12TB? I still wasn't used to the 12TB! Ultimately we decided to get a new physical instance of Occam - what I liked to call a "regeneration" - with 17TB of space and the option of attaching a JBOD - Just a Bunch Of Drives (Yes, that's actually the technical name!) to bring the storage capacity up even more when the time came. But Arne couldn't' stop working on APASS. He had a dead line for the 6th Data Release in time for the AAS Meeting in Alaska. So, I worked on setting up and configuring the new Occam while at the same time trying to make sure Arne wouldn't run out of space on the old Occam until he left for the AAS meeting. While he was in Alaska, I made the transition to the new server, now known as Occam. We figure that the new space will hold him until the end of the year, at which point we'll have to add the "Just a Bunch Of Drives" option to Occam which will bring his capacity to around 25TB. I joke that Arne will have me running petabyte servers before he leaves!
So, what happens to the old servers? The servers we use at the AAVSO are reasonably substantial, and usually we upgrade not because the older server has worn out, but that we've gone past its pretty reasonable resources.
When we last upgraded Mira, the web server, three years ago, the old server, re-dubbed "Old Mira" was repurposed as a web development environment for Will, our Webmaster. Old Occam regenerated into "Olcott" and is being used as an APASS/AID database development server for Matt and I in conjunction with a Virtual Observatory project we have going for those particular databases.
Finally, we've added one more server to our little family here - Cirrus. One of Aaron's last heartfelt projects before he left the AAVSO was putting the web server on the Amazon cloud. We felt and hoped that putting the web server on the cloud would break us out of the last bottleneck - that of bandwidth. On the cloud the web server could take advantage of Amazon's bandwidth, and not our slower link at HQ. Will & I had gone through a concentrated week period where we proved that we could, basically, transition the web server to the cloud by building a prototype. But after that the project sort of stalled. Arne realized he needed more space on Occam, so Occam's upgrade took priority. Meanwhile, Aaron was getting ready to move to Chicago. I suspended the Cirrus instance on the cloud, but kept the hard drive with all the work that Will & I had done.
Then, a week or so after Aaron left, Arne & Rebecca decided that now was the time to move the web server to the cloud! While the cloud instance we'd selected worked wonderfully for the prototype, when it was actually put under a normal load, the instance proved to be too light. Fortunately, it was almost trivially simple, and fast, to upgrade Cirrus. It is now an instance with the equivalent of 8 64-bit 1GHz 2007 Opteron processors and 15GB of RAM.
Mira remains the staff work server and the email server. She is also configured as a warm backup for the web server should we need it.
So, in the last several months we've nearly completely redone and upgraded our entire server farm. It's certainly been a busy time. I have hopes that our membership has found it productive as well.