A Thunderstorm in the Cloud...

I'm sure folks who are interested in such things saw the reports that the Amazon.com Cloud experienced a major failing all over the East Coast. Several major websites were down and affected, including ours.

In simple terms, something was causing problems with data drive connections within a region of the Amazon cloud where our web server is located. Traffic simply couldn't get to the virtual computers. The data was fine. Nothing was lost, certainly not on our server, but it acted as if the computers couldn't find their boot disks.

This was the case between approximately 14:00-23:00 Eastern Time for the AAVSO. From what I have been able to find out, they are still working at getting all the instances working and are not done yet, but communications between our disks and our server is now working, and I expect it to remain so.

I have gone through and verified our data integrity and the AID and the databases are good. The logs show exactly what I expected them to show.

Everyone was suprised at how long this took to fix. We considered moving the web server back to Mira as things dragged on, but it would have taken time for that change to propagate through the Net, I found, and then time for it to propagate back once things were fixed. We do have Mira set up as a "warm backup" and I will be looking hard at specific methods of implementing that over the next few days. Originally that option had been set up in case we'd sustained a massive failure to our data or had been vandalized. This was not the case and everyone was expecting the issue to be solved at any minute. We weren't in a situtation where we knew the problem would take several hours or days to solve like we would be if we had lost data. Perhaps that part of our response could have been handled better, and I'll be looking at that as well.

In the end we apologize - *I* apologize - for the downtime. We really have done pretty well in that department, and I think you can hopefully expect a similar level of service in the future.

Thanks very much, and, of course, please contact me with any questions.
----
Doc Kinne
AAVSO Astronomical Technologist
rkinne@aavso.org [1]