Making the robots robotic
The AAVSO's network of automated telescopes, AAVSOnet, has been a challenging system to set up. From an engineering standpoint, all of the people dealing with hardware (especially Arne Henden, Tom Krajci, Tom Smith, and John Gross) have put in an enormous amount of work to get these systems up and running, and a lot goes on behind the scenes that the average user of AAVSOnet probably doesn't see -- which is in itself a testament to how well the system works. From a software standpoint, a lot has gone on as well. Many of you are probably familiar with Geir Klingenberg's wonderful VPhot site where you can analyze your images. That also took a lot of work to get it to where it is right now, and it's a wonderful tool for the community, amateur and professional alike.
One aspect of AAVSOnet that I've been working on with the staff at Headquarters has been the pipeline that glues everything together and gets data from the telescopes to you. The processing package for all of our telescopes has grown in stages, from Arne's original IRAF and FORTRAN programs -- originally all run by hand by Arne -- to the current system where all of the processing is now done automatically via cron jobs on our computation server. It's been a fun project to work on with our Astronomical Technologist, Doc Kinne and Technical Assistant, Sara Beck, and I'm pleased with how everything has come together. We've got a great system that greatly reduces the amount of human intervention that goes into AAVSOnet data processing, and greatly increases the speed at which you get your data.
There's a lot that happens behind the scenes to get you your AAVSOnet data. The local Telescope Operators have to transmit the data to us every night via ftp, and then we take over for the rest of the processing. When there's raw data available to process, we set in motion a script that does all of the work, getting the results to the staff for review and to you for your own use. We use a Perl-based master script to control the processing and move files around the network, and to trigger automated versions of Arne's IRAF and FORTRAN programs to do the processing work. We also create logs of all the data processed in MySQL, and Sara Beck's Java-based Remark-o-matic image review tool greatly increases the speed and efficiency with which staff can review data every night. What used to be a task that took Arne several hours of time per day to do on his own is now done automatically by our servers, and our staff only take about 15-30 minutes per day to do a quality check of the data.
Doc Kinne and I reached a milestone in AAVSOnet processing this week by setting up the entire system to run autonomously via a UNIX cron job. On July 6, we had our first night of fully automated processing on one of the telescopes, the Wright 28. The staff here used to launch the nightly processing script manually, but we finally got all of the bugs worked out to the point where we felt the server could do everything on its own. Everything seems to check out ok, and so we're now ready to say we're "fully automated". We'll probably hold off on switching all of the telescopes over to the automated system until after the monsoon season in the southwestern United States, but by September of 2011 I expect that all of AAVSOnet will be fully automated between the telescopes themselves and the data's final destination on VPhot or the AAVSO ftp site. The system is such that it will also be easy to add new telescopes to AAVSOnet as they're set up.
We certainly have some other things we can automate, and also some ideas for how to improve the system, but we're a long, long way from where we were a few years ago as AAVSOnet was just getting started. Most importantly, we've lowered the time required from Arne and the rest of the staff to get AAVSOnet data to the users. That gets you your data faster, and makes it easier for staff to work with AAVSOnet and do the many other tasks that it takes to keep AAVSO Headquarters going.
It's been a fun project, and although my main concern has always been to automate and improve the efficiency of the system, it's been a pleasure to do some hacking and create something new and useful. I do a lot of scripting and coding for fun during my off hours, and there's been plenty of that for Doc and I over the past 2-3 years. But the best part is seeing that all the hard work that's been put into AAVSOnet over the past several years by so many people is working together so smoothly. We have a few more advances that we plan to roll out in once the monsoon season ends in the southwestern United States. Stay tuned for those -- we hope you'll appreciate them!
I won't say that AAVSOnet is quite at the level of "plug and play", but we're getting there!