Sorry for the interruption of service this afternoon (~1600UT).
It looks like a bug in an API may have made a very large call to the database. This caused the database to fill up the temp directory and overflow the disk.
When we got the computer back up we detected some anomolies in both the website and the observations table. We got to work on fixing those, which took longer than I wanted. The checking of the AID DB again gave us disk problems. I worked to fix those by moving the temp directory to a larger disk. The check and repair of all databases went as it should at that point. Further, the problems we detected with webobs and the light curve generator went away as I suspected they would.
Please report any new anomolies you see.
What have we done to try to have this not happen again? mySQL traditionally uses the regular /tmp directory of the computer. This directory has grown too small so we've moved it to something 10x as big in an effort to give us more room. I did something similar to the log directory when it gave us a similar problem back in October.
Thanks, everyone, so much for your patience!
Hello Doc and All
Thanks for getting the web site up. It seems to load much faster now that you have made the changes. I hope this speed can be maintained.
I thought I contributed to it, since right after I uploaded 840 transformed observations and 49 untransformed observations, the website went down! I had to re-upload these as they "disappeared" when the site was back up.