Good data, good science: simple advice for better CCD photometry
One of many hats that I wear at the AAVSO is to act as good steward to the AAVSO International Database, one of the AAVSO's greatest assets. Part of the work of stewardship is not just to collect and archive data but to encourage the collection and use of "good data". I don't know of a standard definition of "good data" but I'll take a stab at one: good data is accurate information representing physical reality that enables researchers to productively create and test new and better models for the behavior of the world. More succinctly, good data is that which is useful for doing good science.
The AAVSO International Database is filled with "good data", and the quality of data obtained by the AAVSO observer community is generally very high. But we sometimes see data that won't be useful for science, and it's unfortunate because that's not only a lost opportunity for science, but lost opportunity for both researcher and observer. I traded emails with a researcher the other day, and to paraphrase one blunt comment he made, he said 'I deleted data from observer [XYZ] from my analysis because I don't trust their observations.'
That made me cringe. I bet it makes some of you cringe, too.
It was clear there were several problems with the CCD data set being questioned, and they happened not just for one star but for several stars this particular observer looks at. They're not just bad nights -- they're improper observing, analysis, and reporting practices. Anybody can make the occasional mistake, but it should be a rare exception. When things consistently go wrong, we need to address them. I came up with a short list of things to keep in mind when observing with CCDs -- some will be obvious, and some perhaps not so.
1) Use good observing practices
Take flats and apply them correctly, make good calculations of exposure times (being careful to avoid both under- and over-exposure), and use photometric filters. If you're able to measure your transformation coefficients and properly transform and extinction correct your data please do so. Many of these issues are covered in the AAVSO's CCD Observing Manual, and we encourage you to read through it if you haven't yet done so. The AAVSO also offers CHOICE courses that help you establish and use best practices, and we encourage you to give these a try.
2) Unfiltered data should be taken with care
If you don't transform your data or use filters at all, it's certainly possible to take useful scientific data, but pick your projects carefully. You can easily observe a blue cataclysmic variable without a filter, but an unfiltered observation of a Mira can produce such misleading results as to be worthless. Wavelength responses of unfiltered CCD cameras and telescope systems vary wildly from system to system. For a star that is uniformly bright at all visual wavelengths (like a cataclysmic variable in outburst) it doesn't matter as much. But for a red giant whose spectrum is dominated by red and infrared light, the difference between a filtered V-band measure and an unfiltered measure assuming a V-band comparison star magnitude can be off by several magnitudes. That's too large a difference to be physically useful without knowing much more about your CCD than your observation can tell a researcher. Unfiltered CCD observations should be left for cataclysmic variables and other blue objects.
3) Do not overobserve stars: either make few high-quality observations, or combine multiple measures into a single data point
Some stars vary on rapid timescales, and for those stars a rapid observing cadence with exposures of a few seconds or minutes is required. But for stars that vary on timescales of weeks or months, you do not need to use such a rapid cadence. Spend your time making a few high S/N observations in one or more filters, and then observe more stars in a night in the time you'd otherwise be sitting on one star. Unless you have a good astrophysical reason to submit a large number of rapid observations for a single star, you should combine rapid time-series into a single measurement. A researcher of a long-period variable would much rather see a handful of high-signal to noise observations in a single night than thousands of low signal-to-noise time series measures.
That said, sometimes you do need to do rapid time series. Use what you know about astrophysics to make the best judgement about your observing run, and if you're legitimately looking for some high-speed phenomenon, use the appropriate exposure time. But also realize that short exposures usually lead to lower signal-to-noise, and you should be aware of what S/N you need to detect the variations you expect. If you're unsure how to make observations of a given star, post a question to one of our new Forums, or please ask one of us at Headquarters!
3) Do not underobserve (bright) stars.
This is mainly a problem for bright star observers. Scintillation strongly affects measurements of bright stars, and is a major source of scatter in bright star measures. You can decrease the effects of scintillation by making many observations of a bright star and its comparison stars and then combining them into a single measure. This is how photoelectric photometrists observe bright stars -- they make at least three measures of the variable, and five of the comparison star for every single "observation" that they submit. That's why PEP photometry regularly reaches the 1 percent uncertainty level. [They also transform and extinction correct their observations, but that's a whole different post.] If you're observing a slowly-varying bright star like epsilon Aurigae or P Cygni with a CCD or DSLR, take multiple exposures over a short period of time and combine them into a single measure. Again, your "totals" will decrease but the quality and usability of your data will be greatly improved.
4) Measure your errors for every single observation.
Some observers will report one estimate for their CCD accuracy for all time, using that value over and over again for every single observation they make -- that is not correct. You can measure the errors of your photometry every time you make a measurement, so please do so! Uncertainties are a critical part of interpreting data and doing science, and not knowing the uncertainties hinders those trying to use your data. The AAVSO CCD Observing Manual has a discussion of how to do this. You may also want to consult this paper by AAVSO observer Michael Koppelman, presented at the 2005 SAS meeting.
5) Carefully follow the reporting instructions for the Extended Format file, and be as complete as possible.
A description of the AAVSO extended format is here: please go over it carefully and make sure you (or your photometry software) follow it correctly. We obviously need to know the details of variable measurement (JD, magnitude, filter), but knowing the details of your comparison and check star measurements and your airmass will help researchers interpret your data more easily. We ask for all of those fields for a reason, and if you can fill them in, please do!
When you provide comparison and check magnitudes, they should be instrumental magnitudes, not the catalog magnitude that you assume for the comparison. If you're using a comparison star magnitude from an AAVSO chart, we know the catalog magnitude already (assuming you provide the chartID!), so tell us what your system measured it to be. If you're using a comparison star magnitude from another source, then put the assumed magnitude in the comments, not in the Cmag field.
6) Please review your data prior to submission, and if something is clearly wrong, please fix the problem before submitting them.
A number of problems we see with photometry should be obvious prior to submission to WebObs, so please check your data prior to submitting them. Automated pipelines can process a lot of data very quickly, but don't let it lull you into a false sense of complacency. Such pipelines may also generate a lot of bad data just as quickly as good data. Review your data before and after processing, both images and photometry. Do the point-spread functions of your star images look right? Did you see any evidence of saturation? Did the seeing change during the run? Were there tracking problems? Is your photometry software measuring the right variable, comparison, and check stars? Are your photometry apertures appropriate for all images being analyzed? Do the numbers being output by your photometry software make intuitive sense to you?
Many users keep track of their photometry with spreadsheets, and nearly all spreadsheet software includes the ability to make graphs of data. Do you plot your photometry (magnitude versus time) before you submit? Do you plot comparison star and check star magnitude as well? A single plot with light curves of your variable, check, and comparison star will sometimes let you catch obvious problems before data make their way to the AAVSO International Database. Some problems like an incorrect filter name might be obvious when compared to adjacent measures in the correct filter. Sky problems may also make themselves obvious if you see wild variations in your comparison and check magnitudes along with your variable.
Your CCD data are available to the community as soon as you hit "Submit". CCD data are not validated as visual data were during our Validation Project -- so please take the extra step to make sure your data are valid before you send them. And if you find that something might've gone wrong with your observations, don't be afraid to contact AAVSO headquarters. If you submitted data that has some problem, please let us know and we'll be happy to either flag them or, if they're genuinely in error, delete them. We'd rather catch data before they make it to the AID than have a researcher say "we couldn't use the data."
One of the practical consequences of following all of these guidelines may be that you might wind up submitting fewer observations per year, and it will take longer to reach those CCD milestones in your Annual Totals. Time spent slewing from star to star takes time away from taking data. And if you sacrifice a great photometric night to making standard star observations for your transformation coefficients, that's even more time you're not observing variables. Please realize that's ok. Science is not a totals game. The Annual Totals have been a convenient (and fun) metric as to how much observing you do, but the point of observing is to increase our understanding of variable stars. We want you to do as much variable star observing as you desire, but more than that we want you to make the best observations that you can, that will advance variable star science. The number of measurements submitted does not tell you what your impact on science is. Stay focused on the science, not the Annual Totals.
We know our CCD observers want to do great science and the majority of you definitely do. These guidelines may be obvious and common practices for many of you, and if so, great! CCD observing definitely has a learning curve, but even if you're at the top of the curve, there are still opportunities to do something new and better. And if you're new at CCD observing, it's best to start off on the right foot. We hope we can help you along!