What is an acceptable error?

Affiliation
American Association of Variable Star Observers (AAVSO)
Tue, 04/20/2021 - 19:25

I've been playing around with measurements taken from standard star field SA32. I picked ten reference stars and one other reference star as my test star. I've previously worked out my TCs. For each of the ten reference stars I worked out the magnitude of the test star with and without transformation. Then I averaged the ten magnitudes. The results I got were -

Catalogue B = 12.118

Average B without transformation 12.102  SD 0.07

Average B with transformation 12.116 SD 0.02

Catalogue V = 11.313

Average V without transformation 11.330 SD 0.03

Average V with transformation 11.324 SD 0.015

Are these the kind of figures you'd expect? Would I be ok to submit these measurements?

 

Affiliation
American Association of Variable Star Observers (AAVSO)
Error

Steve:

Considering the experimental rigor you used to make these analyses, it is a little disappointing that you had to ask the question about the quality of your data. Take a look at any target in the LCG2 and you can see how poor the precision and accuracy of typical magnitude measurements can be. Do you think your random and/or systematic errors look better or not?

Your precision (repeatability/random error) is indicated by your standard deviation of your test star mean magnitude from 10 comps (ensemble). You could alternatively take 10 images and calculate the test star mean magnitude from one comp (or all ten comps again). In fact, the latter process with 10 images is more typical of  a normal calculation of precision. I recommend that you try this interesting experiment and even report your results in this thread. However, as you may have surmised, these two procedures should probably give similar precision. Perhaps not identical but similar. There is some potential difference in independence of measurements when using 10 images as opposed to one image with ten comps? In any case, standard deviations of a few 0.01 mags (0.01 to 0.07(?)) is reasonable (not unusual) for replicate, independent images with modest equipment in suburban skies. One finding you might investigate is whether the largest standard deviation (0.07) results from a uniform/normal distribution or not? Is there a particularly unusual/divergent magnitude among the ten mags? Was there perhaps a faint comp or poor image among the group? Outliers are always possible in any experimental process.  BTW, getting to standard deviations of a few 0.001 mags is difficult / requires more effort (e.g., guiding).

Your accuracy (bias/agreement/systematic error) is indicated by the difference (or percent difference) between the catalog/known magnitude and your calculated magnitude. It is not derived from nor the same as the standard deviation. You have bias/differences that range from 0.002 to 0.017 mags. 

I would conclude that your system/process yields very accurate and moderately (typically) precise magnitudes.

The reassuring thing that you have shown is that your imaging, reduction and transformation procedure yields more accurate magnitudes than non-transformation. The other interesting observation is that the removal of systematic errors (bias) in your filter images, through transformation, also tends to improve precision (standard deviation). IMHO, you have done a very good job!

Your last question about submitting this specific data is not clear. Why would you want to submit magnitudes of non-variable stars to the AID? If you are asking the more global question, I think your equipment and process is reliable and you should feel comfortable submitting your magnitudes (non-transformed or transformed) of your target variable stars.

Ken

Affiliation
American Association of Variable Star Observers (AAVSO)
Acceptable error

Ken answered your question very well. Let me give another perspective with a concocted example.

Imagine that you made 10 measures. Five measures were exactly 0.1 magnitude brighter than known magnitude. The other five measures were exactly 0.1 magnitude fainter than the known magnitude. That is, none of the measures were accurate. However...

The average would be exactly accurate. But, the standard deviation of +/- 0.1 would indicate that you need to examine the quality of your data.

So, what your results might show is that (1) you have non-systematic scatter in your individual magnitudes, some higher and some lower (low SNR?) or (2) you may have some outliers that effect the standard deviation. One reason for taking a taking a series rather than a single science image is the "outlier problem." A few bad apples and all that.

My general impression in my own work is that my precision (standard deviation) is better than my accuracy when working with Landolt standard stars as targets.

Ed