[Aavso-photometry] Sources of accuracy and precision in photometric measurements

[Michael Newberry]

This morning there was a thread on the main discussion group discussing the 
photometric accuracy of software. Following on that, I thought it would be 
useful to describe for AAVSO members some of the critical issues that were 
used in developing Mira's photometry algorithms. Some of these may be new 
concepts to many people and will provide some food for thought. As far as 
algorithms go, I can assure people that Mira and other software packages do 
not share the same algorithms except, perhaps only in the generic name of 
the technique.

I will presume that any programmer can write software to compute a sum of 
pixel values. But there is a *lot* more involved in getting good magnitudes 
from images. This gets into the fuzzy world of algorithms---some "good" and 
others not so good.

There are 4 main areas which affect doing accurate, high-precision 
photometry:

1) the sky background estimate
2) the partial pixel algorithm
3) the centroiding algorithm
4) the random error estimates (sigma of magnitude) must be valid

These issues have the strongest effect at the faint end where many of us do 
the interesting work. Here are some specifics:

1. Sky estimation:
Mira offers 3 methods: Mean, Median, and Mode. Knowledgeable photometrists 
have their favorite. And no single method is necessity the best in all sorts 
of images. But generally speaking, the mode method is the best choice. But 
just because a software authors says "ours measures the mode" does not mean 
it measures the mode or gives the same mode as other software. Measuring the 
mode is conceptually straightforward, but in real data, it is a real 
challenge. One person's "mode" is not another person's "mode", but that is 
another topic! If calculate correctly, the mode can give the most consistent 
results when stars and hot/cold pixels contaminate the sky measuring 
annulus. Our website at http://www.mirametrics.com has an example showing 
the effect of background contamination when doing photometry of the minor 
planet Xanthippe (this is linked from the Product Briefs menu item and also 
at the upper right corner of each product page). In this study, 9 images 
were measured for a moving object. In the first couple of measurements, a 
star 0.5 magnitudes brighter than Xanthippe is located in the sky annulus. 
This is usually a horror story for a photometry algorithm. With Mira AL, the 
magnitude estimate of the object was biased only 0.01 magnitudes by the 
contamination. That is darned good performance for a sky rejection 
algorithm.

2. Partial Pixels:
Around the rim of the measuring aperture is a tail of pixels that only 
partially sample the image. How they are used in the magnitude calculation 
is very important. Only Mira handles partial pixels *exactly*--- no 
approximations. No other software on the planet does this for the general 
case (elliptical or circular apertures in Mira AP and Pro, and circular 
apertures in Mira AL). The mathematics is outrageously complex to implement. 
We know that Mira does it right because the algorithm has proven in the 
field by more than a decade of use. How do we know each measurement does 
this correctly? The area of an ellipse is (pi * a * b) using the aperture 
size the user specifies. A circle has a = b, so area = pi * r^2. Mira 
tallies the sum of areas as well as the weighted sum of pixel intensities 
based on the partial pixel areas. If the difference is 0.001 pixels or more, 
Mira pops up an error box asking the user to contact us. During the Mira AL 
beta testing cycle someone contacted us about that error, but it turned out 
there was a different bug that allowed the measuring aperture to be only 
partially on the image. So the algorithm is time-proven. And particularly 
with small apertures, and for faint stars, it does make a difference in the 
precision of the photometry.

3. Centroiding noise:
This is a *widely* neglected source of photometric error. Placing an 
aperture on the center of the light distribution would seem a simple thing 
but it is not, especially for faint stars. I would invite everyone to look 
at a study I did of centroiding noise as a part of the photometric error 
budget. This is shown on our website at http://www.mirametrics.com. On the 
left side menu, click on "Virtual Library" and then see the 7th item down 
the list of topics. In item 1 above, I mentioned an example on our website 
showing measurement of a moving object. The position of moving object and 
standard star were both automatically centroided by Mira on each image. The 
internal consistency of the photometry between images is near the milli-mag 
level, and this value doubtless includes some variation in the asteroid 
brightness itself due to rotation. If Mira did not do robust centroiding 
with high accuracy, the centroiding noise itself would have dominated and 
the magnitude consistency between the images could never have been anywhere 
near the milli-mag level.

4. Good estimates of the magnitude uncertainty:
Of course, the estimates of magnitude errors (random errors!) must be done 
correctly for doing science. But, like the other 3 items, getting error 
estimates also is not straightforward. To give photometrists the information 
they need, Mira quotes 2 errors: an empirical error measured from the noise 
in the data, and a theoretical error based purely upon detector properties, 
aperture area, and the signal. The two estimates are independent except for 
the magnitude value (which is not important in the issue of independence). 
The empirical error estimate should scatter around the theoretical values 
and, the brighter the star, the more they should agree. In Mira, they do.

Michael Newberry


----- Original Message ----- 
From: "arne" <arne at aavso.org>
To: "AAVSO Discussion group" <aavso-discussion at aavso.org>
Sent: Wednesday, December 14, 2005 6:36 AM
Subject: Re: [AAVSO-DIS] Canopus software for variable star work


>> Tom Richards wrote:

[...]

> Arne wrote:
>
> The AAVSO has been considering doing some tests.  The problem is that
> every vendor is sure that their algorithm is correct, and if you
> don't get the right answer, then (a) you aren't using the latest version
> of the software, (b) the software was not designed for that specific
> case (such as trailed asteroids), or (c) you aren't using the
> software correctly.  These are valid points, of course, but published
> tests are always going to result in vigorous discussions.
>
> The easiest first step is to supply a set of images that everyone can
> use to test their software.  That is what I am working on right now.
> Arne
> _______________________________________________
>
> Aavso-discussion mailing list
> Aavso-discussion at mira.aavso.org
> http://mira.aavso.org/mailman/listinfo/aavso-discussion
>