[Aavso-photometry] Cosmic Rays

[Michael Newberry]

Hi Wolfgang,

Thanks for your reply. From what you were describing, I was thinking it was 
the Maximum Value operation. It sounds like that is precisely what it is, 
plus a couple of other ideas. The idea of subtracting the median image from 
the Max, and then normalizing using the standard deviation image sound like 
interesting propositions. I am thinking that stars and galaxies will still 
be visible in all variations, as well as the transients like cosmic rays.

I have some other thoughts about the Max operation but let me first answer 
your questions about Mira. Other readers not wanting to hear about Mira 
*please* skip the following paragraph---I don't want this Email to be 
mistaken as some kind of disguised sales promotion.

    Wolfgang, first of all, I am not clear on your statement about the 
option of blinking images individually. Why is that bad? What software are 
you using? In Mira, you would just open the 75 images, or whatever number, 
and then click a button to animate them at some speed---unless you wanted to 
slide through them manually, or blink them individually. Normalizing is part 
of the combining operation and has a number of options to handle the 
different ways people may need to normalize the images. All intermediate 
normalization and combining calculations are done using double precision 
arithmetic and the results may be saved in any pixel format from 8 to 64 
bits real (just as for all Mira processing operations). If you want to 
Median or Max combine those 75 images, click a button (unless you want to 
modify some parameters of the processing). If you want to subtract that 
median from all 75, click another button. These types of "bulk operations" 
are rather commonly needed in doing serious work. No other software is built 
this way. It is amazing to look back, 15 years ago, we invented the whole 
architecture of Mira around working with "image sets". In Mira, the entire 
suite of displaying, processing, plotting, and measuring images is designed 
to make working with large ikmage sets as natural as working with individual 
images. You asked about combining options. These are listed on our website 
at http://www.mirametrics.com (or http://mirametrics.com works too). Click 
to a product page and then select from the links in the upper right corner. 
There is a product comparison matrix as well as a detailed list of features 
for each product---you will see combining options listed there. Mira is used 
for teaching and research at hundreds of US universities and colleges and at 
several NASA research centers. In Germany, it is used by people at a number 
of German universities, the European Space Agency, etc. --- as well as by 
numerous amateurs. And of course, it is also used by companies and 
government customers outside of astronomy. We have converted quite a number 
of IRAF users and have also been luring quite a growing number of refugees 
from IDL, if you are familiar with that product. I think our unique "image 
set architecture" is one of the reasons. Now, back to thoughts on the Max 
operation...

Subtracting the "Middle" image (a Median or Mean combined frame) would seem 
to have a value in flattening the large-scale trends in the background (like 
a slope or hump), if the background is not already level. But that operation 
shouldn't make stars and galaxies go away because the Max value scales with 
the signal level, so you just scale down the whole image by subtracting the 
middle value. It is not a linear scaling, but things do get scaled down 
according to their signal. The transients will also be there. Scaling the 
Difference image by the standard deviation image will increase the relative 
strength of the transients, but again, the "fixed" stars should still be 
there because the StdDev scales as the square root of the signal. Hence you 
will still see both objects and transients (e.g., cosmic rays) in a Max 
frame, a Difference frame, and a Scaled difference frame. But they will be 
of different relative strengths. Of course, the way these 3 cases (Max, 
differenced Max, and Scaled Difference) are *displayed* will also affect how 
you see things, and maybe this causes some apparent differences in the way 
they are viewed.

Is that what you have found?

You've piqued my curiosity for pushing this idea a bit farther. 
Unfortunately, I have already taken a lot of time typing all this and I have 
to rush out and do something right now. Later today I will do some 
experiments on some images and show what I find.

Thanks for the thought-provoking discussion!

Michael Newberry

----- Original Message ----- 
From: "Wolfgang Renz" <w_renz at onlinehome.de>
To: "Michael Newberry" <mnewberry at mirametrics.com>
Cc: "AAVSO-PHOTOMETRY" <aavso-photometry at mira.aavso.org>
Sent: Sunday, January 29, 2006 5:00 AM
Subject: Re: [Aavso-photometry] Cosmic Rays


> Hello Micheal
>
> You are right, just pickíng the maximum pixel value at each
> location will be sufficient for bias, dark and flat frames as
> these images have a pretty even ADU count level. All you
> might have to do, to see what you want is a simple contrast
> strech. In light frames it will also make transient objects
> appear as streaks or a series of dots and will pronounce
> the (single image) artifacts. It will do that much better than
> every median/sum/average combination algorithm.
>
> But for light frames (that have widely varing ADU counts) a
> "maximum pixel" image will also show the stars and nebula
> in the image. This might be sufficient for blinking against
> single and differently combined frames depending on the
> personal preferences and on what one wants to see.
>
> Subtracting a median combined, sigma rejected or similar
> image should get rid of any constant stuff and the artifacts
> and transients will remain. If you don't want to blink every
> image manually, you need something like an artifact mask
> that can be used in further steps by software.
> There might be also weaker optical artifacts like filter/lense
> reflections and diffraction spikes. If these stay at the same
> position, they should disappear in the subtracted image. If
> they move and/or rotate a bit between the different images
> (depending on the mount type, accuracy of polar alignment,
> dithering, ...), they should get visible as some kind of
> contured variing gradient area in the subtracted image.
>
> Light frames will have widely varing photon noise levels due
> to their widely varing ADU counts. To not show this as artifacts
> in the subtracted image (especially with stars), some kind
> of stddev, mean-min, median-min, meandev scaling should
> help to get these differences down to a more even level. Of
> course this will affect the absolute ADU count detectable in
> these areas.
>
> I'm not common with Mira. Do you have a link to a description
> of the different combination algorithms implemented in Mira ?
>
> Normalizing the image set is essential to get a valid median
> value. How do you do this in Mira ?
> Can it handle moving gradients due to dusk/dawn and moon
> rise/set ?
>
> Clear skies
>  Wolfgang
>
> -- 
> Wolfgang Renz, Karlsruhe, Germany
> Rz.BAV = WRe.vsnet = RWG.AAVSO
>
>
>
> ----- Original Message ----- 
> From: "Michael Newberry" <mnewberry at mirametrics.com>
> To: "Wolfgang Renz" <w_renz at onlinehome.de>
> Sent: Saturday, January 28, 2006 5:59 AM
> Subject: Re: [Aavso-photometry] Cosmic Rays
>
>> Thanks---I think I understand better now what you mean. Believe me, your
>> English is far better than my German!
>>
>> Once you pick the maximum pixel, I don;t see how scaling or subtracting
>> the median helps you get further detection. This happens in Mira at
>> least,
>> because the transfer function is automatically adjusted relative to the
>> median. You just change the stretch. One of the capabilities of Mira Pro
>> is rank combining.If you set the rank to 100% then you get the maximum
>> pixel value at each location. It makes transient objects appear as
>> streaks
>> or a series of dots.
>>
>> Is that what you are describing?
>>
>> Michael Newberry
>>
>>
>>
>> ----- Original Message ----- 
>> From: "Wolfgang Renz" <w_renz at onlinehome.de>
>> To: "Michael Newberry" <mnewberry at mirametrics.com>
>> Cc: "AAVSO-PHOTOMETRY" <aavso-photometry at mira.aavso.org>
>> Sent: Friday, January 27, 2006 9:34 PM
>> Subject: Re: [Aavso-photometry] Cosmic Rays
>>
>> > Hi Micheal
>> >
>> > In begin of October Dean C. Rowe sent an image of NGC 660
>> > with two brighter minor planet streaks ((13494) 1985 RT and
>> > (3580)) to the SBIG list:
>> > http://deanrowe.net/images/ngc660_asteroids.jpg
>> > In the JPG that must base on some kind of mean/sum combined
>> > image (as the very very frequent, bright cosmics are still visible)
>> > just these two minor planet are visible.
>> > But I discovered a third, much fainter one (2005 SN1, ~ 20.1
>> > magV) in his "raw image" annimation:
>> > http://deanrowe.net/images/ngc660_asteroids_animation.gif
>> > Its coming out of the core of NGC 660 and is moving to the top.
>> > In the animation its pretty obvious due to its movement. In a
>> > single image of the animation its close to the background.
>> > But in the mean/sum combined image is not visible at all.
>> > BTW: He missed its discovery by just about a month.
>> >
>> > Having a tool to find even this one would be helpful for minor
>> > planet hunting and for sure would be helpful to detect weak
>> > cosmics in images for photometry.
>> >
>> >
>> > For the "detection of cosmics" sequence of Ben I meant of
>> > course a) but on a per pixel base.
>> >
>> > But such a detection could be easily incorporated into a median
>> > combine (or more complex algorithm) that requires the pixel
>> > at the same position in the different images to be sorted anyway
>> > (by just picking the largest value). By comparing the difference
>> > of the largest value to the mean(/median) of the pixel and maybe
>> > even scaling it by the std(/mean) dev of the pixel one should get
>> > an acceptable estimator on how deviant the outlier is.
>> > But for weaker cosmics very close to the core of brighter stars,
>> > probably just a PSF-based estimator will give sufficient good
>> > results.
>> >
>> > Such an "higest absolute deviant combine" algorithm will require
>> > well registered images. And with undersampled images (without
>> > explicit dithering and without increasing the resolution by real
>> > drizzling), one might get issues due to intra-pixel sensitivity
>> > variations or a high share of non-sensitive inter-pixel areas of
>> > some CCDs. The same might be possible for short exposures
>> > that are seeing limited.
>> >
>> >
>> > Clear skies
>> >  Wolfgang
>> >
>> > -- 
>> > Wolfgang Renz, Karlsruhe, Germany
>> > Rz.BAV = WRe.vsnet = RWG.AAVSO
>> >
>> >
>> >
>> > ----- Original Message ----- 
>> > From: "Michael Newberry" <mnewberry at mirametrics.com>
>> > To: "Wolfgang Renz" <w_renz at onlinehome.de>;
>> > <aavso-photometry at mira.aavso.org>
>> > Sent: Saturday, January 28, 2006 3:14 AM
>> > Subject: Re: [Aavso-photometry] Cosmic Rays
>> >
>> >> Wolfgang,
>> >>
>> >> Can you elaborate a little on what you are describing?
>> >> Are you wanting to
>> >> a)  combine the images having the largest deviant, or
>> >> b) reject the largest  deviant from the combination?
>> >>
>> >> Michael Newberry
>> >>
>> >>
>> >>
>> >> ----- Original Message ----- 
>> >> From: "Wolfgang Renz" <w_renz at onlinehome.de>
>> >> To: "Ben Davies" <ben at davies.net>
>> >> Cc: <aavso-photometry at mira.aavso.org>
>> >> Sent: Friday, January 27, 2006 5:06 PM
>> >> Subject: Re: [Aavso-photometry] Cosmic Rays
>> >>
>> >> > Hi Ben
>> >> >
>> >> > If you average/sum combine the differential images, you'll lower
>> >> > the detectability of the artifacts as they will just appear in a
>> >> > single image. If you combine 10 images, its just 1/10 of the
>> >> > difference in the affected original, if you combine 100 images,
>> >> > its just 1/100.
>> >> >
>> >> > Using something like a "higest absolute deviant combine"
>> >> > instead, should lead to much better results. I don't know if
>> >> > such an algo is implemented anywhere, but it should also
>> >> > be helpful in finding moving minior planets that are just barely
>> >> > above the background.
>> >> >
>> >> > Clear skies
>> >> >  Wolfgang
>> >> >
>> >> > -- 
>> >> > Wolfgang Renz, Karlsruhe, Germany
>> >> > Rz.BAV = WRe.vsnet = RWG.AAVSO
>> >> >
>> >> >
>> >> >
>> >> > ----- Original Message ----- 
>> >> > From: "Ben Davies" <ben at davies.net>
>> >> > To: <aavso-photometry at mira.aavso.org>
>> >> > Sent: Friday, January 27, 2006 10:07 PM
>> >> > Subject: Re: [Aavso-photometry] Cosmic Rays
>> >> >
>> >> >> Here is a method I've come up with to analyze cosmic ray hits in
>> >> >> photometric images:
>> >> >>
>> >> >> - Make a median of all images.
>> >> >> - Subtract this median from each of the images to get a set of
>> >> >> images
>> >> >>   that contain all the artifacts.
>> >> >> - Average combine (or add, depending)  the artifact set.  Now one
>> >> >> image
>> >> >>   contains the defects and another contains the signal..
>> >> >> - Blink the averaged image against the median.
>> >> >>
>> >> >> If no artifacts fall on or near the stars you are interested in,
>> >> >> the
>> >> >> job
>> >> >> is done.
>> >> >> If an artifact does coincide, you just track down the offending
>> >> >> image.
>> >> >>
>> >> >> I'm probably just stating the obvious here, but it took me a while
>> >> >> to
>> >> >> figure it out
>> >> >>
>> >> >> Ben Davies
>> >> >> http://ben.davies.net/
>
>