My own experience is if you ask five pros about flat fielding for spectra you will get five different answers. In another forum on this same site there is a very animated discussion about flat fielding for photometry. Many of the points raised there apply here. Particularly because one main concern of flat fielding is that it is difficult to illuminate your field the same way that the sky illuminates it. Sky flats are really the only best solution for imaging. Dome flats are better than nothing but not even close to as good as a sky flat. However for spectroscopy sky flats are not an option because the sky is not a purely continuous spectrum.
So there are two main problems in spectroscopy flat fielding:
It is difficult to illuminate the slit identically the way that your target illuminated it.
You must illuminate it with a continuous source.
The way most professional spectrographs deal with this is to consider #1 a lost cause and do their best with #2. This means you put a bright continuous source with a diffuser in front of the slit and call that a flat. But because #1 is mostly a lost cause, your flat is never really a true flat, meaning it will not reproduce the blaze function or vignetting within the spectrograph in the same way your target does.
The conclusion many professionals reach is that this is hopeless so why flat field spectroscopy at all? Just ignore the step. When you set the contiuum on your target, flat fielding becomes unnecessary because you measure the fundamental parameters of the spectral features relative to the continuum. Especially since we can rely on modern detectors to have a fairly uniform response from pixel to pixel over one spectral resolution unit. When you set your continuum as a separate step you are pretty much wasting you time flatfielding most spectrographs.
Likewise it is pretty hopeless to even consider flatfielding a grating or objective prism for similar yet more complicated reasons.
The Hubble Space Telescope Imaging Spectrograph basically only uses the flat to correct for small amounts of "roll off" (basically vignetting) at the edge of the field. The relative flux calibration across the field is set by observing standard white dwarfs. This is not a practical option for calibrating most spectrographs!
So my own opinion is that an archive should specify that spectra NOT be flatfielded. However it doesn't really matter if they are. There may be a benefit of making it optional (but not mandatory) to include an observation of a standard star for some applications where setting the relative flux scale across the spectrum is desirable.
To make this more confusing, I always take flats when I do spectroscopy. But I guess that is more for my own feeling that I have done everything I can rather than any well defined practical reason.
Well done. Your response was excellent. After many hours of experimenting I have found that except for very special cases flat fielding spectroscopy while will not hurt, is a waste of time and if not done correctly can mess things up.
Something else to consider: any operation you perform on an image increases the noise in that image.
Let that sink in. If you have two images and you subrtact or divide one from the other the noise in the combined image is greater than the noise in either of the individual images that were combined.
We tolerate this with bias and dark subtraction because the removal of systematic error is preferable to a relatively small addition of random error.
With photometry and flatfielding, the benefit from flattening a field (even with a dome flat) has a greater benefit than the noise introduced to the corrected image by the operation.
With spectroscopy and flat-fielding, I am unconvinced that dividing a flat (with noise) has a benefit that exceeds the addition of the noise to the corrected image. Those of you who were at the workshop on spectroscopy I gave at the SAS meetingn two years ago might remember that I spent some time discussing this ( https://edocs.uis.edu/jmart5/www/SpectroscopyWorkshop/). The tutorial that I posted on simple spectral extraction ( https://edocs.uis.edu/jmart5/www/SpectroscopyWorkshop/ExtractTutorial/index.html) suggested that the best way to use a flatfield (if you really wanted to) was to:
Extract the flat.
Model it with a low order polynomial.
Divide the polynomial into the target spectrum
In that manner you at least don't do the harm of adding noise to your spectrum for the limited return you get from doing a flat.
While I have the confidence that many non-pros could learn how to do this on their own, I think it adds a step and some uncertainty that we do not want to introduce into data going into an archive. So that's why I maintain that my strong preference would be an archive of data that is not flat fielded with an option to upload an associated flat in case anyone wanted to dither around with it themselves at a later time.
As far as I can see, the question of flat fields is simple. If you can justify (potentially to a referee or your peers) that you can ignore the defects that flat fields are there to correct then go ahead dont do them. Otherwise it is good idea to take them. The problem is when you discover something interesting but could be real or a flat field defect, if you dont have a flat nobody will believe you. the subject of flat fields is a tough problem for amateurs and there is a lot to consider but it is something you ignore at your peril. We can argue about the potential deleterious effect on SNR etc but that is irrelevant if for example your carefully collected spectra are rendered unuseable by severe fringing which a flat could have removed. (something that happended to one of the posters on this thread)
I take flat fields so at least I can decide whether they are improving or reducing the quality of my data.
I think that about 95% of professional referees are not going to hold up a spectroscopic paper because the author did not flat-field. There is far greater potential systematic error in setting the continuum level, which is truthfully much more dependent on the experience of the end user than any particular calibration they performed.
Robin does make a good point about field defects. Almost all spectrographs have field defects. The Hubble Space Telescope Imaging spectrograph has a fringing effect at long wavelengths. I once presented a poster that included some unidentified "features" that were actually fringing. The poster was up all day at the AAS and only one person said, "I don't think those are real." Turns out they were not. Fortunately, I figured it out before publishing the result in a journal. A good fringe flat would have fixed the issue right away, but those calibrations were considered optional and had been omitted to make room for more exposures in the program the data came from. Lesson learned.
That said, my experience with flats shows that the flats rarely remove the defects completely. And sometimes they make it harder to identify the field defects that still remain. Another excellent professional spectrograph, the Sandiford Echelle on the 82-inch at McDonald observatory has an internal reflection lovinglly called "the picket" that takes out a substantial chunk of entire rows on the CCD. That can seriously screw up one or more orders. The flat does nothing to get rid of it and in fact the flat can make the issue worse by obscuring the picket just enough that it isn't obvious, leading the end-user to not realize that part of spectrum is completely worthless. However, the flat available for inspection, is helpful for identifying where it is and instructing you as to which parts of which orders should be omitted from your analysis. (Also the flat is used to model and subtract scattered light in that spectrograph so it serves an additionall application not commonly used for flats in most spectrographs.)
And I have also seen flats introduce field defects. Since the slit is never illuminated by the flat exactly the same way it is illuminated by the sky they can introduce bumps and shallow dips into the spectrum that were not there before the flat was applied.
We all through experience learn these things about our own telescope and spectrograph rig. But the challenge with an archive is communicating all that knowledge and information to the end-users in as seemless a manner possible.
Considering all this, I think that the best option for gaurding against the mis-interpretation of field defects is to not apply the flat to the target spectrum but have a flat available for inspection. The STIS example demonstrates the flat could sometimes identify a lurking prolem. But at the same time the Sandiford shows, applying it could obscure the problem just enough to make you forget its there (but it still screws up your data). Having it available as a check and a diagnostic is very helpful.
Yours and Robin's points about defects vs "features" and about the relative importance of setting the continuum are well put. If I might add from my own experience, hardly unqiue, that there are "features" in CCD spectra that have no apparent reason, observational or astrophysical. Having an archived flat available for CCD spectra, but not applied to the archived spectrum, could help decide the issue. The most important requirement of setting the continuum level - "where is the bloody thing?!" - is that the sections of the spectrum used to represent the "true" continuum have as high an SNR as possible. I have had the unfortunate problem of fitting the continuum to low SNR spectra of unusual objects, and it is perhaps the single most frustrating problem to deal with. (With photographic spectra, there are the problem of noise introduced by the emulsion grain, making one want to tear one's hair out!)
For any paper submitted to a professional journal which discusses spectra showing features that may or not be real, the referee should request that the author specifically address the question. I.e., is there a realistic or possible astrophysical effect that is the cause? Is there a new possible or plausible astrophysical reason that the author would like to propose? What might be (or is!) the source of the feature (defect)? Referees may not catch such issues, so the author should be on the alert for anything "weird" and explicitly mention it in the paper. That said, there are often spectral features, particularly in spectra of chemically peculiar stars which are often completely unexpected but whose presence is needed to understand the sources of the peculiarity - star spots, preferential surface abundance enhancements, etc. Some of the observed features are often weak, and having a series of averaged flats handy could be a make-or-break tool! I'd rather have a somewhat low resolution spectrum but with a high SNR than a somewhat higher resolution spectra with a low SNR.
This whole archive opportunity was to provide a "home" for amateur spectroscopy data - not a repository of necessary high quality professional standard data - hopefully with experience a bit of mentoring by the more experienced (and qualified members) they can aspire to reach that level of excellence.
Let's not get carried away by the the holy grail of "ultimate accurate spectral data - suitable for professional journals" etc etc. Novices have to start somewhere, have an opportunity of comparing their results with others and growing their abilities.......
I just noticed you proposed up the thread this procedure for using flats
" the best way to use a flatfield (if you really wanted to) was to:
Extract the flat.
Model it with a low order polynomial.
Divide the polynomial into the target spectrum
In that manner you at least don't do the harm of adding noise to your spectrum for the limited return you get from doing a flat. "
If you do this you are throwing away the wrong bit of the flat. The purpose of a spectroscopic flat is to eliminate the short wavelength effects (pixel/pixel sensitivity variation, dust bunnies, optical fringes, small scale variations in camera QE with wavelength(a big problem with Kodak CCD popular with amateurs when used with wide wavelength range spectrographs and a reason in itself to take a flat) etc, etc
The flat can be normalised to remove the large large scale features though if you like as they are eliminated once the instrument response correction is done using a reference star or failing that by normalising to the continuum (assuming you define where the continuum is - only generally possible at high resolution.) A typical technique used to normalise the flat is to use black body curve to approximate to the flat lamp continuum but it is largely a matter of preference provided the normalisation is consistent.
All this means is that there is indeed a risk of introducing some noise back unless you make sure your flat has significantly higher counts than your star spectrum.(eg >10x means you are introducing degrading your SNR by <10%. This is not difficult and is just a matter of good practise. It is not a reason for not taking flats. To me the trade off between eliminating otherwise difficult systematic errors and a marginally lower SNR is clear. I know which one I choose.
If this discussion was in the photometry section everybody would be in favour of flats. Spectroscopy is no different in this respect. All this sort of thing was thrashed out in the amateur spectroscopy community years ago in consultation with the various PI's on a number of Pro-Am projects. The debate has long since moved on from whether flats are important or not to how to make the best flats and how best to apply them.
You mentioned the difficulty of flats for slitless sprectroscopy. This is indeed a complex area still to be fully sorted in the amateur area at least but a flat should still be taken and inspected, if only to avoid the embarassment of that Si II absorption line in the spectrum of a PSN announced in an ATEL as a confirmed type 1a supernova turning out to be a dust bunny instead !
Examples of improving the generation and use of flats
OK so is the concensus that flats should be taken and where the observer has determined that by applying the flat the quality of their data is more representative of the spectrum of the object under study this should be the archived spectra, otherwise a non flat corrected spectrum would be archived and an apropriate flag set in the fits header ?
In the meantime work continues in the amateur spectroscopy community to develop the methods of taking and use of flats to improve the quality of their spectra further,
A few examples which might be useful for people having problems with flat field defects in spectra
I have processed my spectra both with and without flats. I use a LISA and can image from 3800A to 7400 in one image. Getting reasonable SN in the UV without overexposing the red with flat frames is very difficult. I found that the flat frame introduced much more noise than it corrected. Using a standard star to correct the continuum acts as a pseudo flat for spectra but does not correct small variation between pixel response due to the smoothing applied to the response curve.
When I was using my ST10XME camera it had a sinusiodal patern across the entire flat frame and I found that I needed to use a flat to correct this. It was difficult to correct this effect using the response curve. When I changed to my Atik camera and moved the slit a few pixels one direction the sinusiodal pattern has gone.
There are no dust bunnies on the Atik camera so i dont have to correct for these.
So my take on this is that flats may need to be applied but may not always make the image better and may make it worse with spectra. This of course does not apply to photometry as there is no "instrument response" curve being used apart from the flat frame.
Having recently moved from high resolution spectroscopy with the LHIRES, to low resolution with the ALPY, the lack of light in the UV with Halogen flats compared with the IR/visible region is a problem I am also seeing. You certainly need to take a big stack of flats and average them to beat the noise down. I wonder if there is any possible way of filtering a Halogen light source to give a flatter spectrum without introducing small scale variations. I also wonder if EL panel are any better in this region (They have no output at the IR end though). If anyone knows of a better spectroscopic flat light source I would love to hear. I agree that the smooth response of the Sony CCD compared with the Kodak KAF chips is a real advantage when it comes to spectroscopy, particularly when operating slitless where a flat cannot remove this effect. The effect is not limited to Kodak CCD either. I am seeing it in some of the CMOS sensors which are becoming popular in video type cameras that people are using with the Star Analyser.
A measurement of the contribution of the flat field - ALPY 600
This thread prompted me to have a closer look at exactly what the flat field was doing to my spectra. Atrtached is an example using an ALPY spectrograph which I took when testing my data reduction procedure using stars from the Miles catalogue. The flat was taken using the ALPY calibration module (Halogen lamp) 60x 1 sec exposures were combined.
The systematic variations in the flat are clear at around +-1% and are well above the flat noise (The larger feature at the blue end needs investigating. I was unaware of this before did this check. It is clearly systematic though, not noise)
There is some evidence of an increasing noise contribution at the extreme blue end (seen in the "grass" of pixel to pixel variation) but this is still well below the systematic effects which would have been present in the reduced spectrum had a flat not been used.
Based on this I will continue to use flats to correct my ALPY spectra
Robin: nice work - this remind me to do the same comparison with my Alpy 600 recent data.
How many flats are your taking? Personaly, I am usually taking 19 flats with the Alpy reduced after in ISIS. This takes about 5 minutes so it is not that critical from a "time consumption" perspective...
Terry: for CCD dust, best is to preprocess image with a PRNU (Pixel Response Non uniformity) which is a flat of the camera alone. This is done for eShel data reduction, but unfortunatly not for the other spectrograph data reduction in ISIS. PRNU removes dust "donuts" for exemple. Of course, having a clean CCD is even better... :-)
Anyway, interesting discussion but I still believe it is best to correct from flat field but ensuring a high level of signal specially on the blue side of the spectrum.
Robin: nice work - this remind me to do the same comparison with my Alpy 600 recent data.
How many flats are your taking? Personaly, I am usually taking 19 flats with the Alpy reduced after in ISIS. This takes about 5 minutes so it is not that critical from a "time consumption" perspective...
[/quote]
Hi Olivier,
I used 60x 1 second exposures which gave a maximum count ~25000 per exposure with the ATIK 314L+ but this was 2x binned so your 5 min total exposure time seems reasonable.
The counts at 3800A are only around 500/exposure giving a total of ~30k counts or ~10k e- That gives an SNR /pixel of ~100 in the flat but the SNR contribution / bin in the binned spectrum will be perhaps 2-3x better than this which ~ agrees with what I am seeing. This is just for a couple of minutes dedicated to taking flats though so I am not complaining. If I was looking for say an SNR >200 say I would probably spend a bit more time on taking flats.
By the way the strange feature at the blue end was an artifact caused by the low order spline fit I used for this exercise (no sign if it in the flat itself, the instrument response or the reduced spectrum) It was tricky getting a perfect fit down there where the camera response and halogen lamp tails off so steeply. I now appreciate the advantages of removing the broad shape of the flat first as done in ISIS. It makes fitting the instrument response curve that bit easier.
A comparison of spectroscopic flats taken with and without the filter will immediately show if the filter is having any effect in the region you are measuring but yes, if I was planning to measure in this region I would remove any deliberate UV filtering and probably even if I was measuring elsewhere in the spectrum as there could be a risk of any ripples anywhere in the filter passband response causing problems. The one caveat is high power domestic Halogen lamps usually have glass in front of them for safety (protection from the very hot halogen capsule and risk of it shattering for example). I dont know how much this affects the lamp spectrum though, something that could be worth checking if it has not been already. Anyone know?
The Halogen lamp we use for flats on our high res spectrograph is unshielded with no filter. That said it is a bulb, so who knows that the material in the blub does to the spectral distribution.
Generally the lower CCD sensitivity in the blue means that you don't want to do anything to attenuate your flat lamp in that spectral region. Also, if you are going to do flats for spectroscopy the closer you can get your flat to a smooth continuous blackbody spectral distribution, the better.
I have recently purchased an Alpy600. I have been doing photometry for a number of years on exo-planets, asteriods, variable stars and have used sky flats for flat field correction with proper results.
With the Alpy on my scope I attempted to use the same correction procedure but with very bad results. I have followed this discussion, and have an appreciation for the detailed pixel x pixel issues of using a flat. However the problem I am having is that using "regular" sky flats, I cannot even remove vignetting, let alone deal with the other subtlies in this discussion.
Note, my spectra are taken without a slit. It seems to me that the "normal" application of a sky flat should still make a flat field correction of my data with or without the Alpy???
My own
Thanks Ken.
My own experience is if you ask five pros about flat fielding for spectra you will get five different answers. In another forum on this same site there is a very animated discussion about flat fielding for photometry. Many of the points raised there apply here. Particularly because one main concern of flat fielding is that it is difficult to illuminate your field the same way that the sky illuminates it. Sky flats are really the only best solution for imaging. Dome flats are better than nothing but not even close to as good as a sky flat. However for spectroscopy sky flats are not an option because the sky is not a purely continuous spectrum.
So there are two main problems in spectroscopy flat fielding:
The way most professional spectrographs deal with this is to consider #1 a lost cause and do their best with #2. This means you put a bright continuous source with a diffuser in front of the slit and call that a flat. But because #1 is mostly a lost cause, your flat is never really a true flat, meaning it will not reproduce the blaze function or vignetting within the spectrograph in the same way your target does.
The conclusion many professionals reach is that this is hopeless so why flat field spectroscopy at all? Just ignore the step. When you set the contiuum on your target, flat fielding becomes unnecessary because you measure the fundamental parameters of the spectral features relative to the continuum. Especially since we can rely on modern detectors to have a fairly uniform response from pixel to pixel over one spectral resolution unit. When you set your continuum as a separate step you are pretty much wasting you time flatfielding most spectrographs.
Likewise it is pretty hopeless to even consider flatfielding a grating or objective prism for similar yet more complicated reasons.
The Hubble Space Telescope Imaging Spectrograph basically only uses the flat to correct for small amounts of "roll off" (basically vignetting) at the edge of the field. The relative flux calibration across the field is set by observing standard white dwarfs. This is not a practical option for calibrating most spectrographs!
So my own opinion is that an archive should specify that spectra NOT be flatfielded. However it doesn't really matter if they are. There may be a benefit of making it optional (but not mandatory) to include an observation of a standard star for some applications where setting the relative flux scale across the spectrum is desirable.
To make this more confusing, I always take flats when I do spectroscopy. But I guess that is more for my own feeling that I have done everything I can rather than any well defined practical reason.
Thanks Dr. Martin,
Well done. Your response was excellent. After many hours of experimenting I have found that except for very special cases flat fielding spectroscopy while will not hurt, is a waste of time and if not done correctly can mess things up.
Jeff 187283
Hopkins Phoenix Observatory
Two references worth reviewing re Flat field correction:
"Handbook of CCD Astronomy", 2nd Ed, by Steve Howell, p 67, 79-83
"Introduction to Astronomical Spectroscopy", Immo Appenzeller, p 164-166
General comment:
A uniformly illuminated target in front of the aperture, a "flat" spectrum light source i.e. no absorption/emission features - a typical halogen lamp.
Long enough exposure to give a good SNR.
Each setting of the grating in a hi-res spectroscope needs it's own flat field.
Something else to consider: any operation you perform on an image increases the noise in that image.
Let that sink in. If you have two images and you subrtact or divide one from the other the noise in the combined image is greater than the noise in either of the individual images that were combined.
We tolerate this with bias and dark subtraction because the removal of systematic error is preferable to a relatively small addition of random error.
With photometry and flatfielding, the benefit from flattening a field (even with a dome flat) has a greater benefit than the noise introduced to the corrected image by the operation.
With spectroscopy and flat-fielding, I am unconvinced that dividing a flat (with noise) has a benefit that exceeds the addition of the noise to the corrected image. Those of you who were at the workshop on spectroscopy I gave at the SAS meetingn two years ago might remember that I spent some time discussing this ( https://edocs.uis.edu/jmart5/www/SpectroscopyWorkshop/). The tutorial that I posted on simple spectral extraction ( https://edocs.uis.edu/jmart5/www/SpectroscopyWorkshop/ExtractTutorial/index.html) suggested that the best way to use a flatfield (if you really wanted to) was to:
In that manner you at least don't do the harm of adding noise to your spectrum for the limited return you get from doing a flat.
While I have the confidence that many non-pros could learn how to do this on their own, I think it adds a step and some uncertainty that we do not want to introduce into data going into an archive. So that's why I maintain that my strong preference would be an archive of data that is not flat fielded with an option to upload an associated flat in case anyone wanted to dither around with it themselves at a later time.
As far as I can see, the question of flat fields is simple. If you can justify (potentially to a referee or your peers) that you can ignore the defects that flat fields are there to correct then go ahead dont do them. Otherwise it is good idea to take them. The problem is when you discover something interesting but could be real or a flat field defect, if you dont have a flat nobody will believe you. the subject of flat fields is a tough problem for amateurs and there is a lot to consider but it is something you ignore at your peril. We can argue about the potential deleterious effect on SNR etc but that is irrelevant if for example your carefully collected spectra are rendered unuseable by severe fringing which a flat could have removed. (something that happended to one of the posters on this thread)
I take flat fields so at least I can decide whether they are improving or reducing the quality of my data.
Cheers
Robin
I think that about 95% of professional referees are not going to hold up a spectroscopic paper because the author did not flat-field. There is far greater potential systematic error in setting the continuum level, which is truthfully much more dependent on the experience of the end user than any particular calibration they performed.
Robin does make a good point about field defects. Almost all spectrographs have field defects. The Hubble Space Telescope Imaging spectrograph has a fringing effect at long wavelengths. I once presented a poster that included some unidentified "features" that were actually fringing. The poster was up all day at the AAS and only one person said, "I don't think those are real." Turns out they were not. Fortunately, I figured it out before publishing the result in a journal. A good fringe flat would have fixed the issue right away, but those calibrations were considered optional and had been omitted to make room for more exposures in the program the data came from. Lesson learned.
That said, my experience with flats shows that the flats rarely remove the defects completely. And sometimes they make it harder to identify the field defects that still remain. Another excellent professional spectrograph, the Sandiford Echelle on the 82-inch at McDonald observatory has an internal reflection lovinglly called "the picket" that takes out a substantial chunk of entire rows on the CCD. That can seriously screw up one or more orders. The flat does nothing to get rid of it and in fact the flat can make the issue worse by obscuring the picket just enough that it isn't obvious, leading the end-user to not realize that part of spectrum is completely worthless. However, the flat available for inspection, is helpful for identifying where it is and instructing you as to which parts of which orders should be omitted from your analysis. (Also the flat is used to model and subtract scattered light in that spectrograph so it serves an additionall application not commonly used for flats in most spectrographs.)
And I have also seen flats introduce field defects. Since the slit is never illuminated by the flat exactly the same way it is illuminated by the sky they can introduce bumps and shallow dips into the spectrum that were not there before the flat was applied.
We all through experience learn these things about our own telescope and spectrograph rig. But the challenge with an archive is communicating all that knowledge and information to the end-users in as seemless a manner possible.
Considering all this, I think that the best option for gaurding against the mis-interpretation of field defects is to not apply the flat to the target spectrum but have a flat available for inspection. The STIS example demonstrates the flat could sometimes identify a lurking prolem. But at the same time the Sandiford shows, applying it could obscure the problem just enough to make you forget its there (but it still screws up your data). Having it available as a check and a diagnostic is very helpful.
OK, if we agree to hold -for reference, a flat file, available on request....
Where do we go from here????
Yours and Robin's points about defects vs "features" and about the relative importance of setting the continuum are well put. If I might add from my own experience, hardly unqiue, that there are "features" in CCD spectra that have no apparent reason, observational or astrophysical. Having an archived flat available for CCD spectra, but not applied to the archived spectrum, could help decide the issue. The most important requirement of setting the continuum level - "where is the bloody thing?!" - is that the sections of the spectrum used to represent the "true" continuum have as high an SNR as possible. I have had the unfortunate problem of fitting the continuum to low SNR spectra of unusual objects, and it is perhaps the single most frustrating problem to deal with. (With photographic spectra, there are the problem of noise introduced by the emulsion grain, making one want to tear one's hair out!)
For any paper submitted to a professional journal which discusses spectra showing features that may or not be real, the referee should request that the author specifically address the question. I.e., is there a realistic or possible astrophysical effect that is the cause? Is there a new possible or plausible astrophysical reason that the author would like to propose? What might be (or is!) the source of the feature (defect)? Referees may not catch such issues, so the author should be on the alert for anything "weird" and explicitly mention it in the paper. That said, there are often spectral features, particularly in spectra of chemically peculiar stars which are often completely unexpected but whose presence is needed to understand the sources of the peculiarity - star spots, preferential surface abundance enhancements, etc. Some of the observed features are often weak, and having a series of averaged flats handy could be a make-or-break tool! I'd rather have a somewhat low resolution spectrum but with a high SNR than a somewhat higher resolution spectra with a low SNR.
Cheers,
Thom
Guys,
Caution.....
This whole archive opportunity was to provide a "home" for amateur spectroscopy data - not a repository of necessary high quality professional standard data - hopefully with experience a bit of mentoring by the more experienced (and qualified members) they can aspire to reach that level of excellence.
Let's not get carried away by the the holy grail of "ultimate accurate spectral data - suitable for professional journals" etc etc. Novices have to start somewhere, have an opportunity of comparing their results with others and growing their abilities.......
John,
I just noticed you proposed up the thread this procedure for using flats
" the best way to use a flatfield (if you really wanted to) was to:
In that manner you at least don't do the harm of adding noise to your spectrum for the limited return you get from doing a flat. "
If you do this you are throwing away the wrong bit of the flat. The purpose of a spectroscopic flat is to eliminate the short wavelength effects (pixel/pixel sensitivity variation, dust bunnies, optical fringes, small scale variations in camera QE with wavelength(a big problem with Kodak CCD popular with amateurs when used with wide wavelength range spectrographs and a reason in itself to take a flat) etc, etc
The flat can be normalised to remove the large large scale features though if you like as they are eliminated once the instrument response correction is done using a reference star or failing that by normalising to the continuum (assuming you define where the continuum is - only generally possible at high resolution.) A typical technique used to normalise the flat is to use black body curve to approximate to the flat lamp continuum but it is largely a matter of preference provided the normalisation is consistent.
All this means is that there is indeed a risk of introducing some noise back unless you make sure your flat has significantly higher counts than your star spectrum.(eg >10x means you are introducing degrading your SNR by <10%. This is not difficult and is just a matter of good practise. It is not a reason for not taking flats. To me the trade off between eliminating otherwise difficult systematic errors and a marginally lower SNR is clear. I know which one I choose.
If this discussion was in the photometry section everybody would be in favour of flats. Spectroscopy is no different in this respect. All this sort of thing was thrashed out in the amateur spectroscopy community years ago in consultation with the various PI's on a number of Pro-Am projects. The debate has long since moved on from whether flats are important or not to how to make the best flats and how best to apply them.
You mentioned the difficulty of flats for slitless sprectroscopy. This is indeed a complex area still to be fully sorted in the amateur area at least but a flat should still be taken and inspected, if only to avoid the embarassment of that Si II absorption line in the spectrum of a PSN announced in an ATEL as a confirmed type 1a supernova turning out to be a dust bunny instead !
Cheers
Robin
OK so is the concensus that flats should be taken and where the observer has determined that by applying the flat the quality of their data is more representative of the spectrum of the object under study this should be the archived spectra, otherwise a non flat corrected spectrum would be archived and an apropriate flag set in the fits header ?
In the meantime work continues in the amateur spectroscopy community to develop the methods of taking and use of flats to improve the quality of their spectra further,
A few examples which might be useful for people having problems with flat field defects in spectra
Lothar Schannes work on EL panels
http://www.astrospectroscopy.eu/Einsteiger/Flats/flats_e.htm
Benji Mauclaires summary of fringing caused by CCD cover glass in high resolution spectra
http://www.astrosurf.com/aras/fringing/opus/
Christian Buils work on using flats to remove camera response rippples in low resolution spectra
http://www.astrosurf.com/aras/extinction/complement.htm
Cheers
Robin
I have processed my spectra both with and without flats. I use a LISA and can image from 3800A to 7400 in one image. Getting reasonable SN in the UV without overexposing the red with flat frames is very difficult. I found that the flat frame introduced much more noise than it corrected. Using a standard star to correct the continuum acts as a pseudo flat for spectra but does not correct small variation between pixel response due to the smoothing applied to the response curve.
When I was using my ST10XME camera it had a sinusiodal patern across the entire flat frame and I found that I needed to use a flat to correct this. It was difficult to correct this effect using the response curve. When I changed to my Atik camera and moved the slit a few pixels one direction the sinusiodal pattern has gone.
There are no dust bunnies on the Atik camera so i dont have to correct for these.
So my take on this is that flats may need to be applied but may not always make the image better and may make it worse with spectra. This of course does not apply to photometry as there is no "instrument response" curve being used apart from the flat frame.
Terry
Hi Terry,
Having recently moved from high resolution spectroscopy with the LHIRES, to low resolution with the ALPY, the lack of light in the UV with Halogen flats compared with the IR/visible region is a problem I am also seeing. You certainly need to take a big stack of flats and average them to beat the noise down. I wonder if there is any possible way of filtering a Halogen light source to give a flatter spectrum without introducing small scale variations. I also wonder if EL panel are any better in this region (They have no output at the IR end though). If anyone knows of a better spectroscopic flat light source I would love to hear. I agree that the smooth response of the Sony CCD compared with the Kodak KAF chips is a real advantage when it comes to spectroscopy, particularly when operating slitless where a flat cannot remove this effect. The effect is not limited to Kodak CCD either. I am seeing it in some of the CMOS sensors which are becoming popular in video type cameras that people are using with the Star Analyser.
Cheers
Robin
This thread prompted me to have a closer look at exactly what the flat field was doing to my spectra. Atrtached is an example using an ALPY spectrograph which I took when testing my data reduction procedure using stars from the Miles catalogue. The flat was taken using the ALPY calibration module (Halogen lamp) 60x 1 sec exposures were combined.
The systematic variations in the flat are clear at around +-1% and are well above the flat noise (The larger feature at the blue end needs investigating. I was unaware of this before did this check. It is clearly systematic though, not noise)
There is some evidence of an increasing noise contribution at the extreme blue end (seen in the "grass" of pixel to pixel variation) but this is still well below the systematic effects which would have been present in the reduced spectrum had a flat not been used.
Based on this I will continue to use flats to correct my ALPY spectra
Cheers
Robin
Robin: nice work - this remind me to do the same comparison with my Alpy 600 recent data.
How many flats are your taking? Personaly, I am usually taking 19 flats with the Alpy reduced after in ISIS. This takes about 5 minutes so it is not that critical from a "time consumption" perspective...
Terry: for CCD dust, best is to preprocess image with a PRNU (Pixel Response Non uniformity) which is a flat of the camera alone. This is done for eShel data reduction, but unfortunatly not for the other spectrograph data reduction in ISIS. PRNU removes dust "donuts" for exemple. Of course, having a clean CCD is even better... :-)
Anyway, interesting discussion but I still believe it is best to correct from flat field but ensuring a high level of signal specially on the blue side of the spectrum.
Cordialement,
Olivier Thizy
[quote=Olivier Thizy]
Robin: nice work - this remind me to do the same comparison with my Alpy 600 recent data.
How many flats are your taking? Personaly, I am usually taking 19 flats with the Alpy reduced after in ISIS. This takes about 5 minutes so it is not that critical from a "time consumption" perspective...
[/quote]
Hi Olivier,
I used 60x 1 second exposures which gave a maximum count ~25000 per exposure with the ATIK 314L+ but this was 2x binned so your 5 min total exposure time seems reasonable.
The counts at 3800A are only around 500/exposure giving a total of ~30k counts or ~10k e- That gives an SNR /pixel of ~100 in the flat but the SNR contribution / bin in the binned spectrum will be perhaps 2-3x better than this which ~ agrees with what I am seeing. This is just for a couple of minutes dedicated to taking flats though so I am not complaining. If I was looking for say an SNR >200 say I would probably spend a bit more time on taking flats.
By the way the strange feature at the blue end was an artifact caused by the low order spline fit I used for this exercise (no sign if it in the flat itself, the instrument response or the reduced spectrum) It was tricky getting a perfect fit down there where the camera response and halogen lamp tails off so steeply. I now appreciate the advantages of removing the broad shape of the flat first as done in ISIS. It makes fitting the instrument response curve that bit easier.
Cheers
Robin
Duplicate post.
I am quite new to spectroscopy and was thinking of taking flats with a halogen lamp.
The lamp has a UV filtered glass over it.
Should I take this filter off when taking flats?
Thank you.
Jupe
Hi Jupe,
A comparison of spectroscopic flats taken with and without the filter will immediately show if the filter is having any effect in the region you are measuring but yes, if I was planning to measure in this region I would remove any deliberate UV filtering and probably even if I was measuring elsewhere in the spectrum as there could be a risk of any ripples anywhere in the filter passband response causing problems. The one caveat is high power domestic Halogen lamps usually have glass in front of them for safety (protection from the very hot halogen capsule and risk of it shattering for example). I dont know how much this affects the lamp spectrum though, something that could be worth checking if it has not been already. Anyone know?
Cheers
Robin
The Halogen lamp we use for flats on our high res spectrograph is unshielded with no filter. That said it is a bulb, so who knows that the material in the blub does to the spectral distribution.
Generally the lower CCD sensitivity in the blue means that you don't want to do anything to attenuate your flat lamp in that spectral region. Also, if you are going to do flats for spectroscopy the closer you can get your flat to a smooth continuous blackbody spectral distribution, the better.
I have recently purchased an Alpy600. I have been doing photometry for a number of years on exo-planets, asteriods, variable stars and have used sky flats for flat field correction with proper results.
With the Alpy on my scope I attempted to use the same correction procedure but with very bad results. I have followed this discussion, and have an appreciation for the detailed pixel x pixel issues of using a flat. However the problem I am having is that using "regular" sky flats, I cannot even remove vignetting, let alone deal with the other subtlies in this discussion.
Note, my spectra are taken without a slit. It seems to me that the "normal" application of a sky flat should still make a flat field correction of my data with or without the Alpy???
Joe Garlitz
Elgin, Oregon