I'd like to start a discussion about the best practices in archiving spectra. There are a number of separate groups that have taken different approaches. I'm interested in why we're doing that and getting a discussion going about if there is any one standard we can settle on that works best for the widest number of applications available. To kick it off, I can think of three factors that there are a wide variety of practices on: wavelength calibration, flat-fielding, and 1D vs 2D spectra.
I think that it might work best to take each of these in one forum thread at a time. Let's start with wavelength calibration. If anyone is anxious to start a discussion about one of the other sub topics, be my guest, but I suggest that it might work best to do that on separate threads.
Lets acknowledge that spectra are in many ways more complicated than photometry. There is not really even a professionally agreed on standard. And this makes it harder for Pro-Am collaboration and crowd-sourcing spectroscopy.
As a professional who does spectroscopy, I know that the most useful way to archive spectra can depend on what you want to do with them. If you are doing identification of a transient, then you might not even need a wavelength calibration. You know what a Type II vs a Type Ia supernova spectrum looks like as long as you generally know this is optical or this is IR or UV. But if you want to do radial velocity work or catalog how an H-alpha emission profile evolves overtime, then you want a good wavelength calibration. A good relative (pixel to pixel) wavelength calibration is also needed to measure equivalent widths.
We should also consider that there are different ways to wavelength calibrate spectra. An end user that is doing radial velocity work will probably do an artificial wavelength standard like an arc lamp for good relative calibration but then on top of it observe radial velocity standards to improve their absolute calibration. Whereas someone doing abundance analysis is satisfied with a good relative wavelength scale from the arc lamps but not as concerned about a very well defined absolute scale.
We would all do wavelength calibration all the time to the highest degree possible but we don't because it takes extra steps and it can be hard with some setups. As a result we do it if it is easy or we really need it and otherwise we ignore it. Different archives have approached this different ways. Here are the approaches I can think of:
- The forget about it approach - ignore wavelength calibration all together. The positive is that it is a uniform policy for all spectra that heavily favors those submitting data. The drawback is that the data isn't really useful to people who need wavelength calibration and that can be really challenging for the end-user.
- The let us know if you've done it approach - In this case an archive accepts either type of data (wavelength calibrated or not). I is up to the user to flag their data appropriately. This is a very inclusive approach that makes it easy for people to submit data but it is annoying if they flag their data one way and it is actually the other way. The drawback is that there is not a uniform standard. This can be frustrating to end-users and make the archive harder to mine.
- The you MUST wavelength calibrate approach- Wavelength calibration is required. The negative is that this discourages people from submitting to the archive because it requires more work from those submitting spectra. The positive can be that it raises the quality of the data by imposing a standard that everyone has to meet. End-users love this because the spectra are ready to use with less interpretation.
- Wavecal data stored separately but linked to the observations- This is kind of related to option 2. Often wavelength calibrations are done with arc lamps or observing separate standards. In this model the calibrations are uploaded separately for end users to apply if they want to apply them or ignore them if they want to ignore them. This has the advantages of #2 for the observers. It adds a little more work for the end-users than #3. But it puts a larger burden on the archive, which is now storing in some cases twice as much data and it has to effectively link the observations to the appropriate calibrations or it becomes harder for the end-users to navigate.
I'll say that option 4 is kind of what I do for myself for my own data. But as an archive end-user we all love #3 where the data falls in your lap with no additional work to do. And the observer in each of us has a guilty affinity for the idea of just dumping our data with little additional effort on our part.
In each of these we see a balancing (or conflict depending on how you look at it) of the expectations and requirements of end-users versus observers providing spectra. So many of you on this forum have experience running archives, or as an end-user, observer, or both. What do you think is the best balance and practice for a spectra archive when it comes to wavelength calibration? What have you personally found the best compromise for an archive to make? And I'd also be interested in hearing why.