Typical Values for 2nd Order Extinction coefficients
As part of calculating new transformation coefficients I have calculated second order extinction coefficients using data from 5 nights late last month and the beginning of this month for 5 pairs of stars in standard cluster NGC 7790. Can anyone advise how the 2nd order coefficients below compare to typical 2nd order extinction values near sea level?
Brian D Warner’s book Light curve Photometry and Analysis gives -0.04 as a typical value for k”_b-bv My results were a bit larger (in abosolute terms) at -0.05. Here are the average results from the 5 pairs over 5 nights with the standard errors in parentheses:
k"_b-bv: -0.050(0.027); k"_v-bv: -0.006(0.009); k"_bv:-0.044(0.025);
k"_v-vr: -0.005(0.017); k"_r-vr: 0.016(0.016); k"vr: -0.021(0.020);
k"_v-vi: -0.002(0.009); k"_i-vi: 0.020(0.011); k"_vi: -0.015(0.010);
k"_r-ri: 0.022(0.017); k"_i-ri: 0.041(0.025); k"_ri: -0.019(0.020).
What concerns me is that 2nd order extinction is usually considered negligible for other than b and b-v. However, the results above show that only k”_v-bv , k”_v-vr and k”_v-vi are approximately an order of magnitude smaller than k“_b-bv and k”_bv. The others, although smaller than k“_b-bv and k”_bv, are still the same order of magnitude. Further. Δ(v-i) is about the same size as Δ(b-v) while Δ(v-r) and Δ(r-i) are in the range of one-half or larger of Δ(b-v). As a result, I would expect effect of the differences in color indices other than b-v will result in corrections to r and I and color indices including those bands on the order of around one-quarter as large and in the case of v-i around one-half as large as the correction to b and b-v. With a B-V color difference between target and comp of 0.5 magnitudes, which would not be uncommon, at airmass 2 my coefficients results in a 0.05 magnitude correction in b, and 0.0056 correction in v both using b-v, -0.0081 correction to r using v-r, -0.02 correction to i using v-i, -0.011 correction to r using r-i and a -0.020 correction to i using r-i. While these are all smaller than the correction to b and b-v (since the correction to v using b-v is small), most fall in the range of 0.01 to 0.025 magnitudes which could still be significant for precise photometry and they would be systematic rather than random. In making these calculations I assumed that the difference in color indices between target and comp for v-r and r-i were half the b-v difference and that the differnece in v-i was the same as in b-v
That lead me to question whether this data is suspiciously far from the norm near sea level.
The star pairs used are the following
000-BLJ-964 / 000-BLJ-965
000-BLJ-966 / 000-BLJ-968
000-BLJ-972 / 000-BLJ-970
000-BLJ-966 / 000-BLJ-970
000-BLJ-972 / 000-BLJ-968
The last two pairs are “cross combinations” of the four stars in the second and third pairs.
Nightly airmass ranges were
11/20/2016: 1.2548 - 1.878
11/21/2016: 1.3281 - 3.0777
11/24/2016: 1.1746 - 1.9836
11/29/2016: 1.1894 - 2.3491
12/01/2016: 1.2342 - 1.9642
Runs terminated before reaching at least airmass 2 were terminated because conditions - seeing or transparency - started to deteriorate.
Brad Walter, WBY
Please note that I had to save the spreadsheet in .xls format rather than .xlsx to upload. I set options so that formulas do not re-calculate on opening. Values may change if you recalculate since not all .xlsx functions, particulalry statistical ones, are compatible with .xls functions.
Brad:
I measure k" b_bv for my single-channel systems (I use two photometers, each with their own filters). One system gives a value of -0.043, the other, -0.031. Henden and Kaitchuck claimed that values range from about -0.02...-0.04, while Hall and Genet, in their book, advised assuming a value of -0.03 (I don't have the books with me here - the H&K numbers I am remembering might be for bv_bv).
Since k" is supposed to be stable over long periods of time, I have always assumed that measurments of the value are characterizations not of anything atmospheric, but of your optical system. While airmass is a multiplicative factor in the second order correction, the actual coefficient is a reflection of your system's response curve (I think).
I don't perform nearly as elaborate an analysis of my calibrations as do you, but I do look at a simple graph of delta b vs X*delta(b-v) to see if it's sensible. I didn't see a quick way to make such graphs from your spreadsheet.
Tom
Tom,
Thanks for responding.
Second order extinction is caused by atmospheric conditions rather than your equipment. If it weren't caused by atmospheric conditions the airmass factor, X, would not be included in the independent variable. sensitivity of your equipment is not affected by airmass. Transformation is the process that removes differences between your optics and sensors and those of the standard system.
From Bob's 2005 paper and data I have gathered, it seems to me there is a large relative (percentage) change in k" values from day to day, But even though they vary by fairly large percentages over time, second order extinction is so small compared to first order that, that the effect of the changing values is usually down in the single digit millimag level from those obtained using a good average 2nd order extinction coefficient value. There may be occasions when they change more drastically, such as a volcanic eruption or extensive burning off of fields in northern Mexico when the wind is blowing the wrong way. If I remember correctly from some papers I read on the topic, a big factor affecting wavelength dependent extinction is the concentration of aerosols of different sizes.
Is is actually very easy to add graphs to this spreadsheet. If you look at the sheets labeled with the pair IDs 02-03, 05-06, etc. in the spreadsheet I attached containing the observations for 11/24/2016, you will see two-column sets of data in the "boxed" area at the top of the page for that pair. Each of these sets contains the Y values(left hand column) and the X values (right hand column) for one particular 2nd order extinction coefficient for that pair using that night's data. for example, using the left most two-column set of data for each pair, as indicated in green shading, I constructed a plot of Δb vs. Δ(b-v)X with each pair as a separate series. The data on these pages was pulled from the Data page using indirect addressing functions. As long as you maintain the structure of the data you only have to create these pages once and then just copy and paste into data for other dates. See the attached spread sheet with a chart added. Just remove the _.txt from the end of the file name and it will once again be a .xlsx spreadsheet. you have to do this to fool Drupal. It's stupid but, necessary for now, at least.
The bottom of the "boxed area are LINEST function calculations using this same data. LINEST not only give the k" values that you get from a trend line on a chart but all of the statistics associated with the linear regression. Of particular interest are SE_k" (standard error of the slope) and SE_y (1 sigma vertical distance of points from the regression line to aid in identifying outliers), the r squared measure of fit and the F statistic of the regression.With 3 degrees of freedom you need an F statistic value of 10.1 to have less than 5% chance of having that F value or larger. That means most of the 2nd order extinction coefficients for most of the pairs are not statistically different from zero. There are some notable exceptions that you can pick out from scanning the F statistics on each of linear regressions on the pages with the pair ID tabs. The most common exceptions are for k"_b-bv and K"_bv, but there are others. The reason I do the LINEST fits are to get these statistics that are not available from a linear trend line in an Excel chart.
Brad:
Of course, all extinction is caused by the atmosphere. But the amount of second-order extinction you see in your system will depend upon its response curve: the greater the sensitivity to blue light, the more 2nd order comes into play.
I have long been puzzled by claims that 2nd order does not change much night-to-night and that one may even assume a default value. I am not a physicist, but it makes no sense to me that the amount of 2nd order extinction I experience is (primarily) driven by atmospheric conditions.
Consider the determination of k'' in B band: we plot delta_b versus airmass * delta(b-v), with delta(b-v) being simply delta_b - delta_v. We are, thus, always dealing with magnitude deltas, not magnitudes. If the atmosphere is attentuating B band in some way, it will do so the same for both the red and blue stars, and the effect will drop out in the delta. The same reasoning applies to V band.
There will, however, be more first-order extinction at the blue end of B band than at the red end. Isn't this really what second-order extinction is? The greater the sensitivity of the system to blue light, the greater will be the effect (and the less blue light a star emits, the less its B magnitude will be dimmed). In the absence of a different physical explanation for 2nd order (I have yet to see one), I find it hard to believe the the variations I measure for this effect are due to changing atmospherics.
I welcome any "illumination" you can provide.
Tom
Hi, Brad:
Back in 2005 I did a study of 2nd order extinction at my backyard observatory (183 m ASL, near the coast in southern California). You can get it (free) on the SAS website: http://socastrosci.org/images/SAS_2005_Proceedings.pdf. Go to page 111 for my report ...
Your results seem to be fairly consistent with mine. I was also intrigued by the spread of k" data reported at professional observatories (summarized in the report).
Cheers,
Bob Buchheim
Thanks, Bob. That was a big help.
Attached is the summary of my 5 nights of data. I used the block of calculations in the middle, highlighted in green. This spreadsheet calculate the averages for the 5 nights from the nightly averages of the 5 pairs of stars, i.e. the "averages of nightly averages" The values of the nightly average were copied as values into this "summary spreadsheet from the spreadsheets for the individual nights like the one for 11/24/2016 attached to my first e-mail. As part of the analysis, I calculated the empirical standard deviation of the 5 nightly averages and the "Calculated standard error" using the normal error propagation formula for an average. It is worth noting that in most cases the empirically determined (STDEV.S) standard deviation of the average k" value is significantly greater than the calculated standard error of the average.
As for your data there appears to be significant variation in the k" values, on a relative basis, from night to night. I think this makes sense since aerosol and water vapor concentration can vary considerably from night to night and season to season. That is particularly true in my area where coastal airmasses collide with drier air from the hill country, burn piles are still common from land clearing, smoke from burning off the remnant's of the previous year's cotton crop in northern Mexico, and "the smell of money" from oil fields are all highly variable on a daily and seasonal basis. This may be one of those cases in which the standard deviation is typically greater than the mean and calculating an average from a bunch of data may not give a better estimate of the True value applicable to a given night, but may still give you an average value that is good enough to make the remaining variable component of the error small compared to other errors.
I excluded data east of the meridian for the night of 11/20 because they had bad photometry for the 000-BLJ-972 star which gave nonsensical k" b-bv values in the range of +0.08 to +0.09. Also these values were greatly at odds with pairs not including this star and observations of this pair over similar airmass west of the meridian on the same night. I was lazy and excluded observations of all pairs east of the meridian rather than just those relating to these two pairs, because it was faster and easier to do.
Bob, I noticed that you calculated k"_r-bv rather than k"_r-vr. Was there a particular reason that you used b-v as the color index for the abscissa when determining k"_r instead of v-r? Of course you can, but since normally you would use v-r as the the CI for transforming r in b-v-r three-color photometry, I would think that you would use v-r as the color index for extinction as well. This is just a matter of curiosity outside the scope of the paper.
Thanks again for pointing me toward your paper. It gave me much more confidence in my results.
Brad Walter, WBY
ps. As before our implementation of Drupal will not let me upload a .xlsx file. this file contains functions that are not compatible with an .xls file (sTDEV.S, for example). Therefore rather than convert it to a .xls file I have fooled Drupal by tacking on "_.txt" to the end of the file name. After you download the file, delete the "_.txt" so that the file name ends in .xlsx and Excel will recognize it for the .xlsx file it really is.
This is something that needs to be fixed in our Drupal implementation. You also can't upload a simple .csv file without adding _.txt to the end of the file name. These upload limitations are a real PITA, particularly when you are trying to run a CHOICE course. This problem needs to be fixed. It has existed since the last major Drupal update over a year ago. If the reason for the limitations is security, what good is it doing if I can bypass it simply by adding _.txt on the end of any file, including a .exe file?
Brad:
I plotted k''_i_vi for two data sets (see attached). There is so much scatter that I would not have considered using either of them for my own calibrations (I always make graphs). I don't see how averaging these kinds of results can lead anywhere.
There has been a lot of recent discussion about secondary extinction coefficients and I thought I had better chime in the couple of comments while I have a minute. I have been buried with other tasks that need to get finished and have not had a lot of time to thoroughly read this discussion but I have a couple of comments that will address some of the points that have been made previously. After the first of the year, I may be able to get back to this in more detail.
Secondary extinction is in effect that leads to changes in extinction values that are dependent upon the spectral type of a star. Most stellar type objects radiate energy in a way that is generally fit by a black body curve. For most of our photometric filters, there is little difference from one star to another as far as the way the spectral energy distribution (SED) passes through a given filter. Even though the SED scales differently for different stars, in most cases there are no major spectral features that alter the overall shape of the curve for the light received from the star. This is especially true of most filters that have bandpasses in the range of the V filter or an even longer wavelength. A word of caution, all bets are off if you are not observing stars. For example, red shifted galaxies can have all kinds of major spectral features moved into band passes that would not ordinarily measure any unusual effects. Anyway, this overall consistency in the SED for normal stellar objects is the reason why secondary extinction coefficients are generally insignificant in filters such as VRI. Why is it that U and B filters are different? This is because the Balmer jump is a major feature in the SED of a star. The Johnson U filter is a real problem for this effect because the Balmer jump dominates the entire bandpass. As you move through the temperature sequence of stars, the Balmer jump gets stronger as you move from O type stars through B type stars and hits a maximum in the early A type stars. As you move from the early A type stars through the F, G, K, and M stars, the Balmer jump becomes weaker and weaker. The effect of the Balmer jump on the U filter is that it changes the effective wavelength of the filter. If the effective wavelength is moved toward the blue side of the filter, the resulting extinction coefficient will be larger. If the effective wavelength is moved toward the red side of the filter, the resulting extinction coefficient will be smaller. When the Balmer jump is at a maximum, the effect on the U filter is that much of the light on the blue side of the filter is absorbed in the stellar atmosphere and this effectively moves the central wavelength of the U filter toward the red. Thus, you would measure an extinction coefficient in the U filter that would be smaller for an A2 star than for a K5 star. This change of extinction coefficient with temperature is what is usually referred to as a seconary extinction coefficient. Because the blue edge of the B filter is mildly affected by the Balmer jump, the B filter also suffers from a secondary extinction effect but not to as large a degree as the U filter. This secondary extinction coefficient is measured by comparing extinction measured for hot stars compared to cool stars. This is the reason why extinction pairs of say A stars and K stars were often routinely measured by photoelectric photometrists back in the days when UBV was pretty much the only game in town. These various dependencies of magnitude with spectral type have always made properly reducing U band photometry a real chore. It is also one reason why there is so much variation in U band reductions even to this day. It is just a real pain to get great results.
I don't believe the second-order coefficients for VRI filters that have been mentioned previously are statistically significant as was pointed out by the graphs that Tom posted. A general rule of thumb is that if the standard error for the determination of the coefficient is comparable in size to the coefficient itself, that coefficient is really not statistically different than zero. This is a very common error (or misunderstanding) that is made even in professional journal articles. Oftentimes you will see a data set that is fit with a high order polynomial with lots of bumps and ripples in the resulting fit. Examination often shows that the higher-order terms are not actually different from zero and that the data are fit just as well with a linear or quadratic fit. Remember that you can perfectly fit 100 data points with a 99th order polynomial. Of course, if you do the experiment again and gather another 100 data points, you'll likely not be able to use the same fit. For it to be science, the fit needs to be reproducable for different observers. Transforming doesn't really help if I can't match your particular standard system over and over.
Mike Joner
I just noticed that the spreadsheet containing the summary of 5 nights' data didn't load. Let me try again. As before, just delete the "_.txt" added to the end of the file name and you will have a normal, current generation Excel file.
I took a long time responding because this discussion caused me to re-examine a bunch of things I had taken as givens. As part of that process and to address some of the issues raised so far, I went through several pages of quantitative analysis and thought process which are attached. Also attached are a few supporting documents to the analysis.
At the very end a question arose concerning something I had not previously given a second thought. Perhaps Mike or someone else knowledgeable about quantum mechanics could resolve this question.
If anyone finds errors in the attached analysis please point them out. This is a thought provoking discussion I have found very helpful.
the .doc file attached containing the analysis was converted from a docx so that I could upload it to the forum. If anything doesn't display properly let me know and I will. upload the .docx file with _.txt added to the file name.
Brad Walter
I agree that this is a very interesting discussion! All write-ups of second order that I have seen gloss over the details and it will be quite revealing if we can sort it out. I, also, am having difficulty with the "effective color" or "effective wavelength" aspect.
I realized that my statement about measurements of second order not being atmosphere-dependent is incompatible with my claim that second order is caused by heightened primary extinction at the blue end of a passband. But how, then, to explain claims that k''b stays fairly constant over time and needs only occasional calibration? Are the texts that say this assuming consistent year-round extinction at your observing site?
There is another angle here, which Jim Kay and I noticed doing PEP calibrations: k"b and eB are at least partially interlocked. The process of calibrating eB must take k"b into account - variation in the former changes the latter. When second-order extinction and transformation are both applied, the combined variations in the two parameters tend to cancel out, at least at modest airmasses. Thus, I can assume default values of k"b over a fairly wide range, and when the associated eB transform is applied along with the second order extinction correction, there will be very little difference in the reduced magnitude.
Tom wrote:
"But how, then, to explain claims that k''b stays fairly constant over time and needs only occasional calibration? Are the texts that say this assuming consistent year-round extinction at your observing site?"
I think two things may come into play. First is that if you are at a professional observatory above 5,000 ft the changes in extinction are less. Changes in aerosols are definitely less in normal situations. Also, the changes in Rayleigh scattering will be less since the length of integration is shorter in
Fλ/F0λ = e^(-∫αλ(s)*ds)
Also, the effect of Rayleigh scattering in magnitudes has been determined to vary exponentially with altitude (above sea level not altitude angle). See attached paper that was referenced in the Sky & Telescope article. Therefore, the portion of extinction that is most sensitive to frequency (4.08 power for flux, 4.08 ratio in magnitudes), decreases strongly with altitude. So those of us who aren't at high altititudes are much more affected by small changes in Rayleigh scattering. It seems to me that Rayleigh scattering would be affected by two things. Molecular composition of the atmosphere, which changes most at low altitudes and even then only changes only to a very small degree, and atmospheric density, which probably changes more near sea level due to greater changes in barometric pressure but the change is small compared to the change in aerosols which have a much smaller (about 1x rather than 4x) magnitucde variation with frequency.As a result the change most of us experience in extinction is greater than major observatories, second order extinction effects are about an order of magnitude smaller than first order and the change in effective color of the "atmospheric filter over time is smaller because the changes in Rayleigh scattering are smaller than the changes in the less frequency dependent (in the visual range at least) aerosol scattering. So if the effect of extinction is 0.16 magnitudes and second order extinction is an average of 0.02 but varies by say 20% The effect whether it is 0.016 or 0.022, rather than 0.02 is in the single digit millimag range even at airmass 2 with a full magnitude difference in B-V between comp and target. We have much bigger errors to worry about than that. If I am using empirically determined error estimates (i.e. the standard deviation of several observations of the check - or the target if its time frame of measurable variability is much longer than the time span of observations - that have been averaged into a single submitted observation) I almost never achieve an error of 0.01 or less.
Ton Wrote:
"There is another angle here, which Jim Kay and I noticed doing PEP calibrations: k"b and eB are at least partially interlocked. The process of calibrating eB must take k"b into account - variation in the former changes the latter. When second-order extinction and transformation are both applied, the combined variations in the two parameters tend to cancel out, at least at modest airmasses. Thus, I can assume default values of k"b over a fairly wide range, and when the associated eB transform is applied along with the second order extinction correction, there will be very little difference in the reduced magnitude."
I am not 100% certain how to interpret this information. I am not sure to what "range" refers. It could be arimass or it could be difference in color index. I think my experience is similar to yours, but let me write it a bit differently.
I have calculated eb with and without second order extinction correction applied first. The values of eb are significantly different. If I apply eb derived with no 2nd order extinction correction to data that is not 2nd order corrected but is taken at an airmass very close to the airmass of observations used to calculate eb (usually as close to 1.0 as possible) and the color difference is not very large, I get results very close to those applying the "2nd order extinction corrected" eb applied to data that is also 2nd order extinction corrected whether or not that data is at the same uncorrected airmass as the uncorrected airmass of the data use in the derivation . If, however, I apply eb derived without extinction correction to data taken at very different airmass (say near 2.0) that is not extinction corrected. I get a very different result than if I apply a "2nd order corrected eb to 2nd order corrected data. I also get a significantly different result than if I apply the uncorrected eb to the that same data that has been 2nd order corrected because the data has been corrected to airmass zero, but eb was derived at airmass 1.0. So if you want to apply transform across airmasses you should first apply 2nd order extinction to your standard stars when deriving eb and to the data tto be transformed.
So to summarize, if you want to apply transformations to data taken at different airmass than the airmass of the data used to derive the transformation you should apply extinction correction to the data used to derive the transformation and the data to be transformed. If your transforms were derived from extinction corrected data you must apply them to extinction corrected data. If your transformations were not derived using extinction corrected data both the data used to create the transforms and the data to be transformed should be at airmasses close to 1.0. the difference in the transformed result compared to transforms and data that have 2nd order correction applied will be quite small for "normal" stars because not only is the second order extinction correction small (≤ -0.015 for k" = -0.03, Δ(b-v) ≤ 0.5 and airmass ≈ 1.0) but eb is changed from the extinction corrected one to include some of the 2nd order extinction. It doesn't exactly compensate because the effective Δ(b-v) used to create the transforms isn't the same as between the target and comp being measured.
I actually did this analysis a couple of years ago using transformations, 2nd order extinctions and standard stars in M67. I would offer the spreadsheets but I think they would be incomprehensible to anyone but me. They are barely comprehensible to me looking at them now. I was not only looking at the relationship between 2nd order extinction and transformation but a whole bunch of other simplifying assumptions such as stars in the same field of view having the same airmass. They are extremely large and complicated spreadsheets covering all transforms and extinction coefficients over a range of airmasses. They give me such a headache the thought of looking at them makes me cringe.
Brad Walter, WBY
ps. Drupal won't let me upload a pdf of the paper at the moment, but here is a link.
http://www.icq.eps.harvard.edu/ICQExtinct.html
What I did was to experiment with different values of k''b, applying them to both the establishment of eB and to the reduction of my photometry (the reductions included adjustments for both transformation and second-order extinction).