Refractivity of Air


Although a section in the bibliography is devoted to the refractivity of air, it's buried down near the end of the monster file. Besides, a little more detailed discussion of the dispersion formulae for air seems to be called for.

The main problem is that the refractivity of air is difficult to measure accurately, so that there have been many re-measurements, and several different formulae have been used to represent the dispersion curve of air by different authors. Worse yet, the older formulae, which have long been known to be incorrect, have become fossilized in handbooks, and copied by more recent handbooks from the older ones, so that obsolete and inaccurate formulae are often cited and used.

Note that the refractivity is usually given for 15° C, and refers to dry air, not air with some water vapor in it. The argument is usually vacuum wavenumber, not just the reciprocal of the wavelength in air. Sometimes the formula given refers to air free of CO2 as well as water vapor. Because the refractivity, (n−1), is generally less than 3×10−4, it's quite common to have the formula give either 106 or 108 times the actual refractivity. Bear all these facts in mind when using any refractivity formula. Read all the fine print before using any formula!

Finally, there is more than one “refractive index” of interest, and the right one to use depends on the kind of measurement being made. For angular refraction (which is what I deal with on these Web pages), the appropriate index is the “phase index,” which is the one usually meant if no such distinction is made. However, for radar and laser geodimetry, where an electromagnetic signal is modulated, it is usually the group velocity rather than the phase velocity that is of interest; then the “group index” is required. If observations are made in or near absorption lines, things become still more complex. Fortunately, these niceties are mainly of concern for the geodesists, and will be ignored here.


Early measurements

Although the first crude laboratory measurements of the refractivity of air were made around 1700, and the refractivity was estimated by Newton about this time from astronomical refraction observations, fairly accurate measurements first became available a century later, through the work of Arago and Biot. From the middle of the 19th Century onward, the refractive index of air has been measured repeatedly, as improvements in technique allowed better results to be obtained.

The Barrell & Sears formula (1939)

In its day, the discussion by

H. Barrell, J. E. Sears
The Refraction and Dispersion of Air for the Visible Spectrum
Phil. Trans. Roy. Soc. A, 238, (1939)

was quite influential, though it is now quite obsolete. Its refractivity error approaches a part in 1000 near the atmospheric transmission cutoff just below 300 nm, and is about 1 part in 5000 or 6000 through most of the visible spectrum. (However, the discussion of the physical effects that must be taken into account is still worth reading, even though the numerical values have been superseded.)

Unfortunately, one occasionally still finds papers that use it. (The tip-off that this obsolete formula is being used is that it has the form of a 2-term Cauchy dispersion formula: a constant, a term in λ−2, and a term in λ−4.)

A recent example is the paper by J. Gubler and D. Tytler in PASP 110, 738 (1998). They got the B&S formula from A. T. Sinclair's NAO Technical Note (1982), which doesn't sound too obsolete. But Sinclair got it from the IAG's Bulletin Géodésique (1963), p. 390 — which of course got it from Barrell & Sears (1939). By the time Gubler & Tytler picked it up, the formula was out of date by almost 60 years!

Unfortunately, this incorrect and obsolete formula was also used by Hohenkerk & Sinclair (1985), from which it was copied in the new (1992) edition of the Explanatory Supplement (see the middle of p. 142 for the obsolete formula there); so we can expect to continue to see it crop up again in the future.

The old Edlén formula (1953)

The oldest formula still commonly encountered is the “old Edlén” formula from 1953, which involves 3 terms (a constant and two fractions):
B. Edlén
Dispersion of standard air
J. O. S. A. 43, 339–344 (1953)

This is the formula with a 41 in the denominator of the last term. It's quoted on p. 119 of the 2nd Edition of Allen's Astrophysical Quantities (1963), and on p. 124 of the 3rd Edition (1973). (A modified form of this formula was also included in the IAG's Bulletin Géodésique (1963), p. 390. Recently, the IAG finally got clued in and dumped these obsolete formulae.) Though it's still often cited, it's known to be wrong. Don't use it!

It's inexcusable that it was still given as “the” refractive index of air on p. 262 of the 4th edition of Allen's Astrophysical Quantities in the year 2000 — 36 years after it was repudiated by Edlén himself! (Worse yet, this handbook even misprints it with an egregious typo that reduces the formula to rubbish, on p. 257. They're charging $99 for this?)

The new Edlén formula (1966)

In the 1960s, it became plain that something was wrong with this formula. Edlén re-did the work, and published a new fit. So we got the “new Edlén” formula, of the same form, but with 38.9 instead of 41 in the last denominator:

B. Edlén
The refractive index of air
Metrologia 2, 71–80 (1966)

Owens (1967)

While the “new Edlén” formula still appears in some handbooks, it was already shown to be slightly incorrect the very next year:

J. C. Owens
Optical refractive index of air: dependence on pressure, temperature and composition
Appl. Opt. 6, 51–59 (1967)

Owens, like Barrell & Sears, separated out the contribution from carbon dioxide. He claimed an accuracy of 1 part in 109 for the refractive index, or 3 parts in 106 for the refractivity. Like Barrell & Sears, Owens considered compressibility effects (i.e., deviations from the ideal-gas law); he found them to be about 10−4 at 245 K, increasing rapidly at lower T. (From his figure, it looks like 2×10−4 at 200K).

Owens showed that Edlén (1966) is off by 7×10−8 at −30°C, and by 4×10−7 at 45°C, 100% RH. Unfortunately, the 1966 Edlén formula is still widely quoted and used.

Peck & Reeder (1972)

A few years later,

E. R. Peck, K. Reeder
Dispersion of air
JOSA 62, 958–962 (1972)

made new measurements in the infrared. They showed that Edlén was certainly wrong in the IR, and that one could obtain accuracy as good as the data (except at wavelengths below 0.23 µm) with a 2-term Sellmeier dispersion formula, which involves only 4 parameters instead of 5. This accuracy is a little better than 2 parts in 109 for the refractive index, from 0.23µm in the UV to 1.69µm in the near IR.

Personally, I like to use their simple formula for calculating atmospheric refraction. It's “good enough” for most purposes, as long as you don't need to worry about water vapor, or the middle infrared. (I have a plot of their dispersion curve on another page.)

Jones (1981)

F. E. Jones
The refractivity of air
J. Res. NBS 86, 27–32 (1981)

reconsidered the real-gas effects, and performed a careful error analysis:

The major contributors to the uncertainty in refractivity are the uncertainties in the measurements of temperature and pressure. The magnitude of the uncertainty due to variation in CO2 concentration can approach that of the uncertainties due to the pressure and temperature measurements. Therefore, the CO2 concentration should be treated as a variable and should be observed.

This statement received excellent confirmation a decade later in the 1993 and 1994 papers of Birch and Downs (see below), who measured the effects of varying CO2 concentration in laboratory air.

Unfortunately, Jones seems to have overlooked the Peck – Reeder work. He produced a formula good for the visible region with an error in the refractivity of a few parts in 108.

Water-vapor Problems

Matsumoto (1982)

H. Matsumoto
The refractive index of moist air in the 3-µm region
Metrologia 18, 49–52 (1982)

measured the effects of water vapor, and found considerable effects due to its near-IR absorptions.

Birch & Downs (1988)

Then additional errors due to water vapor, even in the visible, were found by

K. P. Birch, M. J. Downs
The results of a comparison between calculated and measured values of the refractive index of air
J. Phys. E: Sci. Instrum. 21, 694–695 (1988)

and confirmed by

J. Beers, T. Doiron
Verification of revised water vapour correction to the refractive index of air
Metrologia 29, 315–316 (1992)

Ciddor (1996)

Currently, the best refractivity discussion I know of is:

P. E. Ciddor
Refractive index of air: new equations for the visible and near infrared
Appl. Opt. 35, 1566–1573 (1996)

We can regard Ciddor's results as definitive, at least for the present. They have been adopted as the basis of a new standard by the International Association of Geodesy (IAG). Geodetic applications often require the group rather than the phase index; see

P. E. Ciddor, R. J. Hill
Refractive Index of Air. 2. Group Index.
Applied Optics (Lasers, Photonics and Environmental Optics), 38, 1663–1667 (1999)

Bönsch and Potulski

Soon after Ciddor's first paper, a new set of highly accurate measurements at 4 wavelengths in the visible spectrum was published by

G. Bönsch and E. Potulski
Measurement of the refractive index of air and comparison with modified Edlén's formulae
Metrologia 35, 133–139 (1998)
Their work focuses primarily on determining the effects of water vapor and CO2 content, and takes account of a slight adjustment between older practical temperature scales and the ITS-90 scale, which is almost a hundredth of a degree.

They start with the “new” Edlén formula of 1966. The aim is to provide the highest possible accuracy for laboratory conditions, so they consider only a small temperature range about 20°C and a CO2 mixing ratio near 400 ppm, where linear formulae are adequate; their results are not sufficiently general for field use in the actual atmosphere. They give a nice practical summary of their results in an Appendix.

As they essentially confirm the results of previous recent studies — in particular, the water-vapor corrections of Birch & Downs and more recent workers — I would regard their paper as indirectly supporting Ciddor's results. The uncertainty in the refractive index of air is about 1 unit in the 8th decimal place.

For further discussion of these matters, see the Web pages of NIST and the IAG.

Infrared dispersion effects of water vapor

The above discussion refers to the effects of water vapor in the visible spectrum. But water vapor also has strong absorptions in the near infrared, which (by classical dispersion theory) produce considerable effects on the dispersion curve there. The following reference gives an example of the size of these effects out to 25 microns (in spite of the narrower scope suggested by its title):

R. J. Mathar
Calculated refractivity of water vapor and moist air in the atmospheric window at 10 μm
Appl. Opt. 43, 928–932 (2004)

Similar calculations were published by

M. M. Colavita, M. R. Swain, R. L. Akeson, C. D. Koresko, and R. J. Hill
Effects of atmospheric water vapor on infrared interferometry
Pub. Astron. Soc. Pacific 116, 876–885 (2004)

at about the same time; and Mathar subsequently published convenient series expansions for the infrared refractivity of air, given the humidity:

R. J. Mathar
Refractive index of humid air in the infrared: Model fits
J. Optics A: Pure Appl. Opt. 9, 470–476 (2007)

These authors point out that the large decrease in refractivity in the middle IR — in particular, in the important 10μm window — is due to the strong absorptions of the CO2 fundamental near 15μm and the pure-rotational spectrum of water vapor at longer wavelengths (which Colavita et al. repeatedly, and erroneously, call a vibration-rotation band).

CO2 Variations

Speaking of CO2, it's worth mentioning that the steady increase in its abundance during the past century has made measurable changes to the refractivity of air. This is most clearly shown in two more papers by Birch and Downs:

K. P. Birch, M. J. Downs
An updated Edlén equation for the refractive index of air
Metrologia 30, 155–162 (1993)


K. P. Birch, M. J. Downs
Correction to the updated Edlén equation for the refractive index of air
Metrologia 31, 315–316 (1994)

As their titles indicate, these papers were intended to revive the (new) Edlén formula. Although the first paper contains a section headed “Review of Developments Since 1966”, it ignores the works of Owens (1967), Peck & Reeder (1972), and Jones (1981). [Well, there's a reference to the 1967 paper by “Owen” (sic), but only to cite his percentage composition for standard air; there's no mention of his work on the dispersion relation.]

Their 1993 paper discusses a variety of effects: water vapor, compressibility (i.e., real-gas effects in the equation of state), revisions of the temperature scale, and CO2 content. They retained Edlén's dispersion formula, and made measurements only at the He-Ne laser wavelength (633 nm). Unfortunately, as the 1994 paper points out, they put the temperature-scale revision in the wrong way, so that the final results are wrong. (Table 1 of the 1994 paper is a corrected version of Table 4 in the 1993 paper.)

However, the main value of this work was to show convincingly that the CO2 content of laboratory air is usually appreciably higher than in the free atmosphere — I suppose because laboratories are full of people continually exhaling the gas — and that the effect of this on the refractivity is plainly measurable. In this sense, I take their work as experimental confirmation of the work of Jones (1981).

A proper treatment of the CO2 problem would take account of the fact that the increase in its abundance is at the expense of molecular oxygen. The dispersion curves of both gases should be treated accurately, instead of simply scaling the refractivity of air, as Birch & Downs do. In any case, their results are by now completely superseded by Ciddor's.

Thanks to William M. Robertson for providing copies of these Birch & Downs papers to me.

Required Accuracy

It's worth bearing in mind that the refractive index of air at NTP changes by just about 10−6 per degree; that's a change of about 1 part in 300 in the refractivity. As the refraction at the horizon is on the order of 2000 arc sec, we can expect that temperature changes of 0.01° may be significant in accurate positional measurements. Such tiny temperature changes in the temperature of air are exceedingly difficult to measure, even in the laboratory, and are impossible for the observing astronomer to measure.

Similarly, a change of 0.1 mm of mercury changes the density and hence the refractivity of air by about a part in 104, so that still smaller pressure changes might be significant. This is also the level at which deviations from the ideal gas law become appreciable.

Stone says that, for accurate positional astrometry, errors should be kept below 0.05 arc seconds at zenith distances up to 70°, where the sea-level refraction is about 147 arc seconds. This refraction is nearly 3000 times the allowed error.

For most purposes, we can neglect effects smaller than 1 or 2 seconds of arc at the horizon, corresponding to relative errors in density of 1 part in 1000, or perhaps 1 part in 2000. It appears that an error of 1 part in a few thousand is small enough for astronomers.

This means an accuracy of 1 part in 3000 for the refractivity, or 1 part in 107 for the refractive index, is adequate for most astronomical purposes. That would make the Edlén formulae acceptable. But, as the more recent formulae — in particular, the Peck – Reeder formula — are more reliable, as well as less expensive to calculate, I think one of them should be used.

Note that 1 part in 3000 requires measuring temperatures to 0.1°C or better accuracy (not just precision); pressures must be accurate to about 1/3 of a millibar; and the water-vapor mixing ratio should be known to a relative accuracy of about 10%.

For astronomical refraction calculations in the visible spectrum, the Peck – Reeder formula seems a good compromise between accuracy and economy of calculation. I recommend it.

For other purposes, such as geodesy, more accuracy may well be required. Then one should adopt Ciddor's results in the visible, and take account of the large dispersive effects of water vapor and CO2 in the infrared.


Copyright © 2003 – 2007, 2010, 2011 Andrew T. Young

Back to the ...

refraction-calculation page

dispersion-curve page

or the GF home page

or the Overview page