Although a section in the bibliography is devoted to the refractivity of air, it's buried down near the end of the monster file. Besides, a little more detailed discussion of the dispersion formulae for air seems to be called for.

The main problem is that the refractivity of air is difficult to measure accurately, so that there have been many re-measurements, and several different formulae have been used to represent the dispersion curve of air by different authors. Worse yet, the older formulae, which have long been known to be incorrect, have become fossilized in handbooks, and copied by more recent handbooks from the older ones, so that obsolete and inaccurate formulae are often cited and used.

Note that the refractivity is *usually* given for 15° C, and
refers to **dry** air, not air with some water vapor
in it.
The argument is usually vacuum wavenumber, not just the reciprocal of the
wavelength in air.
Sometimes the formula given refers to air free of CO_{2} as
well as water vapor.
Because the refractivity, (n−1), is generally less than
3×10^{−4}, it's quite common to have the formula give either
10^{6} or 10^{8} times the actual refractivity.
Bear **all** these facts in mind when using any refractivity formula.
Read all the fine print before using any formula!

Finally, there is more than one “refractive index” of interest, and the
right one to use depends on the kind of measurement being made. For
angular refraction (which is what I deal with on these Web pages),
the appropriate index is the “phase index,” which is the one usually
meant if no such distinction is made. However, for radar and
laser geodimetry, where an electromagnetic signal is modulated, it is
usually the *group* velocity rather than the phase velocity that is
of interest; then the “group index” is required. If
observations are made in or near absorption lines, things become still
more complex. Fortunately, these niceties are mainly of concern for the
geodesists, and will be ignored here.

In its day, the discussion by

H. Barrell, J. E. Sears

The Refraction and Dispersion of Air for the Visible Spectrum

Phil. Trans. Roy. Soc. A,238, (1939)

was quite influential, though it is now quite obsolete. Its refractivity error approaches a part in 1000 near the atmospheric transmission cutoff just below 300 nm, and is about 1 part in 5000 or 6000 through most of the visible spectrum. (However, the discussion of the physical effects that must be taken into account is still worth reading, even though the numerical values have been superseded.)

Unfortunately,
one occasionally still finds papers that use it. (The tip-off that this
obsolete formula is being used is that it has the form of a 2-term Cauchy
dispersion formula: a constant, a term in λ^{−2},
and a term in λ^{−4}.)

A recent example is the paper by J. Gubler and D. Tytler in PASP **110**,
738 (1998). They got the B&S formula from A. T. Sinclair's
**NAO Technical Note**
(1982), which doesn't sound too obsolete. But Sinclair got it from
the IAG's
*Bulletin Géodésique* (1963), p. 390
— which of course got it from Barrell & Sears (1939).
By the time Gubler & Tytler picked it up, the formula was
out of date by almost 60 years!

Unfortunately, this incorrect and obsolete formula was also used by
**Hohenkerk & Sinclair (1985)**, from which it was
copied in the *new* (1992) edition of the
**Explanatory
Supplement**
(see the middle of p. 142 for the obsolete formula there); so we can
expect to continue to see it crop up again in the future.

B. Edlén

Dispersion of standard air

J. O. S. A.43, 339–344 (1953)

This is the formula with a 41 in the denominator of the last term.
It's quoted on p. 119 of the 2nd Edition of Allen's
*Astrophysical Quantities* (1963),
and on p. 124 of the 3rd Edition (1973).
(A modified form of this formula was also included in the IAG's
*Bulletin Géodésique* (1963), p. 390.
Recently, the IAG finally got clued in and dumped these obsolete
formulae.)
Though it's still often cited, it's known to be wrong.
**Don't use it!**

It's inexcusable that it was still given as “the”
refractive index of air on p. 262 of the 4th edition of *Allen's
Astrophysical Quantities* in the year 2000 — 36 years after it
was repudiated by Edlén himself! (Worse yet, this handbook even
misprints it with an egregious typo that reduces the formula to rubbish,
on p. 257. They're charging $99 for *this*?)

B. Edlén

The refractive index of air

Metrologia2, 71–80 (1966)

J. C. Owens

Optical refractive index of air: dependence on pressure, temperature and composition

Appl. Opt.6, 51–59 (1967)

Owens, like Barrell & Sears,
separated out the contribution from carbon dioxide. He claimed an
accuracy of 1 part in 10^{9} for the refractive index, or 3 parts
in 10^{6} for the refractivity.
Like Barrell & Sears, Owens considered compressibility effects
(i.e., deviations from the ideal-gas law); he found them to be
about 10^{−4} at 245 K, increasing rapidly at lower T.
(From his figure, it looks like 2×10^{−4} at 200K).

Owens showed that
Edlén (1966) is off by 7×10^{−8} at −30°C,
and by 4×10^{−7} at 45°C, 100% RH.
Unfortunately, the 1966 Edlén formula is still widely quoted and
used.

E. R. Peck, K. Reeder

Dispersion of air

JOSA62, 958–962 (1972)

made new measurements in the infrared. They showed that Edlén was
certainly wrong in the IR, and that
one could obtain accuracy as good as the data
(except at wavelengths below 0.23 µm) with a 2-term Sellmeier
dispersion formula, which involves only 4 parameters instead of 5.
This accuracy is a little better than 2 parts in 10^{9} for the
refractive index, from 0.23µm in the UV to 1.69µm in the near
IR.

Personally, I like to use their simple formula for calculating atmospheric refraction. It's “good enough” for most purposes, as long as you don't need to worry about water vapor, or the middle infrared. (I have a plot of their dispersion curve on another page.)

F. E. Jones

The refractivity of air

J. Res. NBS86, 27–32 (1981)

reconsidered the real-gas effects, and performed a careful error analysis:

The major contributors to the uncertainty in refractivity are the uncertainties in the measurements of temperature and pressure. The magnitude of the uncertainty due to variation in CO

_{2}concentration can approach that of the uncertainties due to the pressure and temperature measurements. Therefore, the CO_{2}concentration should be treated as a variable and should be observed.

This statement received excellent confirmation a decade later in the 1993
and 1994 papers of Birch and Downs (see below), who measured the effects
of varying CO_{2} concentration in laboratory air.

Unfortunately, Jones seems to have overlooked the
Peck – Reeder work.
He produced a formula good for the visible region with an error in the
refractivity of a few parts in 10^{8}.

H. Matsumoto

The refractive index of moist air in the 3-µm region

Metrologia18, 49–52 (1982)

measured the effects of water vapor, and found considerable effects due to its near-IR absorptions.

Then additional errors due to water vapor, even in the visible, were found by

K. P. Birch, M. J. Downs

The results of a comparison between calculated and measured values of the refractive index of air

J. Phys. E: Sci. Instrum.21, 694–695 (1988)

and confirmed by

J. Beers, T. Doiron

Verification of revised water vapour correction to the refractive index of air

Metrologia29, 315–316 (1992)

P. E. Ciddor

Refractive index of air: new equations for the visible and near infrared

Appl. Opt.35, 1566–1573 (1996)

We can regard Ciddor's results as definitive, at least for the present. They have been adopted as the basis of a new standard by the International Association of Geodesy (IAG). Geodetic applications often require the group rather than the phase index; see

P. E. Ciddor, R. J. Hill

Refractive Index of Air. 2. Group Index.

Applied Optics (Lasers, Photonics and Environmental Optics),38, 1663–1667 (1999)

Soon after Ciddor's first paper, a new set of highly accurate measurements at 4 wavelengths in the visible spectrum was published by

G. Bönsch and E. PotulskiTheir work focuses primarily on determining the effects of water vapor and CO

Measurement of the refractive index of air and comparison with modified Edlén's formulae

Metrologia35, 133–139 (1998)

They start with the “new” Edlén formula of 1966.
The aim is to provide the highest possible accuracy for laboratory conditions,
so they consider only a small temperature range about 20°C and a
CO_{2} mixing ratio near 400 ppm, where *linear* formulae are
adequate; their results are not sufficiently general for field use in
the actual atmosphere.
They give a nice practical summary of their results in an Appendix.

As they essentially confirm the results of previous recent studies — in particular, the water-vapor corrections of Birch & Downs and more recent workers — I would regard their paper as indirectly supporting Ciddor's results. The uncertainty in the refractive index of air is about 1 unit in the 8th decimal place.

For further discussion of these matters, see the Web pages of NIST and the IAG.

The above discussion refers to the effects of water vapor in the visible spectrum. But water vapor also has strong absorptions in the near infrared, which (by classical dispersion theory) produce considerable effects on the dispersion curve there. The following reference gives an example of the size of these effects out to 25 microns (in spite of the narrower scope suggested by its title):

R. J. Mathar

Calculated refractivity of water vapor and moist air in the atmospheric window at 10 μm

Appl. Opt.43, 928–932 (2004)

Similar calculations were published by

at about the same time; and Mathar subsequently published convenient series expansions for the infrared refractivity of air, given the humidity:M. M. Colavita, M. R. Swain, R. L. Akeson, C. D. Koresko, and R. J. Hill

Effects of atmospheric water vapor on infrared interferometry

Pub. Astron. Soc. Pacific116, 876–885 (2004)

R. J. Mathar

Refractive index of humid air in the infrared: Model fits

J. Optics A: Pure Appl. Opt.9, 470–476 (2007)

These authors point out that the large decrease in refractivity in the
middle IR — in particular, in the important 10μm window —
is due to the strong absorptions of the CO_{2} fundamental
near 15μm and the pure-rotational spectrum of water vapor at longer
wavelengths (which Colavita et al. repeatedly, and erroneously, call a
vibration-rotation band).

andK. P. Birch, M. J. Downs

An updated Edlén equation for the refractive index of air

Metrologia30, 155–162 (1993)

K. P. Birch, M. J. Downs

Correction to the updated Edlén equation for the refractive index of air

Metrologia31, 315–316 (1994)

As their titles indicate, these papers were intended to revive the (new) Edlén formula. Although the first paper contains a section headed “Review of Developments Since 1966”, it ignores the works of Owens (1967), Peck & Reeder (1972), and Jones (1981). [Well, there's a reference to the 1967 paper by “Owen” (sic), but only to cite his percentage composition for standard air; there's no mention of his work on the dispersion relation.]

Their 1993 paper discusses a variety of effects: water vapor,
compressibility (i.e., real-gas effects in the equation of state),
revisions of the temperature scale, and CO_{2} content.
They retained Edlén's dispersion formula, and made measurements
only at the He-Ne laser wavelength (633 nm).
Unfortunately, as the 1994 paper points out, they put the
temperature-scale revision in the wrong way, so that the final results
are wrong. (Table 1 of the 1994 paper is a corrected version of Table 4
in the 1993 paper.)

However, the main value of this work was to show convincingly that the
CO_{2} content of laboratory air is usually appreciably higher
than in the free atmosphere — I suppose because laboratories are
full of people continually exhaling the gas — and that the effect of
this on the refractivity is plainly measurable. In this sense, I take
their work as experimental confirmation of the work of Jones (1981).

A proper treatment of the CO_{2} problem would take account of the
fact that the increase in its abundance is at the expense of molecular
oxygen. The dispersion curves of both gases should be treated accurately,
instead of simply scaling the refractivity of air, as Birch & Downs do.
In any case, their results are by now completely superseded by Ciddor's.

Thanks to William M. Robertson for providing copies of these Birch & Downs papers to me.

Similarly, a change of 0.1 mm of mercury changes the density and hence the
refractivity of air by about a part in 10^{4}, so that still
smaller pressure changes might be significant. This is also the level at
which deviations from the ideal gas law become appreciable.

**Stone**
says that, for accurate positional astrometry,
errors should be kept below 0.05 arc seconds at zenith distances
up to 70°, where the sea-level refraction is about 147 arc seconds.
This refraction is nearly 3000 times the allowed error.

For most purposes, we can neglect effects smaller than 1 or 2 seconds of arc at the horizon, corresponding to relative errors in density of 1 part in 1000, or perhaps 1 part in 2000. It appears that an error of 1 part in a few thousand is small enough for astronomers.

This means an accuracy of 1 part in 3000 for the refractivity, or 1 part
in 10^{7} for the refractive index, is adequate for most
astronomical purposes. That would make the Edlén formulae
acceptable. But, as the more recent formulae — in particular,
the Peck – Reeder formula
— are more reliable, as well as
less expensive to calculate, I think one of them should be used.

Note that 1 part in 3000 requires measuring temperatures to 0.1°C or better accuracy (not just precision); pressures must be accurate to about 1/3 of a millibar; and the water-vapor mixing ratio should be known to a relative accuracy of about 10%.

For astronomical refraction calculations in the **visible** spectrum,
the Peck – Reeder formula
seems a good compromise between accuracy and economy of calculation.
I recommend it.

For other purposes, such as geodesy, more accuracy may well be required.
Then one should adopt
Ciddor's results in the visible, and take account of
the large dispersive effects of water vapor and CO_{2} in the
infrared.

Copyright © 2003 – 2007, 2010, 2011 Andrew T. Young

Back to the ...

or the
**
GF home page
**

or the
**
Overview page
**