The Optical Spectroscopy Pipeline
Introduction and Resources
The SDSS Early Data Release (EDR) paper is the original resource for understanding the processing and data products from the SDSS, describing the pipelines and contents of generated data products. Successive data release papers: DR1, DR2, DR3, DR4, DR5, DR6, DR7, DR8, DR9, and DR10, describe changes between data releases. Pipeline changes for DR12 will be fully described in the DR12 paper, and are also summarized immediately below. The technical summary paper provides more general information on the SDSS-I survey, the SDSS-III summary paper provides general information on SDSS-III, the BOSS overview paper provides general information for BOSS, and the 1D pipeline paper describes redshift measurement and classification. The algorithms page includes links to pages describing algorithms used by the spectroscopic data-reduction pipelines. This page provides a summary of those steps and the associated output files.
Most of the spectroscopic catalog data (but not the spectra themselves) have been loaded into the Catalog Archive Server (CAS) database. Depending on the scientific use case, users may be better off obtaining SDSS data through a carefully constructed CAS query rather than by downloading the data files from the SAS. Simple queries can be used to select just the objects and parameters of interest, while more complex queries can be used to do complex calculations on many objects, thereby avoiding the need to download the data on them at all.
The remainder of this page gives a short description of pipeline changes for DR12, a brief overview of spectroscopic data processing, and sections that describe the steps of data processing in detail. In addition to the descriptions, each section provides references to papers that give additional details, and a table of the files associated with that step of the pipeline that can be found in the SAS. These tables include links to the file format documentation (the “data model”) and templates which can be used to generate SAS URLs for those files. The templates are in “C printf” format, and can be used in C, bash, Python, and many other languages to automatically generate URLs.
Changes for DR12
The spectroscopic data reduction pipeline for DR12 is largely unchanged from DR10. The following are the changes of note:
- Routine updates were made to the pixel-response flats, bias images and subtraction methods, and masks for bad pixels and bad columns. The latter in particular have reduced the incidence of corrupted but unflagged spectra relative to DR10.
- Two versions of the spectroscopic reductions are included in DR12: v5_7_0, which is used for the core BOSS survey analysis, and v5_7_2, which is used for the analysis of SEQUELS data. The DR11 sample of BOSS data obtained through summer 2013 was reduced with the v5_6_5 version of the pipeline.
- Flux calibration vectors are now linearly interpolated over the locations of several observed-frame Balmer-series wavelength ranges in order to mitigate the low-level imprinting of features from calibration stars onto science spectra (see Busca et al. 2013). This effect, and the associated flux-calibration change, are generally detectable only in stacks of many spectra, not in individual spectra.
- For a number of plates containing special targets such as emission-line galaxies, the maximum redshift of galaxy model fits was increased from 1 to 2. The spPlateZrange.par file in the “speclog” product specifies the list of plates for which this increased redshift range was allowed.
Overview
- Spectroscopic Observing
- The spectrographs mounted on the primary 2.5m telescope collected spectra from each plate. There are two spectrographs, each of which collects data from 320 (SDSS) or 500 (BOSS) fibers. Each spectrograph has a dichroic that sends light to red and blue cameras, so the instrument produces a total of four images for each exposure.
- Spectroscopic Data Reduction
- The spectroscopic pipelines extract one dimensional spectra from the raw exposures produced by the spectrographs, calibrate them in wavelength and flux, combine the red and blue halves of the spectra, measure features in these spectra, measure redshifts from these features, and classify the objects as galaxies, stars, or quasars.
Notes
- In the tables below, prepend ‘http://data.sdss3.org/sas/dr12/’ to all ‘URL format’ values to get the full URL.
- URL suffixes are listed for the original SDSS-I/-II survey. Replace “sdss/” with “boss/” to get the equivalent BOSS survey files.
- Data reduction can occur multiple times for both images and spectra. The output of each unique reduction is labeled with a distinct “rerun” number (for photometry) or different “run2d” and “run1d” versions (for spectroscopy).
- The legacy target selection and boss target selection algorithm pages describe how the various spectroscopic target classes are selected for the BOSS and legacy SDSS surveys.
- The tiling algorithms page describes the process by which the spectroscopic plates are designed and placed relative to each other.
Spectroscopic Observing
Plate Plugging (plug)
When the observatory is ready to observe a plate, the observatory staff plugs optical fibers into the holes drilled into the plates, and maps which fiber corresponds to which hole (and therefore which object) by shining light through each fiber. This data is incorporated into one of the HDUs of the spPlate file described below.
Raw Data Collection
Observers mount cartridges containing the drilled, plugged plates on the telescope, and collect a series of 15-minute exposures on each plate until it reached a threshold estimated signal to noise ratio and at least three exposures had been collected.
| File Type | in/out | Description | URL format | format parameters |
|---|---|---|---|---|
| sdReport | out | records exposures collected on a night | Not public | |
| sdR | out | raw spectroscopic data frames | sdss/spectro/data/%d/sdR-%c%d-%08d.fit.gz | mjd, CCD (r or b), camera (1 or 2), exposure id |
Spectroscopic Data Reduction
The idlspec2d software has two major pipeline steps:
- spec2d
- Extract and calibrate 1-dimensional spectra from 2-dimensional raw CCD data
- spec1d
- Measure object classifications and redshift from those 1D spectra.
Two-dimensional Pipeline (spec2d)
References: 2002AJ….123..485S, section 4.10.1
The spec2d pipeline reads science and calibration exposures from the spectrographs, reduces and calibrates the science exposures, extracts the one dimensional spectra from the two dimensional exposures, stacks multiple exposures into combined spectra, and produces corresponding masks and noise estimates.
| File Type | in/out | Description | URL format | format parameters |
|---|---|---|---|---|
| spPlan2d | in | the spectro2d processing plan | sdss/spectro/redux/%d/%04d/spPlan2d-%04d-%d.par | rerun, plate, plate, mjd |
| spPlancomb | in | the processing plan for combining spectra | sdss/spectro/redux/%d/%04d/spPlancomd-%04d-%d.par | rerun, plate, plate, mjd |
| plPlugMapM | in | records which fiber corresponds to which hole in a plate (and therefore objects, and what coordinates on the sky) | Not public | |
| sdReport | in | records exposures collected on a night | Not public | |
| sdR | in | raw spectroscopic data frames | sdss/spectro/data/%d/sdR-%c%d-%08d.fit.gz | mjd, CCD (r or b), camera (1 or 2), exposure id |
| spCFrame | out | calibrated spectra for a single CCD and exposure | sdss/spectro/redux/%d/%04d/spCFrame-%c%d-%08d.par | rerun, plate, CCD (r or b), camera, exposure id |
| spPlate | out | the 640 (SDSS) or 1000 (BOSS) combined flux- and wavelength-calibrated spectra over all exposures (potentially spanning multiple nights) for a given mapped plate | sdss/spectro/redux/%s/%04d/spPlate-%04d-%d.fits | run2d, plate, plate, mjd |
One-dimensional Pipeline (spec1d)
Reference: 2012AJ….144..144B
The spec1d pipeline reads spectra and determines classifications, redshifts, and other spectroscopic parameters. It produces the following files.
| File Type | in/out | Description | URL format | format parameters |
|---|---|---|---|---|
| spZline | out | emission line fits | sdss/spectro/redux/%d/%04d/spZline-%04d-%d.fits | rerun, plate, plate, mjd |
| spZall | out | all spectroscopic classifications and redshifts | sdss/spectro/redux/%d/%04d/spZall-%04d-%d.fits | rerun, plate, plate, mjd |
| spZbest | out | spectroscopic classifications and redshifts | sdss/spectro/redux/%d/%04d/spZbest-%04d-%d.fits | rerun, plate, plate, mjd |
Per-object Spectrum Files
As of DR9, the pipeline also provides a reformatting of the same spectral data into one file per PLATE-MJD-FIBER, including the coadded spectra from spPlate, the emission line fits from spZline, the redshifts and classifications from spZall and spZbest, and optionally the individual exposure spectra from spCFrame. These are useful when you need all of the information for a small subset of objects.
| File Type | in/out | Description | URL format | format parameters |
|---|---|---|---|---|
| spec | out | All spectral information for a single PLATE-MJD-FIBER | sdss/spectro/redux/%d/spectra/%04d/spec-%04d-%05d-%04d.fits | run2d, plate, plate, mjd, fiber |
| speclite | out | All spectral information for a single PLATE-MJD-FIBER except the individual exposures | sdss/spectro/redux/%d/spectra/lite/%04d/spec-%04d-%05d-%04d.fits | run2d, plate, plate, mjd, fiber |
Stellar Parameters Pipeline (sspp)
References: 2008AJ….136.2022L, 2008AJ….136.2050L, 2008AJ….136.2070A
The SEGUE stellar parameters pipeline produces a number of files, stored together:
| File Type | in/out | Description | URL format | format parameters |
|---|---|---|---|---|
| ssppOut | out | SSPP stellar parameters ([Fe/H], log g, etc.) | sdss/sspp/%d/%04d/output/param/ssppOut-%04d-%5d.fit | rerun, plate, plate, mjd |
| ssppOut_lineindex | out | SSPP line indices | sdss/sspp/%d/%04d/output/param/ssppOut-%04d-%5d.lineindex.fit | rerun, plate, plate, mjd |
Galaxy Parameter Pipelines
For spectra of galaxies, several additional analysis pipelines are applied to derive catalogs of physical galaxy parameters. These pipelines and their outputs are described fully in the Galaxy Parameter Pipeline pages.
