The Imaging Pipeline

Table of Contents

This page describes the SDSS imaging pipeline, which processes the raw telescope imaging to produce calibrated FITS files and catalogs of imaging parameters. Each step in the pipeline is outlined below.

More detail is available in the Algorithms section of this site. The Early Data Release (EDR) paper is the fundamental resource for understanding the processing and data products from the SDSS, describing the pipelines and contents of generated data products. Successive data release papers describe changes between data releases – see SDSS Data Release Publications for a complete list, with links. The Data Release 9 paper describes some changes to astrometry algorithms that occurred with DR9 and later.

In addition to these data release papers, the SDSS technical summary provides more general information on the survey. The SDSS Project Book, written for a funding proposal early in the survey, is the most exhaustive description of the survey but was last updated in 1997. The SAS datamodel has a table of the most commonly useful files in the SAS. Truly gory details can be found on Robert Lupton’s page of photo documents, including the draft photometric reduction paper as well as a treasure trove of photometric data reduction wisdom.

The remainder of this page starts with a brief overview of imaging data processing, followed by sections that describe the steps in data processing in detail. In addition to the more detailed description, each detail section provides references to papers that give additional details, and a table of the files associated with that step of the pipeline that can be found in the SAS. These tables include links to descriptions of the formats of those files and templates that can be used to generate SAS URLs for those files. The templates are in “C printf” format, and can be used in C, bash, Python, and many other languages to automatically generate URLs.

Most of the catalog data (but not the images themselves) have been loaded into the Catalog Archive Server (CAS) database. Users are often better off obtaining SDSS-III data through a carefully constructed CAS query than they are downloading the data files from the SAS. Simple queries can be used to select just the objects and parameters of interest, while more complex queries can be used to do complex calculations on many objects, thereby avoiding the need to download the data on them at all.

Overview

Imaging Observing: We used three instruments when collecting imaging data: the imaging camera mounted on the primary 2.5m SDSS telescope, which collected the imaging data themselves; the 0.5m photometric telescope, which collected images of photometric standard stars and reference fields; and a 10 micron all sky scanner (and later camera), used to detect clouds. The photometric telescope is no longer used in imaging data reduction and its data is not described here.
Imaging Data Reduction: The SDSS data processing factory used a collection of pipelines to process and calibrate the data from the imaging camera, ultimately producing a variety of data products including images with instrumental signatures removed, a photometric solution for the night, and a catalog of objects found in the data.
Resolution of Multiple Detections: The resolve step declares one observation on an object the primary observation, and others secondary, thereby avoiding unintentional duplication of objects. This step ultimately produces an astrometrically and photometrically calibrated catalog of objects found in the data from the imaging camera.
Recalibration: Improved photometric calibrations became possible at the end of the survey; we used a separate pipeline to refine the photometric calibrations.
Final Photometric Catalog: The results of the resolve and calibration stages are combined with the uncalibrated object catalogs to produce final catalogs.

Notes

In the tables below, add ‘http://data.sdss3.org/sas/dr12/’ to all ‘URL format’values to get the full URL.
Data reduction could occur multiple times for both images and spectra. Each time we repeated a data reduction, we labeled the output from that reduction process with a distinct rerun number.
In the case of imaging rerun numbers, different decades in the rerun number designate significant differences in the data reduction software. For example, rerun 1 and rerun 2 of a given run would have been processed by identical (or at least very similar) versions of the data reduction software, while rerun 40 would have been reduced with a significantly different version of the pipeline. However, the only relevant reruns for DR8 are 157 and 301, with the bulk of the data in rerun 301.

Imaging Observing

Imager

The imager collected survey data from the imager in drift scan (also called TDI) mode. We use the term “run” to designate a single scan along a great circle. The data acquisition system divided the data from each CCD into frames, stored as FITS files; the observatory sent data to be processed as a collection of FITS files, each corresponding to an arc along the great circle 1361 rows (539 arcseconds) long from a single CCD. In addition to the images themselves, the data acquisition system also generated a variety of metadata and other engineering files, which included pixel statistics and a catalog of bright stars for each field.

SAS files generated in imager data collection
File Type	in/out	Description	URL format	format parameters
idReport	out	records runs collected on a night	boss/photo/redux/%d/%d/logs/idReport-%05d.par	rerun, run, mjd
idR	out	raw imaging frames	boss/photo/data/%d/fields/%d/idR-%06d-%s%d-%04d.fit.Z	run, camcol, run, filter, camcol, field
idGang	out	assorted data gathered by the data acquisition system	boss/photo/data/%d/gangs/idGang-%06d-%s%d-%04d.fit	run, run, r(ow) or c(ol), row number or column number, frame
idFrameLog	out	camera position information for each frame	boss/photo/redux/%d/%d/logs/idFrameLog-%06d-%d.par	rerun, run, run, crate
opCamera	out	imaging camera geometry	boss/photo/redux/%d/%d/logs/opCamera-%05d.par	rerun, run, mjd (of generation)
opConfig	out	CCD readout configurations	boss/photo/redux/%d/%d/logs/opConfig-%05d.par	rerun, run, mjd (of generation)
opECalib	out	CCD electronic calibrations	boss/photo/redux/%d/%d/logs/opECalib-%05d.par	rerun, run, mjd (of generation)

Infrared All-Sky Camera

Clouds can be clearly seen at night in a 10 micron all-sky camera. We used two such cameras at different times in the survey to monitor the sky for cloud (and therefore non-photometric conditions).

SAS files generated by the infrared all sky camera or scanner
File Type	in/out	Description	URL format	format parameters
cR	out	A single image from the infrared camera	apo_staging/ircam/%05d/cR%06d.fit	mjd, UTC time (HHMMSS)
irsc	out	A log of statistics on ircam frames	apo_staging/ircam/%05d/irsc.log	mjd

Other metadata

The observatory produces an assortment of engineering and other metadata.

Other SAS files generated in observing
File Type	in/out	Description	URL format	format parameters
night log	out	A prose account of the night	apo_staging/astrolog/%05d/manualLog-full.txt	mjd
idWeather	out	A log of weather data	boss/photo/redux/%d/%d/logs/idWeather-%05d.par	rerun, run, mjd

Imaging Data Reduction

Serial Stamp Collecting Pipeline (SSC)

The SSC repackages some of the data produced by the data acquisition system and produces postage stamps of bright reference stars.

SAS files read by the SSC pipeline
File Type	in/out	Description	URL format	format parameters
scPlan	in	the SSC processing plan	boss/photo/redux/%d/%d/ssc/scPlan.par	run, rerun
idReport	in	records runs collected on a night	boss/photo/redux/%d/%d/logs/idReport-%05d.par	rerun, run, mjd
opCamera	in	imaging camera geometry	boss/photo/redux/%d/%d/logs/opCamera-%05d.par	rerun, run, mjd (of generation)
opConfig	in	CCD readout configurations	boss/photo/redux/%d/%d/logs/opConfig-%05d.par	rerun, run, mjd (of generation)
opECalib	in	CCD electronic calibrations	boss/photo/redux/%d/%d/logs/opECalib-%05d.par	rerun, run, mjd (of generation)
opBC	in	CCD bad pixel file	boss/photo/redux/%d/%d/log/opBC-%d.par	rerun, run, mjd_bp (see fpPlan)
idR	in	raw imaging frames	boss/photo/data/%d/fields/%d/idR-%06d-%s%d-%04d.fit.Z	run, camcol, run, filter, camcol, field
idGang	in	assorted data gathered by the data acquisition system	boss/photo/data/%d/gangs/idGang-%06d-%s%d-%04d.fit	run, run, r(ow) or c(ol), row number or column number, frame
koCat	in	the catalog of known bright stars	boss/photo/redux/%d/%d/ssc/koCat-%06d.fit	run, rerun, run

Postage Stamp Pipeline (PSP)

References: 2002AJ….123..485S, section 4.3

The PSP measures the bias and global sky level, and fits a model for the point spread function for each field.

SAS files used or generated by the PSP pipeline
File Type	in/out	Description	URL format	format parameters
psParam	in	PSP tunable parameters	boss/photo/redux/%d/%d/photo/psParam.par	run, rerun
psPlan	in	the PSP processing plan	boss/photo/redux/%d/%d/photo/psPlan.par	run, rerun
opCamera	in	imaging camera geometry	boss/photo/redux/%d/%d/logs/opCamera-%05d.par	rerun, run, mjd (of generation)
opConfig	in	CCD readout configurations	boss/photo/redux/%d/%d/logs/opConfig-%05d.par	rerun, run, mjd (of generation)
opECalib	in	CCD electronic calibrations	boss/photo/redux/%d/%d/logs/opECalib-%05d.par	rerun, run, mjd (of generation)
opBC	in	CCD bad pixel file	boss/photo/redux/%d/%d/log/opBC-%d.par	rerun, run, mjd_bp (see fpPlan)
idB	in	bias vector	boss/photo/redux/%d/%d/photo/calib/idB-%06d-%c%d.fit	rerun, run, calibration run, filter, camcol
psField	out	the initial photometric calibration and point spread function fit by field (see the extractingPSF images page)	boss/photo/redux/%d/%d/objcs/%d/psField-%06d-%d-%04d.fit	rerun, run, camcol, run, camcol, field

Astrometric Pipeline (astrom)

References: 2003AJ….125.1559P

The astrometric pipeline calculates the astrometric solution. The astrometry and astrometry QA (UPDATE LINK) web pages provide more information on the algorithms used and the proper
interpretation of the output.

SAS files used or generated by the astrom pipeline
File Type	in/out	Description	URL format	format parameters
asParam	in	astrom tunable parameters	boss/photo/redux/%d/%d/astrom/asParam.par	rerun, run
asPlan	in	the astrom processing plan	boss/photo/redux/%d/%d/astrom/asPlan.par	rerun, run
opCamera	in	imaging camera geometry	boss/photo/redux/%d/%d/logs/opCamera-%05d.par	rerun, run, mjd (of generation)
opConfig	in	CCD readout configurations	boss/photo/redux/%d/%d/logs/opConfig-%05d.par	rerun, run, mjd (of generation)
idReport	in	records runs collected on a night	boss/photo/redux/%d/%d/logs/idReport-%05d.par	rerun, run, mjd
idWeather	in	A log of weather data	boss/photo/redux/%d/%d/logs/idWeather-%05d.par	rerun, run, mjd
asTrans	out	transformation coefficients from row, column to great circle coordinates	boss/photo/redux/%d/%d/astrom/asTrans-%06d.fit	rerun, run, run

Frames Pipeline (frames)

References: 2002AJ….123..485S, section 4.4 The frames pipeline applies flat field and bias corrections to each frame, and interpolates values for pixels in bad columns and bleed trails and those corrupted by cosmic rays. In previous data releases, this pipeline would produce a “corrected frame” or fpC file. These files are no longer produced. Instead, a frame file is produced by a different method, though it contains substantially the same information. The frames pipeline described here also detects bright objects and estimates the varying sky background, detects fainter objects, and measures a variety of properties for each object. Several algorithms pages describe details of the processing performed by frames.

Measuresof flux and magnitudes describes the variety of methods frames uses to measure the magnitude of each object,
including radialprofiles, fiber, model, cModel, Petrosian and PSF magnitudes and when to use each.
Classificationand Morphology describes other measurements of objects, including star/galaxy separation, model fits, surface brightnesses, ellipticities,
adaptivemoments and isophotal quantities.
IMPORTANT! Photometricprocessing flags describes the variety of flags frames sets for each object, essential for properly interpreting the catalog.
Flatfield creation and quality. See the Ubercal paper for a more detailed discussion of flat-fielding.
Sky measurement
Deblending describes how the frames pipeline decides if an initial single detection is in fact a blend of multiple overlapping objects, and, if so, how it separates (or “deblends”) them.
Creation of imaging masks

SAS files used or generated by the frames pipeline
File Type	in/out	Description	URL format	format parameters
fpParam	in	frames tunable parameters	boss/photo/redux/%d/%d/photo/fpParam.par	rerun, run
fpPlan	in	frames processing plan	boss/photo/redux/%d/%d/photo/fpPlan.par	rerun, run
idReport	in	records runs collected on a night	boss/photo/redux/%d/%d/logs/idReport-%05d.par	rerun, run, mjd
koCat	in	the catalog of known bright stars	boss/photo/redux/%d/%d/ssc/koCat-%06d.fit	run, rerun, run
opConfig	in	CCD readout configurations	boss/photo/redux/%d/%d/logs/opConfig-%05d.par	rerun, run, mjd (of generation)
opECalib	in	CCD electronic calibrations	boss/photo/redux/%d/%d/logs/opECalib-%05d.par	rerun, run, mjd (of generation)
opBC	in	CCD bad pixel file	boss/photo/redux/%d/%d/log/opBC-%d.par	rerun, run, mjd_bp (see fpPlan)
idB	in	bias vector	boss/photo/redux/%d/%d/photo/calib/idB-%06d-%c%d.fit	rerun, run, calibration run, filter, camcol
psField	in	the initial photometric calibration and point spread function fit by field (see the extractingPSF images page)	boss/photo/redux/%d/%d/objcs/%d/psField-%06d-%d-%04d.fit	rerun, run, camcol, run, camcol, field
asTrans	in	tranformation coefficients from row, column to great circle coordinates	boss/photo/redux/%d/%d/astrom/asTrans-%06d.fit	rerun, run, run
fpFieldStat	out	field statistics	boss/photo/redux/%d/%d/objcs/%d/fpFieldStat-%06d-%d-%04d.fit	rerun, run, camcol, run, camcol, field
fpObjc	out	the (uncalibrated) object catalog	boss/photo/redux/%d/%d/objcs/%d/fpObjc-%06d-%d-%04d.fit	rerun, run, camcol, run, camcol, field
fpBIN	out	a 4×4 binned version of the corrected image after masking of objects and subtraction of sky, an image of the estimate of the sky, an image of the uncertainty in the sky estimate, and a table of bright star wings	boss/photo/redux/%d/%d/objcs/%d/fpBIN-%06d-%c%d-%04d.fit.gz	rerun, run, camcol, run, filter, camcol, field
fpM	out	the frame masks (read using readAtlasImages)	boss/photo/redux/%d/%d/objcs/%d/fpM-%06d-%c%d-%04d.fit.gz	rerun, run, camcol, run, filter, camcol, field
fpAtlas	out	the atlas images for all objects detected (read using readAtlasImages)	boss/photo/redux/%d/%d/objcs/%d/fpAtlas-%06d-%d-%04d.fit	rerun, run, camcol, run, camcol, field

Resolve Multiple Detections

A given area on the sky may be observed by multiple runs. The final calibrated catalogs contain a field declaring whether the specific observation of the object is “primary,” and the canonical measurement of the object, or “secondary,” a duplicate. These catalogs are generated twice, once based on the single run in isolation by exportChunk, and again in the resolve step. The catalogs produced by exportChunk declare an object primary or secondary based on position alone, introducing the possibility of inconsistencies due to errors in position; it is possible, for example, that two separate observation of an object be declared primary if the position measured for the object is slightly different in each detection. The catalog generated by the resolve step, on the other hand, uses an operational database to search for other detections, and guarantees that one and only one detection be declared primary. See the EDR paper, 2002AJ….123..485S, section 4.7 for details on how objects are resolved. The catalogs produced by exportChunk can be found in the run directory of each run:

SAS files used or generated by the resolve stage
File Type	in/out	Description	URL format	format parameters
reObjGlobal	out	Global resolve status for every object in a field	boss/resolve/%s/%d/%d/resolve/%d/reObjGlobal-%06d-%d-%04d.fit	resolve rerun, rerun, run, camcol, run, camcol, field
thingList	out	The full list of catalog entries for the resolved survey	boss/resolve/%s/thingList.fits	resolve rerun
thingIndex	out	The full list of unique primary objects, gives position of an object in the thingList	boss/resolve/%s/thingIndex.fits	resolve rerun
window_flist	out	The full list of fields used to determine the window function	boss/resolve/%s/window_flist.fits	resolve rerun
window_unified	out	The full list of polygons determining the primary area of the window function	boss/resolve/%s/window_unified.fits	resolve rerun
window_blist	out	The full list of balkans determining the primary area of the window function	boss/resolve/%s/window_blist.fits	resolve rerun
window_bcaps	out	The full list of caps determining the primary area of the window function	boss/resolve/%s/window_bcaps.fits	resolve rerun
window_bindx	out	Matches balkans to fields in the window function, sorted by balkans	boss/resolve/%s/window_bindx.fits	resolve rerun
window_findx	out	Matches balkans to fields in the window function, sorted by sorted by fields	boss/resolve/%s/window_findx.fits	resolve rerun

Photometric Recalibration (Ubercal)

References: 2008ApJ…674.1217P The “ubercal” recalibration recalibrates the survey by simultaneously solving for the calibration parameters and relative stellar fluxes using overlapping observations. The algorithm used decouples the problem of relative calibrations from that of absolute calibrations; the absolute calibration is reduced to determining a few numbers for the entire survey. See the photometric calibration algorithms page.

SAS files generated in photometric recalibration
File Type	in/out	Description	URL format	format parameters
calibPhotomGlobal	out	Ubercal results (nanomaggies/count) and associated flat fields	boss/calib/%s/%d/%d/nfcalib/calibPhotomGlobal-%06d-%d.fits	calibration rerun, rerun, run, run, camcol
calib	out	Uberal quality assurance plots	boss/calib/%s/plots/calib-%06d-%c-%s.png	calibration rerun, run, filter, type (flat, hist, run)

Photometric Catalogs

The final photometric catalogs contain the raw data that will be loaded into the CAS database.

SAS files produced by the final catalog
File Type	in/out	Description	URL format	format parameters
photoObj	out	Full, calibrated outputs of the imager photometric pipeline	boss/photoObj/%d/%d/%d/photoObj-%06d-%d-%04d.fits	rerun, run, camcol, run, camcol, field
photoField	out	Summary outputs of the properties of every field	boss/photoObj/%d/%d/photoField-%06d-%d.fits	rerun, run, run, camcol
photoRunAll	out	Summary information of the properties of every photometric run	boss/photoObj/photoRunAll-%s.fits	release (dr12)