IRAF TUTORIALS: DOHYDRA

Abstract

This tutorial and exercise illustrates the use of dohydra. As an exercise, artificial multifiber data will be created and used. The reader may then execute the steps illustrated here as well as explore the options. As a tutorial this document may be read alone and the sample output may be viewed. The version you are looking at has the graphical output as part of the document making it self-contained and more easily read as a tutorial but it is also slower to load. A version of this document which uses optional links to a Postscript viewer may be selected here.

The dohydra reduction task is specialized for scattered light subtraction, extraction, flat fielding, fiber throughput correction, wavelength calibration, and sky subtraction of Hydra (or other) multifiber spectra. It is a command language script which collects and combines the functions and parameters of many general purpose tasks to provide a single complete data reduction path. The task provides a degree of guidance, automation, and record keeping necessary when dealing with the large amount of data generated by this and other multifiber instruments.


Introduction

This tutorial and exercise presents a simple, though complete, use of the dohydra task. This task is a complex script that guides the user through the various steps of flat fielding, scattered light subtraction, extraction, dispersion calibration, and sky subtraction for Hydra or other multifiber data (the dofibers task in the generic specred package is essentially identical). In addition to organizing and guiding the user through these steps the task attempts to provide additional record keeping such that only images and steps not yet performed will be done. This exercise assumes some knowledge of how to use IRAF such as starting the IRAF session, moving about directories, and editing parameters.

There is on-line help for this task as well as a typeset user's guide. These are essentially identical and differ only in visual appearance and where the parameters are described. To read the on-line help type

hy> phelp dohydra
[output]
A printed copy of this help can be produced with the command
hy> help dohydra | lprint
where the lprint command will print to the default printer which may be changed using the hidden parameters. Finally, the typeset Postscript version may be obtained and read by clicking here

Getting Started

Start an IRAF session and go to a directory in which to do this exercise; you could use your home directory if you wish. Now load the hydra package as shown below.

cl> noao
[List of tasks]
no> hydra
[List of tasks]
hy> 
The hydra package includes all the individual tools for processing the spectra apart from the basic CCD processing. However, in this exercise we are only concerned with the dohydra task which reduces the data and itself calls the other tasks it needs.

To create some sample Hydra data, type the following command. If the data already exists then nothing will happen.

hy> demos mkdohydra
Creating image demoobj ...
Creating image demoflat ...
Creating image demoarc ...
Creating image demostd ...
hy> dir
demoapid1       demoarc.imh     demoobj.imh
demoapid2       demoflat.imh    demostd.imh     

The data created consists of a flat field image demoflat, an arc image demoarc, a bright standard star image demostd, and an object image demoobj. These images have already been processed to subtract the overscan bias level, trim the overscan region, and apply a zero level calibration. In addition there are two aperture identification files demoapid1 and demoapid2; but we will only use the first one.

Let's begin by looking at one of the images. This step requires that you have a display server running on your console (SAOimage or XImtool) and that your session has been setup to display to that window.

hy> display demoobj 1 fill+
[This display is shown rotated to save space]

You may also wish to look at the other images and list out a header with imhead. You will see that this data has only 11 fibers and the image is only 100 by 256 pixels in size. This sample data is actually much smaller than typical data which has up to 100 fibers and much larger images. However, it has all the basic characteristics of multifiber data including a broken fiber.

The observing software for Hydra that positions the fibers for a particular field naturally knows the set of objects requested by the observer. This software creates an aperture identification file which is used by the reduction software. However, for other similar instruments the user may need or wish to create this file themselves. The file contains the fiber number, an identification of the type of spectrum in the fibers (-1=broken or unused, 0=sky, 1=object, 2=arc), and an optional title. Take a look at the sample file provided which does not contain any object titles though actual files typically would. Notice that there are 7 objects, 4 skys, and one broken fiber.

hy> type demoapid1
36 1 
37 0 
38 1 
39 1 
41 0 
42 1 
43 1 
44 0 
45 1 
46 -1 
47 0 
48 1 

Running DOHYDRA

In this section we will run dohydra on the sample multifiber data. Since dohydra combines many different operations the discussion below is broken up into the logical steps. The discussion will show the various prompts and interactions with commentary added. Sample output is also given.

In general the one dimensional spectra produced will have the same image name as the original image with the extension .ms which stands for multispec; a collection of one dimensional spectra in a two dimensional image. However, for calibration images such as the flat field and arc, you will notice funny names are used for the extracted spectra. The names are a concatenation of the original name and part of the aperture identification file. This is to allow reducing the data with different aperture identifications or flat fields. In this exercise you don't need to be concerned with this.

Parameters

To begin we set the package and task parameters. The package parameters set some global things used by many of the tasks in the package. Below are the suggested parameters for this exercise. To insure the results are as expected in this exercise it would be a good idea to first initialize all the parameters to their default values with unlearn.

hy> unlearn hydra
hy> epar hydra
                                   I R A F  
                    Image Reduction and Analysis Facility
PACKAGE = imred
   TASK = hydra

(observa=          observatory) Observatory of data
(interp =                poly5) Interpolation type
(dispaxi=                    2) Image axis for 2D/3D images
(nsum   =                    1) Number of lines/columns/bands to sum for 2D/3D i

(databas=             database) Database
(verbose=                  yes) Verbose output?
(logfile=              logfile) Log file
(plotfil=                     ) Plot file

(records=                     )
(version= HYDRA V1: January 1992)
(mode   =                   ql)
($nargs =                    0)
hy> epar dohydra
                                   I R A F  
                    Image Reduction and Analysis Facility
PACKAGE = hydra
   TASK = dohydra

objects =              demoobj  List of object spectra
(apref  =             demoflat) Aperture reference spectrum
(flat   =             demoflat) Flat field spectrum
(through=                     ) Throughput file or image (optional)
(arcs1  =              demoarc) List of arc spectra
(arcs2  =                     ) List of shift arc spectra
(arcrepl=                     ) Special aperture replacements
(arctabl=                     ) Arc assignment table (optional)

(readnoi=              RDNOISE) Read out noise sigma (photons)
(gain   =                 GAIN) Photon gain (photons/data number)
(datamax=                INDEF) Max data value / cosmic ray threshold
(fibers =                   12) Number of fibers
(width  =                   4.) Width of profiles (pixels)
(minsep =                   5.) Minimum separation between fibers (pixels)
(maxsep =                   7.) Maximum separation between fibers (pixels)
(apidtab=            demoapid1) Aperture identifications
(objaps =                     ) Object apertures
(skyaps =                     ) Sky apertures
(arcaps =                     ) Arc apertures
(objbeam=                  0,1) Object beam numbers
(skybeam=                    0) Sky beam numbers
(arcbeam=                     ) Arc beam numbers

(scatter=                  yes) Subtract scattered light?
(fitflat=                  yes) Fit and ratio flat field spectrum?
(clean  =                   no) Detect and replace bad pixels?
(dispcor=                  yes) Dispersion correct spectra?
(savearc=                  yes) Save simultaneous arc apertures?
(skysubt=                  yes) Subtract sky?
(skyedit=                  yes) Edit the sky spectra?
(savesky=                  yes) Save sky spectra?
(splot  =                  yes) Plot the final spectrum?
(redo   =                  yes) Redo operations if previously done?
(update =                  yes) Update spectra if cal data changes?
(batch  =                   no) Extract objects in batch?
(listonl=                   no) List steps but don't process?

(params =                     ) Algorithm parameters
(mode   =                   ql)
In this case we only set the verbose package parameter so that we can see more clearly the various steps taken by the task. In the task parameters we set the number of fibers, the widths and minimum and maximum fiber separations (which can be measured with a task such as implot), the aperture identification file, and processing steps to perform. For this exercise we will do all the steps except applying the cosmic ray cleaning algorithm. The redo parameter is set so that if you want to repeat this exercise, possibly with some changes in the parameters, the operations will be repeated. Normally, dohydra will only do each operation on a data image if it has not been done previously, as determined by looking at the images in the directory.

There is a long set of algorithm parameters contained in the parameter set with the name params. In this exercise we will use all the default parameters. Very few of these parameters generally need to be modified. You may look at these parameters with epar params.

Defining the Fiber Apertures

The first thing the task does is to find the positions of the requested 12 fibers from the average of the 10 center lines in the flat field image. The flat field is used since it will have a strong signal in all fibers. It is smart enough to determine that one fiber is missing and broken. The aperture identification file is used to assign the fiber numbers and fiber data types. Initially all the apertures are the same size as specified by two of the algorithm parameters. The Resize query allows the user to have the apertures adjusted to a width where the fiber profile falls to a certain fraction of the peak (in this case 5% of the peak).

hy> dohydra
List of object spectra (demoobj): 
Set reference apertures for demoflat
Resize apertures for demoflat?  (yes): 
Edit apertures for demoflat?  (yes): 

We now look at the aperture identifications to make sure nothing is amiss such as misidentified fibers. Note that since all data are assumed to have the same number of fibers in essentially the same place, this is the only time you will need to look at and edit the apertures. The apedit aperture editing mode has many commands, which you can list with the '?' key, and allows you to do many things. For this data mostly one would correct misidentifications and adjust aperture widths. The window can be zoomed in various ways such as with 'X' or 'w' followed by 'x' (see gtools). When you are satisfied exit this stage with 'q'.

Fit traced positions for demoflat interactively?  (yes):
Fit curve to aperture 36 of demoflat interactively?  (yes):

The positions of the fibers are traced, as described for the aptrace task, from the central lines to the rest of the image. This is done in steps of 10 lines with an average of 10 lines. Once the positions of this subsample of lines have been measured a function is fit to the positions. This may be done interactively allowing you to adjust the fitting function and order of the function. The interactive graphical step uses a standard IRAF tool called icfit for interactive curve fitting. Within this mode you can also type '?' to get a list of the commands. Typically one would use only :order to change the order of the fit, and 'f' to cause the fit to be redone after changing any fitting parameter. Feel free to experiment and when you are done and have an acceptable fit (one which fits the measured positions to fractions of a pixel except for noise points caused by the influence of cosmic rays) exit with 'q'.

You will be queried whether to fit the trace for each aperture interactively. The fitting parameters used will be those last set, so normally after the first fit there should be no need to change the fitting parameters. You may answer with yes to examine the fit, no to skip examining this particular fiber, YES to examine all fibers without further queries, or NO to accept all remaining fits. For example, to adopt the fits for all remaining apertures answer with NO.

Fit curve to aperture 37 of demoflat interactively?  (yes): NO

Scattered Light Subtraction

The next step is to subtract the scattered light (as described for the apscatter task). This follows the aperture definitions because the positions of the fibers must be known in order to determine where to measure the scattered light.

Subtract scattered light from demoflat
Fit scattered light for demoflatnoscat interactively?  (yes): 
Smooth the scattered light for demoflatnoscat interactively?  (yes): 

You have the option of whether to do the scattered light fitting across the dispersion and smoothing of the cross dispersion fits interactively or not. Thte interactive graphical mode is again icfit. What you see are the data points for a particular line (the middle by default) excluding all the apertures and with a buffer of some number of pixels. You can adjust the fitting parameters and select regions to fit. When you are satisfied, or if you want to look at some other image line or change the buffer value, exit with 'q'. You will get the query which you can answer with a new buffer value, a new line number, or quit.

Command (quit, buffer , line ): q

Feel free to examine other lines and when you are done finish with quit. At that point the last set of fitting parameters are applied to all the lines in the image and the fits are written as a scattered light image.

Because all of the image lines are fit independently but the scattered light is likely to be a more smoothly varying function, the next step is to fit a smoothing function along the dispersion to all the lines in the scattered light image created by the previous step. Again the icfit tool is used to examine and define the fitting/smoothing parameters. Exit with 'q' and then you may select other columns to look at or quit to apply the last set of fitting parameters to all the columns. Then the fitted and smoothed scattered light image is subtracted from the flat field image.


Command (quit, column ): q

Flat Field and Throughput Corrections

The next step is to create a flat field and throughput correction. Because the flat field consists of observations of a flat field source through the fibers a direct division of the two dimensional flat field image is not desirable. Instead, the approach is to create a one-dimensional flat field spectrum which is divided into the object spectra after extraction. This is equivalent to flat fielding the individual pixels and then extracting them since the fiber positions and profile shape don't change significantly.

Create response function demoflatdemoa1.ms
Extract flat field demoflat
Fit and ratio flat field demoflat

The flat field fibers are extracted and all the fiber flat field spectra are added together to make a master flat field spectrum. You then fit a function to this spectrum to define the flat field source spectrum shape to use in normalizing each flat field fiber spectrum. The order and function should be high enough to represent the overall shape of the flat field but not so high that it follows any fringing which may be present. Fringing is a response effect we wish to remove with the flat field. Again the fitting is done with the icfit commands. These commands include several ways to look at the fit. Try the 'k' key to look at the ratio of the data and fit. To exit type 'q'.

[Output fter 'k' key]

The fitted function to the master flat field spectrum is then divided into each individual flat field fiber spectrum. The mean of all the pixels is used to normalize the collection of flat field spectra to unit mean. If a sky flat is available it will be extracted, flat fielded, and the total counts through each fiber used to adjust the mean of each flat field fiber so that the total sky flat counts would all come out the same. In the absence of a sky flat, such as in this exercise, the flat field fibers are assumed to also provide the throughput and the mean normalized signal through each flat field fiber is computed and printed. The values are the fiber throughput values. You could monitor these between different nights or configurations to see if a fiber changes radically or if the signal is very low one might decide to ignore it.

Create the normalized response demoflatdemoa1.ms
demoflatdemoa1.ms -> demoflatdemoa1.ms  using bzero: 0.  and bscale: 1.000001
    mean: 1.000001  median: 1.053185  mode: 1.27325      upper: INDEF  lower: INDEF
Average aperture response:
1.  1.150906
2.  0.4517221
3.  1.250312
4.  1.286971
5.  1.271164
6.  0.6814409
7.  1.164362
8.  0.7500941
9.  1.009004
10.  1.053699
11.  0.930317

After the one dimensional normalized, throughput corrected, flat field has been created all subsequent extractions will be automatically divided by this flat field.

Dispersion Calibration

The dohydra task now extracts the first arc spectrum specified in the arc spectra list. This will be the reference arc against which all arcs will be automatically reidentified using the same set of arc lines and dispersion function parameters. The middle fiber of this master arc is displayed (with the identify task) and you must now identify a few lines to set the basic dispersion scale against which other lines from the line list may be found.

Extract arc reference image demoarc
Determine dispersion solution for demoarc

In this data mark the first line on the left (as shown in the example output) by placing the cursor over it and typing 'm'. You will be shown the measured pixel coordinate and some other values and queried for the wavelength of this line. In this case enter 5852. The task will then look in the line list for the exact value, mark the line with a tick, and the status line will show the line identification. We need two lines to define an initial dispersion so mark the last line on the right with 'm' and respond with a wavelength of 7281. Since this data is fairly linear we can now type 'l' to have the task automatically find the lines in the line list using the linear dispersion function derived from the two lines we have identified.

[Output after 'l' key]

The dispersion function fitting is done with the icfit fitting tool. To look at the fit and modify it type 'f'. Another useful view of the data in this tool, particularly for dispersion function fitting which to first order is linear, is one in which the linear trend, defined by the two endpoints of the fit, is subtracted to allow us to look at the non-linear component of the function. To look at this plot type 'l'.

[Output after 'l' key]

We see a bad point automatically rejected. At the top we see the RMS (in Angstroms) which is reasonable. You can adjust the fit if you wish and then exit with 'q'. The original spectrum plotted using the dispersion function is shown again. If we are satisfied with this dispersion function we quit with 'q'.

The other fibers in the master arc reference are now reidentified using the set of lines from the middle fiber and new dispersion functions are fit using the same function fitting parameters. Since this can generally be done automatically only a statistical summary of this step is shown. This includes how many lines were found, shifts, and the RMS (root mean square) of the fit. In order to allow intervening in case a fit is poor, usually determined by a significant increase in the RMS, after each fit a query is given to allow you to enter the interactive line fitting. You can say yes or no for each fiber. If you trust the algorithm you can say NO to go ahead with all fibers automatically. Conversely an answer of YES will automatically go to the interactive fitting for all remaining fibers.

REIDENTIFY: NOAO/IRAF IRAFX valdes@puppis Thu 20:50:17 20-Oct-94
  Reference image = demoarcdemoa1.ms, New image = demoarcdemoa1.ms, Refit = yes
          Image Data    Found     Fit Pix Shift  User Shift  Z Shift      RMS
demoarcdemoa1.ms - Ap 41 26/27   26/26    0.00619      0.0386  5.63E-6    0.299
Fit dispersion function interactively? (no|yes|NO|YES) (yes): N
demoarcdemoa1.ms - Ap 41 26/27   26/26    0.00619      0.0386  5.63E-6    0.299
demoarcdemoa1.ms - Ap 39 27/27   26/27     0.0112      0.0701  1.10E-5    0.289
demoarcdemoa1.ms - Ap 38 26/27   26/26    0.00501      0.0314  4.69E-6    0.284
demoarcdemoa1.ms - Ap 37 26/27   26/26    0.00993      0.0616  9.40E-6    0.276
demoarcdemoa1.ms - Ap 36 27/27   26/27   -7.06E-4    -0.00309  -5.0E-7    0.339
demoarcdemoa1.ms - Ap 43 26/27   26/26     0.0116      0.0721  1.10E-5    0.323
demoarcdemoa1.ms - Ap 44 26/27   26/26   -0.00606     -0.0364  -5.2E-6    0.292
demoarcdemoa1.ms - Ap 45 26/27   26/26     0.0113      0.0702  1.12E-5    0.331
demoarcdemoa1.ms - Ap 47 26/27   26/26   -5.21E-4    -0.00239  1.29E-7    0.334
demoarcdemoa1.ms - Ap 48 26/27   26/26    0.00596      0.0374  5.98E-6    0.285

The master arc is also used to set the final linear dispersion scale to be applied to all data. The endpoints of the dispersion functions from all the fibers are evaluated and the minimum and maximum values are selected. From these values a dispersion per pixel, based on the original number of dispersion pixels, is computed. These default linear dispersion parameters are then displayed. You have the option to modify any or all of these until you have a desired set. If the exact values used don't matter you can simply say no to the prompt. After this step all extracted spectra will be automatically resampled to this dispersion scale.

Dispersion correct demoarc
demoarcdemoa1.ms: w1 = 5786.142248358455, w2 = 7351.628227406043, dw = 6.139160702147401, nw = 256
  Change wavelength coordinate assignments? (yes|no|NO): n

Extraction of Object Spectra

Having now set the flat field and dispersion calibrations, dohydra will subtract the scattered light, extract, and calibrate each object spectrum. The first step is scattered light subtraction. You can do this interactively for each image by answering yes or have the scattered light subtraction be done noninteractively using the last set of parameters by answering no (or NO if you have a list of spectra).
Subtract scattered light from demoobj
Fit scattered light for demoobjnoscat interactively?  (yes): n
Smooth the scattered light for demoobjnoscat interactively?  (yes): n

The extraction and dispersion calibration operations are noninteractive.

Extract object spectrum demoobj
Assign arc spectra for demoobj
[demoobj] refspec1='demoarc'
Dispersion correct demoobj
demoobj.ms.imh: REFSPEC1 = 'demoarcdemoa1.ms 1.'
demoobj.ms.imh: w1 = 5786.142, w2 = 7351.628, dw = 6.139162, nw = 256

The sky subtraction is done by collecting all the sky fibers, as identified by the beam numbers set in the aperture identification file, and combining them into a master sky spectrum. Because sky fibers are generally randomly placed there is a chance some of the sky fibers will actually have light from an object in them. Dohydra allows you to overplot all the sky spectra and delete undesirable ones with the 'd' key. There are other options from this specplot task which might be useful to spread the spectra out.

Sky subtract demoobj:  skybeams=0
Edit the sky spectra? (yes): 

Sky rejection option (none|minmax|avsigclip) (avsigclip): 

When you are satisfied quit with 'q'. You will then be given a choice of options for combining the spectra with rejection of bad pixels such as cosmic rays. The default avsigclip should generally be fine. After combining the master sky will be subtracted from each fiber. The master sky spectrum may be saved (which is the case in this exercise) and the individual sky fibers may be removed (which is not the case in this exercise). It may be useful to save the individual sky fibers which are subtracted by the master sky to look at the residuals.

One of the options when you run dohydra is to allow you to look at each spectrum after it has been extracted using the general spectrum examination task splot. In this interactive task you can step through each fiber with ')'. In the example output below are what the first fiber (an object) and the second fiber (a sky) look like.

demoobj.ms.imh:
Splot spectrum? (no|yes|NO|YES) (yes): 
Image line/aperture to plot (0:) (1):
[Output of line 1]

[output after ')' key]

Additional Extractions

Unless the redo parameter is set to yes, any new object spectra specified in additional executions will be processed largely automaticly and the calibrations steps will not be done. Also any spectra which have already been processed will be ignored. For instance try

hy> dohydra redo-
List of object spectra (demoobj): 

You can also extract the data from demostd by

hy> dohydra demostd redo-

That is all for this tutorial exercise. You may run through it again, look at the logfile with page, and look at any of the extracted spectra with splot.

References

This task has an on-line help page and a typeset Postscript version as noted earlier.

There are on-line help pages for the various tasks used by the dohydra script. These tasks are: icfit, gtools, apedit, aptrace, apsum, apscatter, identify, reidentify, splot, specplot.


Francisco Valdes
November, 1994