Re: Photometry

[Arne Henden <aah@nofs.navy.mil>]

Ok, before everyone thinks I am being negative, I'd like to
give a few words regarding my own personal CCD observing.  I save
both raw and processed images; raw on tape and processed on CD.
Part of this is paranoia; using different media guards against
a drive failure.
   I do my photometric extraction and any other project that might
be in progress (such as stacking GRB afterglow images), and then
delete the processed and raw frames from hard disk.  This works
because my data rate is not high; if I am using one of the big
CCDs, I tend to be taking longer exposures to compensate.
   Why do I save raw frames?  I've run into several circumstances
where I had to go back to the raw data.
   - a processed CD was damaged or for some reason did not have
     all of the frames copied (an example: operator error that
     forgot that the CD was being written in ISO9660 8x3 naming
     convention, and the software only saved one of multiple files
     that had the first 8x3 identical).  Making two copies of
     data is always a good idea anyway, keeping the second copy
     in a separate location.
   - later examination showing the flatfielding was done wrong.
     A dust speck fell on the CCD window through an observing night,
     so flats taken at the beginning of the night were not correct
     for the entire night.  A whole two-year span had to be reprocessed
     because we discovered a scattered light problem that was present
     in the flats as well as the data frames; fun to correct.  Even
     inspection of the data during the subsequent day can't catch all
     of these errors; it often takes comparison with other datasets.
   - a bias frame where a cosmic ray snuck through, and fell right
     on top of an important star.
   - dewar warmed up, and I needed to redo the overscan correction
     to account for the time-variant dark current during readout.
   - twilight flats were taken with a nearby moon, causing a gradient
     in the images that was only discovered after comparing several
     nights of data.
While it is theoretically possible to take processed frames, multiply
by the flatfield, add dark/bias, etc. to recover a raw frame, which
can then be rerun through the pipeline with new calibration data,
it is *much* simpler to do this with the original raw frames.  If
one is saving the master flats anyway, then providing a processed
copy of any raw image should be easy to do on the fly.  Or, if
Tom's assumption is correct, you should be able to look at the raw
images themselves since flatfielding is not important.
   I go back to at least processed images quite often.  My software
is automated, but it fails in crowded regions.  When I find a new
'variable' star, it is often because of pipeline problems, obvious
when looking at some image.  I may want to stack several nights of
data to go faint and look for an underlying galaxy or quiescent CV.
Someone may point out that a variable is actually a close double,
and I have to go back to the original data and psf-fit it to split
the double.
   Saving that, our philosophy on our 1.3m mosaic camera has been
stated several times in meetings: save only starlists and discard
the images.  This is because a single exposure takes 100MB to store.
When the camera was first being designed, this was a lot of space,
but now we have TB disk farms everywhere.  Archival is still a problem,
though; not many images fit on a DVD.  So the archive-don't archive
question is still here and unanswered.  My guess is that people will
archive personal science frames and not archive survey frames, where
the data can be retaken.  We'll see.
   For Tom's case, the problem is manpower.  It takes time to archive
a night onto multiple disks, especially if two copies of every frame
need to be written.  I fully understand the reluctance to continue
the process!  I am currently only saving raw images from ARNE, and
using lossless compression so that only two CDs/night need to be
scribbled.
Arne