Re: Bits and pieces

[Arne Henden <aah@nofs.navy.mil>]

Tom:
   My basic philosophy has not changed, for my personal data.
I keep both raw and processed frames, and generally refer to
the processed frames since I know and understand my system.
   The problem I saw was the discussion over the past few weeks
regarding how the database was going to be used.  Projects mentioned
included flares on Mira variables, studying the stars with one
high point, looking for optical transients, etc.  These events
all depend on very few data points, and to convince someone
else that they are real requires convincing the critics that there
were no flaws in the original images nor in their processing.
If you only save the processed frames, you cannot say for sure
that whatever event was not a product of the processing.  You
also lose some engineering data, such as what is going on in
the dark columns as a function of row number.  That
is why I say the best method, if only one image type is to be
saved, is to save the raw images and add some program or script
(since Michael says it ain't available in the pipeline) to recreate
the exact copy of the processed frame.
   There are several alternatives here. We discussed these years
ago, but perhaps a repeat is in order.
     - save both raw and processed frames, as you have been doing.
       This solves both the archival problem as well as the programming
       problem, but involves considerable extra effort on Tom's part
       as well as more media, wear/tear on hardware, etc.
     - save only processed frames.  You cannot restore exactly the
       full original data frame if you do this, but perhaps the loss
       is worth it.
     - save only raw frames, but include a program/script that will take
       the nightly master flat/dark frames and give you rapid access
       to an equivalent processed frame.  This is the best method IMHO.
     - save compressed versions of either or both.  If lossy, you lose
       some information, but we ran tests that indicate a factor of
       10 or so compression really doesn't hurt much, especially if
       you only go back on rare occasions.
     - don't save any images, but rely on the extracted starlists.
       You lose the potential of searching for rare events, looking
       at something the starlist creator didn't anticipate (surface
       photometry, psf fitting in crowded fields, or whatever).

I think I have a fair amount of experience at precision photometry,
and can say with considerable confidence that Flats Do Matter.
Maybe you are happy with 0.03-0.05mag errors, but I am not.  Lousy
locations does not matter much when doing differential photometry;
I've seen millimag work from worse areas than Batavia.

> 
> 
>>My basic point is that everyone agrees to save some kind of image.
>>As long as the processing steps are archived somewhere, it should
>>be only a matter of seconds to convert a raw image into a processed
>>image, with absolute fidelity and leaving the original data intact.
> 
> 
>   This is true only if
> 
>        a) we store the master darks and flats used to reduce a night's
>                data together with the raw frames 
> 
Yes, that is my suggestion, along with new fits keywords to indicate
which flat/dark should be used for a given frame.

>        b) we have a separate tool to perform basic image reduction
>                on a single frame, and then analyze the cleaned frame
> 
I was thinking more along the lines of someone who wants to visually
inspect images, rather than reanalyzing images.  For reanalyzing,
I would think you might want to redo an entire night more than just
one or two images.
   Tom has already reprocessed thousands of images, so I understand a
reluctance to change the process.  One could get around this by using
a "logfile" rather than changing the fits headers; the logfile can
indicate what processing files are required for each raw data file.
Tom mentioned way back when that he expected to reprocess the data
a few times, and perhaps he will have to do it yet again after Michael
has had the opportunity to study the results from this massive latest
effort.  It all depends on what you consider "good enough", and I
don't think we are there yet.  Sorry.
Arne