Re: Data Processing Switch

[Arne Henden <aah@nofs.navy.mil>]

Stupendous Man wrote:
>   Chris wrote:
> 
> 
>>There may be a mis-understanding here.  What I thought Tom
>>proposed to save was a FITS file with un-processed pixels
>>(except for trimming the overscan) and corrected values
>>in the header. If that's true: raw pixels and "corrected"
>>headers then nothing is lost.  Such files should be able
>>to be re-processed.
> 
>  
>   Nope.  Tom's proposal saves images in which the pixels
> have been corrected, too. 
> 
>   Arne wrote:
> 
> 
>>My feeling is to save the raw images and discard the processed
>>images after you have done the star extraction.  That way you
>>save the same amount of data, but can recreate anything.
> 
> 
>   Let me put some words in Tom's mouth, if he'll let me.
> 
>        a) it is a rare occasion that one wants to go back and
>              look at an image
> 
Speak for yourself!  There are many uses for the archived images
beyond simple star extraction, especially when you get close to the
sky limit.  For example, a nova could occur and you may want to coadd
several of the original images to go deeper to look for a precursor
or start of the outburst.

>        b) on almost every such occasion, looking at the processed
>              image (plus associated flat) is just as good as
>              looking at the raw image.  For example, Tom discovered
>              that some strange events were caused by satellite trails;
>              obvious on the processed as well as the raw.
> 
Yes, but "almost as good" is not the same as "exact".  As Rob mentioned,
there may be occasion to reprocess images, look at dark columns if
engineering data is needed to trace problems, etc.  I reprocess
quite often when dealing with transient phenomena; rarely when
dealing with calibrations or repeating stars.  It just depends on
what you want to do with the data.

>        c) the time it takes to look at a processed (or raw) image
>              is, oh, the time to locate the appropriate CD-Rom, 
>              find the right file on that CD-Rom, and display it.
>              If the header is (nearly) correct, then one can move
>              to a given (RA, Dec) in a few seconds
> 
How is that different between processed and raw frames?  Or perhaps
what I should say is that you should be able to archive the
raw frames in just as convenient a fashion as the processed
frames.  As mentioned previously, I'd suggest modifying the header
of the *raw* frames to include proper WCS.  That is the only
processing step I would recommend for the archived images,
though you could, for example, store in the header the names of
the dark/flat frames that might be required to process the given
raw frame at a later date.

>        d) the time it takes to reprocess a raw image is in the hours
> 
Now, this I don't understand.  Why should it take hours to reprocess
an image?  Please explain.

>   If these are accurate reflections of reality, then I would
> keep the processed images.  
> 
>   It is often the case that it takes less work to go and re-observe
> a given part of the sky than to locate old data and interpret it
> properly.  In some situations, such as verifying a "flash", that's
> no good; but in other situations, such as checking the magnitude
> or position of a particular star, it's just fine.  We should not
> forget that we can use the sky as our archive, in some cases.
> 
You cannot tell whether that corrupted datapoint for that important
star was caused by a satellite trail, cosmic ray hit, etc. when
using the "sky as our archive".  Rare events like multiple-year
eclipse periods may only happen once during a given TASS site's
lifetime.  Novae only happen once.  Was that Mira "flare" real or
not?  If you are talking about the run-of-the mill star, I fully
agree that you can reobserve.  Tom has expressed interest in the
rare events, and those you may need original images to prove
or disprove.

My basic point is that everyone agrees to save some kind of image.
As long as the processing steps are archived somewhere, it should
be only a matter of seconds to convert a raw image into a processed
image, with absolute fidelity and leaving the original data intact.
I strongly encourage heading in that direction.  Tom rarely takes
my advice, so I will keep quiet after this posting.
Arne