Re: minor feature enhancement to Mark IV pipeline

[Tom Droege <tdroege2@earthlink.net>]

Michael R and all,

Michael kindly made a fix to his pipeline.  Thank you Michael.  I have 
tested it and everything worked as planned.  I used a data set where I had 
mistakenly stored different numbers of V and I frames.  The frames faded 
into saturation and near dawn.

Not only did the program not die as before, but it did the right thing with 
the different numbers of V and I frames.  (Well, I think it did, I did not 
check that it did not match different V and I frames.)

I was amazed that the magnitudes reported did not change noticeably as dawn 
came.  So far I have just scanned a few points.  I will leave it to someone 
else to study this.  The flags increased to the point that almost every 
star was flagged.  But there was no appreciable change in magnitude 
reported.  A good sign, I think.

To those who have not looked at much raw data, as dawn approaches, more and 
more stars become saturated.  Pretty soon, they all grow two tails.  You 
can think of it as the stars sticking up above a base.  As it gets lighter, 
the base moves up.  Pretty soon more and more stars are hitting their heads 
against the ceiling.

This is encouraging.  Dawn is not a problem.  Now we have to worry about 
the effects of small clouds, contrails, and the like.   These will probably 
cause dispersion.

Note that we can detect some of these problems by watching the mean value 
of the image and throwing out images where the mean value 
increases.  Another possibility is to look for differential mean value 
changes.  I.e look for a difference say between the East and West half of 
an image.  (Here the clouds mostly move from West to East.)  This is a 
project for someone that wants to work on it.

Tom Droege

At 02:53 PM 5/22/02 -0400, you wrote:

>   Tom Droege recently explained to me that the pipeline will halt
>in the middle of its work when it encounters a very cloudy image,
>or an image taken as the sky brightens at dawn.  He asked if there
>was some way to keep the pipeline going, despite such bad images,
>so that all the good images could be reduced to completion.
>
>   I have found at least some of the conditions which can cause
>the pipeline to halt, and modified the main driver program so that
>it handles the errors and attempts to continue instead of halting.
>If people want to try out the fix, please do.  There was only
>one file which changed: "markiv_driver.tcl".  You can download a
>new version at
>
>          http://spiff.rit.edu/tass/pipeline/markiv_driver.tcl
>
>Or, if you're compulsive, you can grab a tar file for the entire
>pipeline and re-install:
>
>          http://spiff.rit.edu/tass/pipeline/pipeline.tar.gz
>
>   Feedback is _most_ welcome.  I'm not _sure_ that this will deal
>with the problems Tom and others were having, but it might.
>If problems persist, I can try to fix them, too.
>
>                                       Michael Richmond