Cleaning Out Clouds

[Tom Droege <tdroege2@earthlink.net>]

I have spent the last several weeks mucking about with the pipeline 
attempting to find the cause of the scatter at bright magnitudes.   I am 
fearless, so I have tried changing everything I could get my hands on.  I 
have also checked a few things.  I have been running the computer full time 
trying things.

1)  I did two different linearity checks.  The chip I checked is linear to 
about 43,000 ADU.  Somewhat more than the 80,000 e-  (32000 ADU) that is 
advertised.  Indications are that the newer chips are not so good.  So I 
will set the linearity limit at 32,000 counts up from the present 
20,000.  I will get out a tech note on this.

2) I tried using different sets of reference stars.  Using a smaller more 
accurate set fools you at first.  It makes the mag vs mag sigma plot look 
tighter.  But it does this by increasing the error of the fainter 
stars.  So the largest catalog gives the best distribution.

3)  I found a light leak in the flat field box at IR.  I thought this would 
explain everything.  But a new flat field made with a sealed up box in 
darkness did not make an improvement.  The old one was as good as the 
new.  (It was made in the tower during daytime with the doors 
closed.  Pretty dark.)  Making flats from a large number of sky images 
seemed to be a little better than the flat box.  This as viewed by looking 
at the cleaned .fits images from ccdproc.  But it does not seem to be a 
large improvement.

4)  I tried different apertures.  Again, this gave a misleading 
result.  Larger apertures tightened up the grouping of the mag vs mag sigma 
plot but again did this by increasing the error at the faint end.  An 
aperture of 4 is near optimum.  There is the possibility that 3 will be 
slightly better.

5)  Yet to be checked, the scatter as a function of location in the 
sky.  The idea is to make sky flats from a narrow range of telescope 
pointing.  Another thing yet to be checked is the quality of the darks.  I 
give little hope that either of these things will do anything.

OK, now the thing that was successful.

Get rid of clouds.

To do this, I looked at the dark subtracted and flatfielded images as 
produced by ccdproc.  I looked with DS9 in the line analyze horizontal and 
vertical cut mode.  I just rejected any frame where the base line varied 
more than the noise level.  This eliminated about half of one data set but 
significantly cleaned up the mag vs mag sigma plot.  This with images that 
really looked pretty good when you looked at them.  I had previously tried 
a cruder scheme to reject frames with no significant result.  I soon 
realized that there was nothing easy to do with the raw images.  You needed 
to dark subtract and flat field them.  So why not use the result of the 
pipeline?

OK, here is what I propose to do:

1)  Run the pipeline through the ccdproc step using a light box flat.
2)  Look at the resulting dark subtracted and flat fielded .fits images 
with some program.  Discard the images where there is any gradient.  This 
with some parameters to set so that I can tune the process.
3) Restart the pipeline from the make flat step and use the remaining 
images to make a sky flat if there are enough left, otherwise use the light 
box flat.  (Which is not so bad.)

Does anyone want to write a program that does 2?  I can do the rest.  It 
appears to be worth doing.  One just looks in a specified directory and 
uses your favorite program to somehow look at each image and determine if 
they have any significant gradient to some specification, then delete the 
.fits files that fail.  When you look at cloudy images, there are white 
smudges that can be in any location.  So you have to cover most of the 
images.  I would think fitting three or four top to bottom and side to side 
lines would do.  Just fit second order? lines and reject images where the 
deviation from flat was greater than some number, like the noise level.  Or 
compute the mean image and some random small areas.  Delete the image if 
the small area means differ from the image mean by more than the noise 
level.  Or some such.  You do it and you get to do something clever.

If no one takes it on I will have a go at it.  Michael has shown me how to 
use the tools in the xvista package, and I can use these (somehow) to do 
2).  But you are letting me write code and that is probably a big mistake.

Comments from experts?

Tom Droege