Re: General questions

[Tom Droege <tdroege@veriomail.com>]

Rob and all,

(Hmmm!  just found this was not sent when I thought it was.  I apologize if 
it actually went out.)

Life is not perfect, and the data sets are not perfect.  The first thing to 
do is to read the .txt file on each disk to see if there is anything funny 
about the data set.  This will tell you thing like there was a sticking 
shutter on one or the other channel, or what sequence I was using to take 
the data.  It is a bit risky to just run through data without looking at 
part of it. I know it is easier to just start up a program and have it 
grind through that 40 GB of data, but whenever anything is out of order you 
will want to look at it.

Generally there are problems at the beginning when things are cooling down, 
and at the end when dawn comes.  I also tended to fill up disks on the 
theory that it was good to have some bad data to look at.  This gives 
eamples of saturated frames at dawn, clouds, and the like.  In general I 
tried to comment on these out of order frames in the .txt file.

Looking at my log book, not all the data was taken with the dark four sky 
sequence.  For example, 1814 was taken with the dark light sequence, and 
with 200 second exposures.  Looks like the first runs taken with a dark 
followed by four sky started with the run on JD 1820.  The log shows the 
runs on JD1816 and 1817 were taken with a dark followed by 9 sky.  All this 
information should be in the .txt files.  At some point I made a table of 
all the dark frames.  If something looks dark and I did not mark it as such 
in the .txt file, then there is always the possibility that I made a typing 
mistake.

A good cold dark frame will have a sigma in the 10-20 range in ADU.  The 
sigma of space between obvious stars (there is always some star content) 
will have a sigma in the 40-80 range for most frames, I think.  But this is 
one of the things that should come out of the Data Analysis Group 
work.  High sigmas for the space between stars generally means 
clouds.  Higher than the usual mean values also probably means a cloud is 
coming by.

If the .txt file says a frame is dark, and it is obviously not dark, then 
it is probably a stuck shutter.  You can tell a stuck shutter since there 
will be streaks caused by the shutter being open while the frame is being 
read out.  Lots of E-W streaks generally means that something is wrong with 
the shutter or the tracking.

There are lots of decisions to be made in processing data.  There are no 
"answers".  Sometimes you will be able to save data that would otherwise be 
lost by using darks or flats from another day.  Sometimes such use will 
compromise the data quality.  I am afraid that if one only uses perfect 
data that one will not get very much.  So there are constant compromises to 
be made.  Reduced data needs to somehow carry with it a "quality" 
measure.  Again, one of the goals of this exercise is to start facing such 
issues.  Later we will combine various cuts of our data for a final 
catalog.  We might end up with several catalogs.  A "photometric" catalog 
which contains all the really good data and is intended as the reference 
magnitude source for the stars.  Another catalog might contain more data 
and will contain relative magnitudes over time.

If you spend much time looking at tenxcat, you will find measurement 
sequences with "outliers".  Are these true measurements and thus 
indications of an eclipsing star?  Are they just bad data points.  One 
would like to be able to look at the data for such points to make some 
value judgement.  An outlier on an otherwise great day might be given more 
weight than one on a cloudy day.

Since there is too much data to look at by hand, our software will have to 
learn to make decisions about data quality.

Tom Droege


At 06:20 PM 12/25/00 -0700, you wrote:

>Tom (and all),
>
>How do you calculate the background (sky?) level?  I could use sps
>to run though the data since it spits out the sky level for each star it
>finds...
>Anyone have thoughts on finding airplanes (streaks).  Thanks to
>Doug, I've seen what an airplane can look like in an image.
>
>I've a question for everyone.  I generated a script which catagorizes
>the images as OBJECT or DARK based on Andrew's suggestion
>and tech note 71, and they don't all match what Tom believed the
>general format to be i.e. dark, 4 objects...  I'd be happy to send the
>zipped file (~50K) of the data I calculated (col 5 mean of each
>image), but here is a synopis:
>
>1803 - 16 Object, 8 Dark, 40 Object, 8 Dark, 8 Object
>1804, 1805 - 4 Object, 2 Dark, 10 Object, 2 Dark...
>1813 - no Darks
>1814 - Dark, Object, Dark, Object...
>1815, 1816, 1817 - Dark, 9 Object...
>1820, 1827, 1829, 1836, 1838, 1839, 1853  - Dark, 4 Object...
>1824 - Dark, 4 Object except for 14 Object in the last set
>1828 - h3 Dark, 4 Object, h4 series has no darks
>1837, 1851 - Missing 1st dark, otherwise Dark, 4 Object
>
>Could CCD temp cause some of the missing darks - like on 1837,
>1851?
>
>'Nother question in light of my catagorization.  Will creating flat field
>images without subtracting dark frames (1813) still be "valid"?
>Should any different processing be done?  I could use a dark from
>another night in a pinch.
>
>1828 - Did the shutter get stuck, or could it of been a CCD temp
>problem?  I didn't run through the logfiles to try to corelate temps the
>problems, but probably will after thinking of it just now.
>
>Thanks,
>Rob