Re: Data Cleaning

["Thomas Droege" <tdroege2@earthlink.net>]

Arne,

I don't understand what you are asking below or how it is relevant?  I
believe that Michael has programmed ensemble photometry as you describe,
but he can answer that, or you can look at the pipeline software.

The thing I am addressing is that  (would you believe?)  the data is not
perfect.  Some bad frames sneak through the data reduction process.  This
results in some star measurements that probably have larger errors than is
normal;  My discussion has been about how to mark these so that the user
has the option of including them or discarding frames that are not of the
best quality.   Odd points which can be visually identified as spurious
throw off automatic variable star detection programs.  So I am working to
find ways to ease the detection problem.  

The frames vary from just a little bad to very bad.  The very bad ones are
mostly eliminated by the software.  But what about slowly increasing or
decreasing haze?  At what point do you shut down or turn on?  I solve this
by turning on every night that might have some clear sky.  The pipeline
throws out frames where the sky brightness is above some limit (indicating
haze) and the irregularity of the sky (indicating clouds) fails some other
cut.  There are other tests.  All conditions here are imperfect to some
degree, so if I waited for perfect sky I would take no data.  I am very
anxious to compare my data to ARNE and ROB to see how much the suburban
conditions affect the data.  

Why aren't all frames perfect you might ask?  There are 100,000 or so of
them so far as we discussed earlier.  They do not all get a visual
inspection.  A camera can frost over.  If it is not enough to trigger the
software that looks for irregularities in the frame, then some bad
measurements from that frame can get through.  Since this can be a quite
small spot, it can throw off the ensemble photometry.  A few dozen or a few
hundred measurements might be in error.  I could as well said a bug decided
to sit on the lens.  But I have never actually seen this.  The wind can
blow and knock the focus off.  I am learning new things all the time that
can cause problems.  So there has to be an automatic way to mark images
that may have a problem.  That is what I am working on. 

> that the logical thing to do for those stars that are obvious
> variables in the Mark IV data set is to forget about the frame-to-frame
> zero point determination, which makes mistakes, and just do local
> calibration for those stars using such an ensemble approach.  Has
> this been done?  Get decent light curves and worry about absolute
> photometry later.

OK on one night we do this and get one measurement of a star.  On another
night we do this and get a different measurement.  OK we are just talking
about degree here.  This is guaranteed.  Measurements differ.  What I see
is an error limit larger than I expected.  The limit is clearly above
photon statistics for the brighter stars.  I think I now understand what is
going on.  I am really interested in seeing a sigma vs mag plot for a large
number of measurements of a large number of stars taken with ARNE to see if
you are seeing what I see. (And what Pojmanski apparently also sees if I
correctly interpret the curve in his paper.)

I think I really don't understand what you are getting at above.  We are
presently scanning all the sky that crosses the meridian on an evening.  We
measure each field once.  We are thus comparing a measurement taken one day
with another measurement taken another day.  The stars can be in different
positions in the field on the two days.  We search for variables by looking
at this data.  You seem to imply that we are following a star that we
suspect to be variable.  We are not doing that.  That is something that
could be done, but that is not the current strategy.  It is a planned
future strategy, particularly for the Mark V.  

It is one thing to spend the night making measurements on one or a small
number of stars with a large long focal length telescope.  It is quite
another to measure frame after frame, night after night with a small short
focal length telescope.  I think a very large fraction of the CCD
experience has been with detectors on large (long focal length) telescopes.
(OK, this will produce a bunch of examples where you and others have done
this.  I said "large fraction".  I bet it is true.)  Long focal length
telescopes would seem to have a different set of problems from the short
focal length telescopes which have very large (in terms of arc seconds)
pixels.  There are different problems and I have not seen much discussion
of them.  I am patiently waiting to hear of your experience with ARNE
taking such data.  

Tom Droege




> [Original Message]
> From: Arne Henden <aah@nofs.navy.mil>
> To: <tdroege2@earthlink.net>
> Cc: tass <tass@listserv.wwa.com>
> Date: 11/3/2003 7:16:33 PM
> Subject: Re: Data Cleaning
>
> I may have missed something along the way.  If you remember way
> back when, I posted software for the Mark III system that would
> do inhomogeneous ensemble photometry for variables.  It would seem
> that the logical thing to do for those stars that are obvious
> variables in the Mark IV dataset is to forget about the frame-to-frame
> zeropoint determination, which makes mistakes, and just do local
> calibration for those stars using such an ensemble approach.  Has
> this been done?  Get decent light curves and worry about absolute
> photometry later.
> Arne
>
> Thomas Droege wrote:
> > The weather looks awful for the near future so I am trying schemes to
clean
> > the data.
> > 
> > It appears that it is possible to improve the data by cleaning it by a
> > relatively simple and unbiased (I hope) cut.
> > 
> > The result is not that the data looks much better on a sigma mag plot,
but
> > that it is easier to pick out the variables if there are not so many
> > "outliers".
> > 
> > Progress is being made.  The present plan puts a number in the V flag
> > position that contains the fraction of "bad" points in the image that
> > contains the particular measurement.  How can one tell a bad point? 
Well
> > it is not possible to detect a single bad point.  One can only pick out
> > points that are away from the norm for a particular star.  One expects a
> > small fraction of points away from the mean in an image because some
stars
> > are variable.  However images with a large fraction of their
measurements
> > away from the mean are probably "bad" while images with a small fraction
> > are probably "good".  Putting the fraction of bad points in all the data
> > lines allows a user of the data to set his own cut.  
> > 
> > Stay tuned, boys and girls, for the first time I am making progress in
> > "cleaning" the data.
> > 
> > Tom Droege
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
>