Re: A New Record

["Thomas Droege" <tdroege2@earthlink.net>]

Patrick and all,

I think I did say that I "thought" I saw problems.  What I am repeating in
saying is that I have not yet found a way to identify bad data.

You (I think) are saying "I know bad data when I see it".  The problem is
the same as with pornography.  If you can't define "bad" data then you
can't pass a law against it.  (Make a cut to remove it)

The risk in removing data is that you create a bias in the remaining data. 
I believe that the way we are processing data is bias free.  

There is a lot of information saved now with the data.  I challenge all of
you to figure out better cuts to improve the data quality.  I am more or
less out of ideas.  

You should note that we have gradually improved the apparatus over time. 
Flocked paper is an example.  I am currently adjusting the cameras to
remove a tilt component that changes the image focus top to bottom. 
(Michael take note, this is no doubt the cause of the problem you discussed
in one of your tech notes) These things all contribute a little to the
error, so the errors will get better over time as I understand problems and
make adjustments.  I think most of these things contribute smaller errors
than are caused by the character of the CCD and the statistics.  

Patrick says:

> On the other hand, when you look at the individual data points of a
> star, these bad points do matter, because they distort the light curve.

Yep, that is just life with real data.  However images that contain
"obvious" bad points may also contain "good" points that allow
determination of something about some other star.  So one has to have a
method that makes sense to throw out whole images.  It is not so easy to
do.  So that is the problem.  Work in this area would be appreciated.  

Tom Droege


> [Original Message]
> From: Patrick Wils <patrickwils@yahoo.com>
> To: <tdroege2@earthlink.net>
> Cc: <tass@listserv.wwa.com>
> Date: 10/21/2003 10:30:37 AM
> Subject: Re: A New Record
>
> Hi Tom,
>
> > Some of you seem to think you see "bad" nights in the tass data.  I
> > also
> > "think" I see bad nights.  But I am yet to prove it.  I have done
> > many
> > hours of computation to try to select data that might be better.  
> > 
> > Here is an example of one of the things I did. Let us suppose that
> > there
> > are bad images. Bad images should result in measurements away from
> > the mean
> > for fixed stars. Most stars are fixed, so most stars in the data base
> > are
> > fixed. If we look at all the stars in the data base and select points
> > away
> > from the mean for each star, then if the above theory is true  more
> > of
> > these points should be from "bad" images than from "good" images. Now
> > take
> > all the bad points and sort them by image. Compute the percentage of
> > "bad"
> > points in each image and use this to make a cut on all the data. That
> > is
> > throw out all the images that have a high percentage of bad star
> > measurement by the above definition. Now look at the quality of the
> > result. 
> > 
> > I have done this and no cut I can find improves the quality
> > significantly as measured by a mag, sigma mag plot.  
>
> This means that most of your data is good, and that the bad data does
> not influence the overall quality.  That is, the average magnitudes of
> stars and their precision will not change much by throwing out "bad"
> nights.  
> On the other hand, when you look at the individual data points of a
> star, these bad points do matter, because they distort the light curve.
>  
>
> For NSV 8866 (the plot you sent to the list some time ago with a faint
> Ic point) this faint Ic point deviates nearly 5 sigma from the average
> (of 60 points), while all other points deviate less than 2.5 sigma.  
>
> When you look at stars in the vicinity of NSV 8866, you will see that
> for that particular image, Ic deviates much from the average for a lot
> of stars (whilst V agrees fairly well with other images).  This may be
> a statistical "issue", but still, chances are very small that a large
> number of stars deviate much on the same image, unless you have
> millions of images.  Clouds, fog, or statistical conspiracy may all be
> explanations, but in my opinion the image should not be used for the
> study of individual stars.
>
> Patrick
>
>
> __________________________________
> Do you Yahoo!?
> The New Yahoo! Shopping - with improved product search
> http://shopping.yahoo.com