Re: Throwing away more data ...

[Tom Droege <tdroege2@earthlink.net>]

Andrew,

As I point out in TN-90, all this data was reduced using sky flats.  I now 
understand that this produces an incorrect result due to the gradient in 
the sky.  I am not sure that it is possible to unfold this error once made 
in the data reduction.

There is hope for the future.  I have 10,000 or so images to reduce once we 
get the pipeline polished up.  By the time Andrew finishes his 
spring/summer/fall job, I should have a nice data set to keep him occupied 
for the winter.  I should also be able to go back and reprocess the earlier 
data.  Then we will have a really large data set to study.

Tom Droege

At 02:16 AM 3/15/03 +0000, you wrote:
>After I had finished the work on Tom's "Sorted"
>data (TN94) I went back and ran the code on Data
>Set 23. Results were dreadful: large numbers of
>obviously spurious "variables" all with the same
>time variation, seen on the same set of images.
>More images need to be thrown away!
>
>Recap: I am fitting an overall spatial polynomial
>correction (4th order) plus individual lower order
>corrections for each image; 1st order, i.e. just
>linear gradients, for TN94. The fit is iterative
>starting with slack tolerances. At each stage,
>measurements that are within tolerance (that is,
>eliminating variables and "bad" measurements) are
>used to fit a new set  of corrections and images with
>large scatter from the new fit are rejected. This process,
>one hopes, converges to give a list of acceptable
>images, each with photometric corrections. As a final
>step, all the data are recalculated using the corrections,
>Welch-Stetson statistics computed and a list of "variable"
>stars is the end product.
>
>Analysed in this way, Data Set 23 gave poor results.
>I got some improvement by going to 2nd order for the
>individual image fits - even though I flatly stated in
>TN 94 that there isn't enough data to pin down 1st
>order let alone 2nd order. This improved fit claimed
>94 variables with W-S > 2.0. Tedious examination showed
>2 probable variables
>2 possibles
>5 maybes
>and nearly 80 from one region of the sky which turned out
>all to involve the same set of images. The amount of
>variation found is enough to downgrade one's confidence
>in the "variables" claimed as "probable" or "possible".
>
>So I went away and played with the code. Adding a further
>test to eliminate images giving large correction coefficients
>as well as those with large scatter helps. Cutting images
>with corrections more than 0.05 magnitudes relative to
>the mean cuts the number of claimed variables from 94 to 26.
>Encouragingly, both "probables", both "possibles" and
>several "maybes" survive this process.
>
>The process is rather draconian:
>
>DS23
>1470 images
>   27 not enough data to fit
>  644 high scatter
>  146 coefficients > 0.05 magnitudes
>leaving 533 images to process
>The first 619 consecutive images are rejected ...
>
>Going back to Tom's "Sorted" 5 month set
>4992 images
>  184 not enough data to fit
>1638 high scatter
>1520 coefficients > 0.05 magnitudes
>leaving 1650 to process
>(The top 7 out of 231 with W-S > 2.0 are, I think,
>real variables! Number 8 shows large but implausible
>magnitude shifts. Things are not perfect.)
>
>The images that are thrown away presumably suffer
>from clouds, haze etc. One could probably get rid
>of a lot of them, together with some more which should
>have been thrown out but managed to get through, by
>using the sky background.
>
>Andrew Bennett, Avondale Vineyard