Throwing away more data ...

[Andrew Bennett <andrew.bennett@ns.sympatico.ca>]

After I had finished the work on Tom's "Sorted"
data (TN94) I went back and ran the code on Data
Set 23. Results were dreadful: large numbers of
obviously spurious "variables" all with the same
time variation, seen on the same set of images.
More images need to be thrown away!

Recap: I am fitting an overall spatial polynomial
correction (4th order) plus individual lower order
corrections for each image; 1st order, i.e. just
linear gradients, for TN94. The fit is iterative
starting with slack tolerances. At each stage,
measurements that are within tolerance (that is, 
eliminating variables and "bad" measurements) are 
used to fit a new set  of corrections and images with 
large scatter from the new fit are rejected. This process,
one hopes, converges to give a list of acceptable
images, each with photometric corrections. As a final
step, all the data are recalculated using the corrections,
Welch-Stetson statistics computed and a list of "variable"
stars is the end product.

Analysed in this way, Data Set 23 gave poor results.
I got some improvement by going to 2nd order for the
individual image fits - even though I flatly stated in
TN 94 that there isn't enough data to pin down 1st
order let alone 2nd order. This improved fit claimed
94 variables with W-S > 2.0. Tedious examination showed
2 probable variables
2 possibles
5 maybes
and nearly 80 from one region of the sky which turned out
all to involve the same set of images. The amount of
variation found is enough to downgrade one's confidence
in the "variables" claimed as "probable" or "possible".

So I went away and played with the code. Adding a further
test to eliminate images giving large correction coefficients 
as well as those with large scatter helps. Cutting images
with corrections more than 0.05 magnitudes relative to
the mean cuts the number of claimed variables from 94 to 26.
Encouragingly, both "probables", both "possibles" and
several "maybes" survive this process.

The process is rather draconian:

DS23
1470 images
  27 not enough data to fit
 644 high scatter
 146 coefficients > 0.05 magnitudes
leaving 533 images to process
The first 619 consecutive images are rejected ...

Going back to Tom's "Sorted" 5 month set
4992 images
 184 not enough data to fit
1638 high scatter
1520 coefficients > 0.05 magnitudes
leaving 1650 to process
(The top 7 out of 231 with W-S > 2.0 are, I think,
real variables! Number 8 shows large but implausible
magnitude shifts. Things are not perfect.)

The images that are thrown away presumably suffer
from clouds, haze etc. One could probably get rid
of a lot of them, together with some more which should
have been thrown out but managed to get through, by
using the sky background.

Andrew Bennett, Avondale Vineyard