Re: Quick note on Disk 18a

[Tom Droege <tdroege@veriomail.com>]

Michael and all,

Well, it is a question of what you mean by a "large peak" and a "small" peak.

The specks of the ADC call for "no missing codes" to 15 bits.  They are 
actually better than that.

What you are describing is called "differential linearity".

You did not say where the spike is.  It is common for there to be a problem 
at the large bit change boundaries.  For example in going from 111111 (63) 
to 1000000 (64).  The problem can be in either direction.

Suppose that we look at "white" noise.  That is all values in the range are 
equally likely.  If we take 1000 counts in each bin, then we expect to see 
1000 counts in 63 (+/- 32) and 1000 counts in 64.  Suppose we see 2000 in 
63 and 0 in 64.  This looks awful.   It also meets the specification.  In 
fact it means that we are in error by one count in the measurement for half 
of the data at this point.  But what does it mean to our data?

It means that some of our stars may be measured in error by one count.  OK, 
in a systematic way.  So we have to consider if that hurts us.  But note 
that our sky noise is typically 50 ADU, so we are looking for effects much 
less than the noise.

Suppose we measured a zillion stars.   Then made a plot of magnitude vs 
number of stars at that magnitude.  This would have a shape due to the 
distribution of stars in the sky.  It would (with enough data) have a fine 
structure that was due to the ADC.  I think such fine structure will be 
smaller than the error.  But this needs to be given proper analysis.

Suppose we consider a bunch of buckets spread out over a line catching 
random rain drops.  Differential linearity is a measure of the diameter of 
each bucket.  In our case, we are allowed a bucket of zero width.  Integral 
linearity is a measure of the uniform spacing of the buckets.  In our case, 
this is specified to be +/- four ADU over the full range.  Thus bucket 
23,567 should be within 16 positions of its specified position.  This means 
that all the buckets in between 1 and 23,567 have to be stretched or 
squished so that no two adjacent buckets are of zero width or more that 
twice their average diameter.  One can have regions of wide and narrow 
buckets as long as no bucket is off by more than 16 positions.  But note 
that this is better than 0.03%.   We thus measure any block of photons to 
at least one part in 4,000.  One tends to be able to calibrate out the 
Integral Linearity, so it is really better than this.

>   Perhaps some sticky bits in the ADC?

Yep, all ADCs have sticky bits.  Well, not all, there is the Wilkensen 
type, but it is slow.  The system noise actually helps a lot to mask this 
generic problem.   And as I say, this one seems to meet specs, and I think 
is good enough for our system.

>   It helps to combine the data into bins at least 8 values wide,
>but that leaves the every-64'th-pixel peaks.  I can't make the bins
>64 values wide without losing resolution -- I think.

I think this is the wrong thing to do.  You are introducing a ruler with 
graduations of 64 because the error in position of the lines is 
one.  (Well, actually a few)

If this bothers you, (I think it should not) then try using non binary 
binning.  Using binary bins does little good for a binary problem.  Say bin 
by 3s or 5s.   Better a weighted average over 5 bins.

I would be happy to discuss this further for anyone that is bothered by the 
errors in the system.  As a system designer, I have some options that can 
trade off problems.  Perhaps we can make the system better.  There is also 
a slightly better grade ADC.  I bit better in both Integral Linearity and 
Differential Linearity.  I think we could not see the difference.  But we 
could discuss this.

I am delighted to hear of the progress.  I can't wait to get hold of some 
star lists.  At least I think I can't.  In actual fact it will be a couple 
of years before I will be pondering data.  Just too many things to fix.

Tom Droege

At 12:59 PM 5/20/01 -0400, you wrote:

>   I've been working on a new distribution kit for my Mark IV
>reduction pipeline.  I grabbed the first few images off Data Disk 18a
>to test the code, and noticed something odd.
>
>   If one makes a histogram of the pixel values in the raw images
>on Disk 18a, one finds
>
>        - the distribution of pixel values in I-band images shows
>              a small peak every 8'th value, and a very large
>              peak every 64'th value
>
>        - the distribution of pixel values in V-band images shows
>              a small peak every 2'nd or 4'th value; the amplitude of
>              the variations is smaller than those in the I-band images
>
>   Perhaps some sticky bits in the ADC?
>
>   I have to be careful here -- the pipeline uses an XVista program
>called "sky" to calculate the sky value in each image.  That program
>computes a single "sky" value for the frame by fitting a gaussian
>to the distribution of pixel values.  The results of this fit may
>not be well defined when there are big jumps every N'th pixel.
>
>   It helps to combine the data into bins at least 8 values wide,
>but that leaves the every-64'th-pixel peaks.  I can't make the bins
>64 values wide without losing resolution -- I think.
>
>   Anyway, just a warning to other data munchers.
>
>   On the bright side, when I ran the pipeline on a set of 8 images
>in the middle of disk 18a -- which did show evidence for clouds --
>it successfully matched up every one with stars from the Tycho catalog
>and found a decent astrometric solution.  Rah!
>
>   I discovered that I need at least 45 or so reference stars per
>field in order to achieve success (with a 3rd-order polynomial distortion
>model).  I wasn't getting that many Tycho stars when I cut the catalog
>at sigma(Vt) <= 0.05 mag, so I relaxed the requirement to
>
>        sigma(Vt) <= 0.07 mag and sigma(Bt) <= 0.07 mag
>
>This gives me enough reference stars in every field I've checked
>so far.
>
>                                       Michael Richmond