Re: Bad pixel determination with better software

[Tom Droege <tdroege@veriomail.com>]

Rob and all,

To look for bad pixels, I would use a stack of dark frames from the same 
detector.  I would take as many dark frames as I could get (that is why you 
have so many disks), sort them by mean value in the middle of the frame, 
say x 500-1500, y 500 -1500.  Then I would take the mean of a bunch where 
the variation in mean value was small compared to the sigma of the area.

Now I would take the stack of comparable images - the above process got you 
a stack taken at roughly the same temperature - and take the mean of the 
full frames.  Now you can take the histogram as below.  This should show up 
any "hot" pixels.  Now you have to go through and "find" the hot pixels by 
x-y coordinate.

OK, the experts will probably jump all over me.

Not that this is just the first step to get a dark frame and to identify 
pixels with high dark current.  There are many more problems to come.

I suspect that you will not find enough hot dark pixels to worry 
about.  But it is worth doing the experiment.  However bad they are, they 
get much better with reduced temperature.  The dark current goes down by 
half for a 5 C decrease in temperature.  But it is really much better than 
that.  There are hot pixels that tend to turn off at some temperature.  So 
they go from a high current to near zero as their critical temperature is 
passed.

I do not know of any mechanism that causes change of dark current (at the 
same temperature) over time.  Or for a pixel to be lost.  Well, not quite 
true.  A lot of radiation will do it.  I suppose that one could get really 
unlucky and get clobbered by a cosmic ray.  But I think such problems heal.

For the most part, what you have gotten in the past at any specified 
temperature is what you expect to get in the future.  The drift of the 
electronics should also be small.  I would expect the pedestal (the mean 
value of a dark frame) to change only about 3 counts per C of the ambient 
temperature.  That is the electronics temperature.  There is a thermometer 
that is relatively close to the electronics that are critical.

Exercise for the student:

1) Measure the mean value of a bunch of dark frames.  Use the center area 
as above.
2) Plot the mean value of the above against the VCO temperature.  (Woops, 
we are not recording the VCO temperature.  So this cannot be done. )
3)  Plot the mean value of the above against the CCD temperature.  We have 
been recording this.

OK, there are two (main) things that can make the pedestal vary.
1) Dark Current.  This doubles for each 5C increas in 
temperature.  (roughly).  This can be sorted out from the pedestal.
2) Electronics drift.

Lets take the electronics drift first.  I expect about 3 ADU per C.  This 
is based on a 100 ppm expected typical component value drift and the 25000 
ADU pedestal.  It will depend on how all the errors add up.  While the mean 
value of the pedestal will move around, the sigma should remain 
constant.  It is just the noise from the ADC and other stuff on the printed 
circuit board.  So the mean value of the frames are expected to move around 
with the ambient temperature, but the noise due to this movement will not 
change (much).

Now the dark current.  The dark current changes the pedestal by 
accumulating electrons in the pixel.  The accumulation of electrons causes 
the pedestal to shift by 1/2.2 (roughly) ADU per electron.  The noise (as 
expressed in electrons) should increase as the square root of the number of 
electrons.  So pedestal change caused by electronic drift causes no 
increase in noise, pedestal drift caused by dark current causes the noise 
to increase as the square root of the number of electrons.  OK, lets say we 
saw the mean value of the pedestal to shift during a long dark exposure by 
1000 adu.   This means the mean value of the pixel changed by 2200 
electrons.  The square root of 2200 is 47 electrons.  So we expect a sigma 
of 47 electrons.  (To get this we compute the sigma of all 4 million pixel 
values) But we measure in ADU which is roughly 2.2 electrons.  So we would 
expect the sigma taken over many pixels to increase by 22 ADU.   OK, this 
is a random process.  While the mean pixel has a leakage of 2200 electrons, 
this is *not* 2200 electrons for each and every pixel.  Because it is 
random, they will be distributed.  Most pixels will be within 47 electrons 
of 2200.  But we have a lot of pixels, so some will be 4 or 5 sigma away - 
so it would not be surprising to find one with 2500 or 1900 
electrons.  Further there are other processes going on, so while a 1800 
electron pixel would be rare, one could easily have a 10,000 electron 
pixel. (from a "hot" pixel - one where there was some impurity in the 
silicon which acts as a source of electrons)  The distribution will be 
skewed.  In fact, if you look at dark frames, you will see that at room 
temperature the pixel distribution is quite skewed, but as the temperature 
is lowered the distribution becomes more and my symmetrical.

OK, it is never as easy as this.  There are lots of noise sources and we 
hope we can add them all up in quadrature.  We have a "floor" in this 
system somewhere around 15-20 electrons.

OK, having introduced the topic, I will retire, and let the experts beat me 
up.  ;^)  I have tried in the discussion above to keep away from terms for 
which I do not know the precise definition.  I am just trying to outline 
what happens.  The dark current is probably the least of our problems when 
we can operate the CCDs below -20 C or so.  By -30 C it is pretty negligible.

Tom Droege


At 05:59 PM 12/7/00 -0700, you wrote:

>As some of you noticed, my previous histograms were incorrect.  Something
>about mixing the rows in the columns in CFITSIO routines...  Well anyhow,
>the question still stands, even though the data looks better (I also removed
>7 lead in/out pixels on each row/column).  Is there a "standard" way of
>determine the locations of bad pixels in the image?  Do you use the dark
>frame, fabricated flat field?  Or do you even not worry about the pixels
>being 'bad' unless they are at the extremes (0 or 65535) from a dark?  Will
>CCD's loose a single pixel, or row/column over time?
>
>% ./histo h3r1836.524 21
>     0 to  3120 =       0
>  3120 to  6241 =       0
>  6241 to  9362 = 4102506
>  9362 to 12483 =     123
>12483 to 15603 =       7
>15603 to 18724 =       6
>18724 to 21845 =       0
>21845 to 24966 =       2
>24966 to 28086 =       1
>28086 to 31207 =       1
>31207 to 34328 =       0
>34328 to 37449 =       0
>37449 to 40569 =       0
>40569 to 43690 =       2
>43690 to 46811 =       0
>46811 to 49932 =       0
>49932 to 53052 =       0
>53052 to 56173 =       0
>56173 to 59294 =       0
>59294 to 62415 =       0
>62415 to 65536 =       0
>
>% ./histo h4r1836.524 21
>     0 to  3120 =       0
>  3120 to  6241 =       0
>  6241 to  9362 = 4101496
>  9362 to 12483 =    1086
>12483 to 15603 =      63
>15603 to 18724 =       1
>18724 to 21845 =       0
>21845 to 24966 =       0
>24966 to 28086 =       0
>28086 to 31207 =       0
>31207 to 34328 =       0
>34328 to 37449 =       2
>37449 to 40569 =       0
>40569 to 43690 =       0
>43690 to 46811 =       0
>46811 to 49932 =       0
>49932 to 53052 =       0
>53052 to 56173 =       0
>56173 to 59294 =       0
>59294 to 62415 =       0
>62415 to 65536 =       0
>
>
>Robert S. Creager
>Senior Embedded Software Development Engineer
>Multi Platform Tape Library Development
>Ph:      303-673-2365
>Fax:    303-661-5379
>Pager: 888-912-4458
>StorageTek
>INFORMATION made POWERFUL