RE: Bad pixel determination with better software

["Albertson, Chris" <CAlbertson@primeadvantage.com>]

To find bad pixels, I think you'd be better off using
flat field data then using dark frames.  The (well, my)
definition of a bad pixel is one that has a hugely non-linear
response to light.  In other words it stays low even in bright
light or hot in the dark.  It's the fact that the pixel's ADU
value is not in proportion to it's illumination that makes it
bad not the absolute ADU value.  So, even if it is stuck at the
mid point of the range it is still stuck and "bad".  Using 
only dark frames gives you only one illumination
level to test linearity with.

I figure you can determine the general level of illumination
by the average ADU value of the surrounding couple hundred pixels.
If you find a pixel that does not respond to changing light then
Yes, I'd put it in the bad pixel mask.

Dark values are repeatable and can be subtracted out.

All that said, it should be clear that a pixel stays at one end
of the ADU range in every dark fram is likely "bad".  But I think
these are just the easy to find bad pixels.  


> -----Original Message-----
> From: Tom Droege [mailto:tdroege@veriomail.com]
> Sent: Thursday, December 07, 2000 8:37 PM
> To: Creager, Robert S; tass@listserv.wwa.com
> Subject: Re: Bad pixel determination with better software
> 
> 
> Rob and all,
> 
> To look for bad pixels, I would use a stack of dark frames 
> from the same 
> detector.  I would take as many dark frames as I could get 
> (that is why you 
> have so many disks), sort them by mean value in the middle of 
> the frame, 
> say x 500-1500, y 500 -1500.  Then I would take the mean of a 
> bunch where 
> the variation in mean value was small compared to the sigma 
> of the area.
> 
> Now I would take the stack of comparable images - the above 
> process got you 
> a stack taken at roughly the same temperature - and take the 
> mean of the 
> full frames.  Now you can take the histogram as below.  This 
> should show up 
> any "hot" pixels.  Now you have to go through and "find" the 
> hot pixels by 
> x-y coordinate.
> 
> OK, the experts will probably jump all over me.
> 
> Not that this is just the first step to get a dark frame and 
> to identify 
> pixels with high dark current.  There are many more problems to come.
> 
> I suspect that you will not find enough hot dark pixels to worry 
> about.  But it is worth doing the experiment.  However bad 
> they are, they 
> get much better with reduced temperature.  The dark current 
> goes down by 
> half for a 5 C decrease in temperature.  But it is really 
> much better than 
> that.  There are hot pixels that tend to turn off at some 
> temperature.  So 
> they go from a high current to near zero as their critical 
> temperature is 
> passed.
> 
> I do not know of any mechanism that causes change of dark 
> current (at the 
> same temperature) over time.  Or for a pixel to be lost.  
> Well, not quite 
> true.  A lot of radiation will do it.  I suppose that one 
> could get really 
> unlucky and get clobbered by a cosmic ray.  But I think such 
> problems heal.
> 
> For the most part, what you have gotten in the past at any specified 
> temperature is what you expect to get in the future.  The 
> drift of the 
> electronics should also be small.  I would expect the 
> pedestal (the mean 
> value of a dark frame) to change only about 3 counts per C of 
> the ambient 
> temperature.  That is the electronics temperature.  There is 
> a thermometer 
> that is relatively close to the electronics that are critical.
> 
> Exercise for the student:
> 
> 1) Measure the mean value of a bunch of dark frames.  Use the 
> center area 
> as above.
> 2) Plot the mean value of the above against the VCO 
> temperature.  (Woops, 
> we are not recording the VCO temperature.  So this cannot be done. )
> 3)  Plot the mean value of the above against the CCD 
> temperature.  We have 
> been recording this.
> 
> OK, there are two (main) things that can make the pedestal vary.
> 1) Dark Current.  This doubles for each 5C increas in 
> temperature.  (roughly).  This can be sorted out from the pedestal.
> 2) Electronics drift.
> 
> Lets take the electronics drift first.  I expect about 3 ADU 
> per C.  This 
> is based on a 100 ppm expected typical component value drift 
> and the 25000 
> ADU pedestal.  It will depend on how all the errors add up.  
> While the mean 
> value of the pedestal will move around, the sigma should remain 
> constant.  It is just the noise from the ADC and other stuff 
> on the printed 
> circuit board.  So the mean value of the frames are expected 
> to move around 
> with the ambient temperature, but the noise due to this 
> movement will not 
> change (much).
> 
> Now the dark current.  The dark current changes the pedestal by 
> accumulating electrons in the pixel.  The accumulation of 
> electrons causes 
> the pedestal to shift by 1/2.2 (roughly) ADU per electron.  
> The noise (as 
> expressed in electrons) should increase as the square root of 
> the number of 
> electrons.  So pedestal change caused by electronic drift causes no 
> increase in noise, pedestal drift caused by dark current 
> causes the noise 
> to increase as the square root of the number of electrons.  
> OK, lets say we 
> saw the mean value of the pedestal to shift during a long 
> dark exposure by 
> 1000 adu.   This means the mean value of the pixel changed by 2200 
> electrons.  The square root of 2200 is 47 electrons.  So we 
> expect a sigma 
> of 47 electrons.  (To get this we compute the sigma of all 4 
> million pixel 
> values) But we measure in ADU which is roughly 2.2 electrons. 
>  So we would 
> expect the sigma taken over many pixels to increase by 22 
> ADU.   OK, this 
> is a random process.  While the mean pixel has a leakage of 
> 2200 electrons, 
> this is *not* 2200 electrons for each and every pixel.  Because it is 
> random, they will be distributed.  Most pixels will be within 
> 47 electrons 
> of 2200.  But we have a lot of pixels, so some will be 4 or 5 
> sigma away - 
> so it would not be surprising to find one with 2500 or 1900 
> electrons.  Further there are other processes going on, so 
> while a 1800 
> electron pixel would be rare, one could easily have a 10,000 electron 
> pixel. (from a "hot" pixel - one where there was some impurity in the 
> silicon which acts as a source of electrons)  The 
> distribution will be 
> skewed.  In fact, if you look at dark frames, you will see 
> that at room 
> temperature the pixel distribution is quite skewed, but as 
> the temperature 
> is lowered the distribution becomes more and my symmetrical.
> 
> OK, it is never as easy as this.  There are lots of noise 
> sources and we 
> hope we can add them all up in quadrature.  We have a "floor" in this 
> system somewhere around 15-20 electrons.
> 
> OK, having introduced the topic, I will retire, and let the 
> experts beat me 
> up.  ;^)  I have tried in the discussion above to keep away 
> from terms for 
> which I do not know the precise definition.  I am just trying 
> to outline 
> what happens.  The dark current is probably the least of our 
> problems when 
> we can operate the CCDs below -20 C or so.  By -30 C it is 
> pretty negligible.
> 
> Tom Droege
> 
> 
> At 05:59 PM 12/7/00 -0700, you wrote:
> 
> >As some of you noticed, my previous histograms were 
> incorrect.  Something
> >about mixing the rows in the columns in CFITSIO routines...  
> Well anyhow,
> >the question still stands, even though the data looks better 
> (I also removed
> >7 lead in/out pixels on each row/column).  Is there a 
> "standard" way of
> >determine the locations of bad pixels in the image?  Do you 
> use the dark
> >frame, fabricated flat field?  Or do you even not worry 
> about the pixels
> >being 'bad' unless they are at the extremes (0 or 65535) 
> from a dark?  Will
> >CCD's loose a single pixel, or row/column over time?
> >
> >% ./histo h3r1836.524 21
> >     0 to  3120 =       0
> >  3120 to  6241 =       0
> >  6241 to  9362 = 4102506
> >  9362 to 12483 =     123
> >12483 to 15603 =       7
> >15603 to 18724 =       6
> >18724 to 21845 =       0
> >21845 to 24966 =       2
> >24966 to 28086 =       1
> >28086 to 31207 =       1
> >31207 to 34328 =       0
> >34328 to 37449 =       0
> >37449 to 40569 =       0
> >40569 to 43690 =       2
> >43690 to 46811 =       0
> >46811 to 49932 =       0
> >49932 to 53052 =       0
> >53052 to 56173 =       0
> >56173 to 59294 =       0
> >59294 to 62415 =       0
> >62415 to 65536 =       0
> >
> >% ./histo h4r1836.524 21
> >     0 to  3120 =       0
> >  3120 to  6241 =       0
> >  6241 to  9362 = 4101496
> >  9362 to 12483 =    1086
> >12483 to 15603 =      63
> >15603 to 18724 =       1
> >18724 to 21845 =       0
> >21845 to 24966 =       0
> >24966 to 28086 =       0
> >28086 to 31207 =       0
> >31207 to 34328 =       0
> >34328 to 37449 =       2
> >37449 to 40569 =       0
> >40569 to 43690 =       0
> >43690 to 46811 =       0
> >46811 to 49932 =       0
> >49932 to 53052 =       0
> >53052 to 56173 =       0
> >56173 to 59294 =       0
> >59294 to 62415 =       0
> >62415 to 65536 =       0
> >
> >
> >Robert S. Creager
> >Senior Embedded Software Development Engineer
> >Multi Platform Tape Library Development
> >Ph:      303-673-2365
> >Fax:    303-661-5379
> >Pager: 888-912-4458
> >StorageTek
> >INFORMATION made POWERFUL
> 
>