[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
RE: Data Disk Format
Tom,
Ok, here is some results. I'm still using h3r1838.720 since I don't have
1824 yet here at work. Note that I'm using the BZERO, so my results are all
greater than zero. What I did was to take every 190000th pixel (21 samples)
for the first data set, and every 100000th pixel (41) for the second set.
I've included the data, the perlscript, and the output of the perl script,
which calculates the mean and sigma.
> When I look with DS9, the minimum value is around 8850 and
> the saturated
> star value is around 50600 for a range of 41750. This matches the IS
> result to within the accuracy with which I did the test.
...the key being RANGE of 41750...
>I still suggest that you look at a few pixel values and compute their sigma
>by hand (OK, (max-min)/3 is close enough to compare 40 to 700).
Take the difference from saturated star to minimum values as you show above,
41750 and divide by 3 for 13916. This is patently incorrect, but a lot
closer to 700 than 40. To me, just looking at the range of 41740 as you
compute it, guarantees a sigma of larger than 40. Lets say for the image
above, there are 4,000,000 pixels and the mean is 8000. Lets say 0.01%
(400) of the pixels are saturated, and have a value of 30,000 above the
mean. Lets also say that the rest of the pixels are at 8000, so they don't
contribute to the sigma. To find the sigma, evaluate
sqrt((400*(30,000^2))/(4,000,000 - 1)) = 300. If you were to look at this
image or sample it, you'd probably never see those 400 hot pixels, yet the
sigma is quite high. For the sigma to be at 40, there can only be 7 pixels
with a value of 30000 above the mean - 40^2 * (4,000,000 - 1) / (30,000^2) =
7. How many saturated star values do you believe there are in the image you
looked at with DS9? I'll bet there is more than 7, with each one more than
1 pixel.
Got you convinced yet? I'll keep trying if not...
Mean is 7545.65 sigma is 120.889804106228 samples are 20
7413.000000
7461.000000
7632.000000
7615.000000
7620.000000
7414.000000
7653.000000
7702.000000
7432.000000
7492.000000
7418.000000
7659.000000
7396.000000
7445.000000
7514.000000
7411.000000
7814.000000
7632.000000
7574.000000
7616.000000
Mean is 7644.53658536585 sigma is 214.873462479778 samples are 41
7953.000000
7784.000000
7796.000000
7552.000000
7748.000000
7523.000000
7600.000000
7830.000000
7655.000000
7502.000000
7591.000000
7600.000000
7564.000000
7417.000000
7540.000000
7409.000000
7449.000000
7615.000000
7434.000000
7722.000000
7769.000000
7459.000000
7444.000000
7747.000000
7640.000000
7457.000000
7600.000000
7523.000000
7398.000000
7432.000000
7621.000000
7422.000000
7516.000000
7558.000000
7625.000000
7607.000000
7922.000000
7850.000000
8396.000000
8130.000000
8026.000000
#!/usr/local/bin/perl -w
use strict;
my $value;
my $sum = 0;
my @values;
while ( $value = shift )
{
$value = $value;
push @values, $value;
$sum += $value;
}
my $samples = scalar( @values );
die "Please give me some data on the command line\n" if !$samples;
my $mean = $sum / $samples;
$sum = 0;
while( $value = pop( @values ) )
{
$sum += ($value - $mean)**2;
}
my $sigma = sqrt( $sum / ($samples - 1));
print( "Mean is $mean sigma is $sigma samples are $samples\n" );
> -----Original Message-----
> From: Tom Droege [mailto:tdroege@veriomail.com]
> Sent: Friday, December 22, 2000 12:13 PM
> To: Creager, Robert S
> Subject: RE: Data Disk Format
>
>
> Rob,
>
> >I must disagree with you about where the problem lies.
> Raw-to-fits is
> >adding in a BZERO of 32768 and a BSCALE of 1 to the fits
> header for the MARK
> >IV images. What this says is take each pixel value as
> stored in the data
> >portion of the image, multiply by 1 and add 32768. If this
> is not what was
> >intended, then the raw-to-fits program is incorrectly
> writing the fits
> >header. Can you try DS9 on one of your dark files please?
>
> I am just trying to say that we have a format problem
> somewhere. It may
> well be that RawToFits is writing the wrong thing in the
> header. It may
> also be that this can be fixed by setting the right parameters in the
> reading program.
>
> I have tried DS9 with my files, and I now see that it is working just
> fine. When I look with Image Scientist, I see a background
> value of -24000
> and a saturated value of 18000. This is a range between
> background and
> saturation of 42000. When I run the calculations, I get a
> mean of -24102
> and a sigma of 25.8. When I look at a few pixels in the
> background they
> are consistent with a sigma of 25.8.
>
> When I look with DS9, the minimum value is around 8850 and
> the saturated
> star value is around 50600 for a range of 41750. This matches the IS
> result to within the accuracy with which I did the test.
>
> It looks like IS ignores the BZERO and BSCALE parameters and
> just reads the
> data as placed in the array.
>
> It seems to me that if the fits header says to add 32768 and
> multiply by 1,
> then you better do this in something more that a 16 bit
> format. Otherwise
> the result will be wrong. My bet is that is what is being
> done to get the
> high (900) values for sigma. This is just about the expected
> result for
> sigma with the two bytes are mixed up. I know, I have done this many
> times, and it is now a red flag to my brain.
>
> >The reason I chose
> >http://www.cv.nrao.edu/fits/data/samples/image/swp05569slg.fi
ts for you to
>look at, is it has a BSCALE of something other than 1 (.4)
When I look at this with DS9 I get of order 1300 for the background and
27000 for the bright spot.
When I look at it with IS, I get -26700 for the background and +20500 for
the bright spot. I also get very high values for sigma. 419 for a small
area and 292 for the whole thing. There is something wrong with this
file. It is not consistent with any of my data files. Looks like it has
had bytes swapped somewhere.
I still suggest that you look at a few pixel values and compute their sigma
by hand (OK, (max-min)/3 is close enough to compare 40 to 700).
>If I cannot convince you Tom with two programs (FTOOLS and DS9) agreeing,
>then I gues we will agree to disagree :-)
NO! We cannot agree to disagree, this is mathematics, and we have to get
it right. In this case, a sigma of 700 or 900 is so large that there has
to be something wrong in the way the data is being handled.
Just remember, Garbage in, Garbage out. Somewhere the real data is being
thrown away, and garbage is being substituted.
Tom Droege
At 10:23 AM 12/22/00 -0700, you wrote:
>Tom,
>
>I must disagree with you about where the problem lies. Raw-to-fits is
>adding in a BZERO of 32768 and a BSCALE of 1 to the fits header for the
MARK
>IV images. What this says is take each pixel value as stored in the data
>portion of the image, multiply by 1 and add 32768. If this is not what was
>intended, then the raw-to-fits program is incorrectly writing the fits
>header. Can you try DS9 on one of your dark files please?