Re: data corruption

[hjohnson@pluto.njcc.com (Herbert R Johnson)]

>From reading the responses to my "checksum all files" suggestion,
I'd say that Chris has the solution in hand: the "md5sum" program for
UNIX or MS-DOS. I'd urge Chris to get the sources for this or a similar
program, complile them for Unix, Linux, and MS-DOS, run them over a
set of Tom's files under those OS's to confirm consistent results.
Then make sources and executables available for TASS members and
TASS end users. They can use them or not.

Tom says this is not a problem; I disagree, otherwise what was with all
the email about bad CD-ROM's? Also, Tom does not want to reread every
CD he produces; OK, just create the checksums ONCE, before creating
the first CD; then write the CHECKSUM list as yet another file on
your CD - run it ONCE per *distribution*, not per disk. Rob Creager suggests
(as Chris notes) that CFITSIO has checksumming available. The problem
is, not every TASS data receipient may have or use CFITSIO: and in
the future who knows what computers will be in use - we need a set of
STAND ALONE checksum programs.

I offer below summary responses and my more detailed replies, but it just
fleshes out the discussion and my responses, nothing much additional. Thanks
for all the discussion but I think Chris tagged it.

Herb Johnson

Here's a summary of responses so far to my file checksum idea:

**Date: Fri, 8 Jun 2001 09:54:17 -0700 (PDT)
**From: Chris Albertson <chrisalbertson90278@yahoo.com>

I believe there is a convention for checksumming FITS files.  The
checksum is written as a line in the header.  You can checksum the
whole file of just the data or both.  I think in the software I write I
did both.

I doubt a readable file would fail a FITS checksum test but still I
like the belt and suspender method.

I think it is a good idea to use an external program to compute a
checksum.   The best one of these out is "md5sum"  On UNIX or in a DOS
box you can do "md5sum *.fits > sums.md5" and create a checksum file
for all files in the directory.  Later on you can do "md5sum sums.md5"
and check all of the files.  It takes a while to scan 600MB of data but
the method is close to fool proof.

From: "Creager, Robert S" <CreagRS@LOUISVILLE.STORTEK.COM>
Date: Fri, 8 Jun 2001 11:46:39 -0600 

I just looked at CFITSIO, and indeed, there is a group of routines for
creating/checking checksums.  I can easily add these in (provided they work
with the windows version) if desired.  There would be two new keywords
added, DATASUM and CHECKSUM, with associated ASCII string of quoted digits.

Let me know.
Cheers,
Rob

From: Tom Droege <tdroege@veriomail.com>
Date: Fri, 08 Jun 2001 13:12:18 -0500

I come from the Seymore Cray school of designers.  "Parity is for 
Amateurs".   Hmmmm!  I guess we claim to be amateurs.  Still, a checksum 
will tell you that data that you already know is bad is bad.  Not much 
worth the effort, I think.  OK, a checksum would detect one missing bit 
somewhere that would not otherwise be detected.  Most of our data is 
noise.  A missing bit here and there is not a disaster as it would be if we
were transferring billions between banks.  There are a lot of other things
much higher on my list. At the moment it is tracking down whole images of
bad data, fixing cameras, finding a spot to put Rob's program so that it 
does not use any time, etc..

Note this all comes from me sending out a lot of bad data because I used 
the minimal time process to copy disks.  If I had done read after write
testing, they would still not be out.  As it is, most of you have 9 out of 
10 good disks and much good on the tenth.  Good enough for this preliminary 
work, I think.  Later I can send out replacement copies of disk 18e.   I am 
waiting for someone to send me a curve of a new variable that would benefit 
from one more measurement.  ;^)

Tom Droege

Here's my comments in reply:  I won't waste time being gentle.

1) "is there a program to do this?" Chris says:

"I think it is a good idea to use an external program to compute a
checksum.   The best one of these out is "md5sum"  On UNIX or in a DOS box.."

As I suggested, it's an old problem and already solved.
I'll bet the source to md5sum, or a workalike, are available. Almost certainly
they are in C; as universal a programming language as exists. Sounds like
Chris has the solution.

2) "Why not use CFITSIO? routines?". I don't think that will completely
solve the problem. From my previous message:

> In the case
> of TASS, it's fair to say that its checksum program set will have to
> run in Windows, Unix/Linux (x86 version and perhaps Sparc), and (if
> you please) MS-DOS. I'd suggest the source be in C and (perhaps)
> BASIC. Least effort may be for one person to grab appropriate C
> source
> and compile for all the above. I myself cannot do this.
> 
> IT IS IMPORTANT THAT ALL VERSIONS IN ALL OPERATING SYSTEMS PRODUCE
> THE SAME
> CHECKSUM PER FILE.

So the idea of using the "CFITSIO library" and to only checksum the data
in the imagefiles won't work. The reciepient of the CD-ROM data files
may not have access to that library, or to programs using that library,
or to OPERATING SYSTEMS which are supported by that library. Also, keep
in mind this is a SIMPLE SIMPLE program: a file checksum program has NO
IDEA of the contents or schema of the input files: all it knows is a
filename and a file length.

ALso, if the source of the checksum program
is provided, some end-user 20 years from now will be able to migrate it
to whatever operating system is in use at that time. (in 1981, 20 years
past, the operating systems commonly available were CP/M, MS-DOS, some
flavors of UNIX, VAX VMS, ....).

3) "Is this really a problem?" Tom says no.

He says " Still, a checksum
will tell you that data that you already know is bad is bad.  Not much 
worth the effort, I think.  OK, a checksum would detect one missing bit 
somewhere that would not otherwise be detected.  Most of our data is 
noise....Note this all comes from me sending out a lot of bad data because I
used the minimal time process to copy disks. ...There are a lot of other
things [to fix] much higher on my list."

Um, "missing bits" is not the issue, Tom. The problem is corruption in
COPYING DATA FILES. This has nothing to do with noise in imaging data.
The basic question is: "How does someone who recieves a TASS imaging data
file know if the file is an accurate copy of the originally collected data?"
My comments noted that this will ALSO be a problem 10, 20 years from now
if the "corruption" is due to literal damage to CD-ROMS or whatever other
media is in use then. Also, the problem may occur at the USER'S END;
their CD-ROM drive may have trouble reading someone's copy of TOM's copy
of his original data.

As to: "if I did read after write testing, I'd still be copying disks".
Tom, this solution ELIMINATES THAT WORK. You would only read and checksum
files ONCE, on your hard drive; then write the checksum lists as a file
along with the data. Let the USER read all the files and verify. If you
want quality control, you can grab a random CD, type ONE COMMAND LINE
IN MS-DOS, and walk away from the computer. Look at ONE output file
to see if any file checksums changed.

Finally, as to "there are other things to fix"; well, Chris already has
a solution in hand. As the programs are independent of our data, data
format, imaging, optics, astronomy, etc.; I don't see that a technical
debate about standards is necessary. It's already fixed!

Tom offered his "school of thought"; my school is the old A-B-C priority
list, where A priorities MUST be resolved NOW, B's later, and C's don't
matter a lot. *However*, there is a time when, if you don't solve the
B's and C's, they pile up, and THAT situation becomes an A priority.
So C's that can be fixed quickly - if there are not too many of them -
should be solved to avoid the "pile-up".

Herb Johnson

Herbert R. Johnson              http://pluto.njcc.com/~hjohnson
hjohnson@pluto.njcc.com         voice 609-771-1503, New Jersey USA
             amateur astronomer and telescope tinkerer
   reseller of classic Macs & accessories from Plus to PowerMac
   S-100 & 8-inch drive manuals and parts, call for "Dr. S-100"