The "X Files" and other Trivia

[Tom Droege <tdroege@veriomail.com>]

Rob,

An ADU is the value of an ADC least significant bit.  It is just 
astrospeak.  I had never heard of it until I got around astronomers and I 
have been building instrumentation systems with ADC's in them since 1959.

OK, most of the files you have were written with Ao8m2.bas.  But the 
program is on the disk, so you can figure it out as to which images were 
written with which program.  I wrote the original code, and Arne modified 
it.  I have never really looked at the log file since Arne worked on 
it.  Sigh!  So I will have to do what you will have to do to decode what is 
written.

Well, I just looked at one of the disks, and the above is not even 
true.  This one was written with A08m1.bas.  So you have to look at the 
basic program on each disk to see what the format is.  It is not really 
that bad, as I think there are just two versions.  The version before I 
took ARNE to AZ and the one after.  You can find all the references to 
writing the log file by searching the QBasic program for #5.  All the 
writes to the logfile are done using I/O channel #5.  Some of the stuff is 
written by the subroutine "LogEntry" and some is written by code Arne wrote.

I think the message is to not take the log file too seriously.  I wrote 
junk code to just make a few temperature recordings so that I could solve 
some problem that I have that I have since forgotten.  Now it is etched in 
stone.  This is junk code.  Eventually we will do it right.

OK, what to do for the present???  Well, I suggest that we only try to look 
at it by eye when we have a specific problem to solve.  I don't think it is 
worth the effort to try to automate pairing log file readings with data 
files for  logs that were so poorly done.  Any effort we have should be 
going into designing a proper log file from the experience that we gain 
from this work.  This is sort of why I am after people to work on this data 
set.  They will quickly see that they want certain things in the log 
file.  It is not obvious that a log file is the way to do it.  The right 
way, I think, is to write a proper .fts file from the start, or to never 
write a .fts image file at all but to write star lists in .fts table 
format.  There has been a lot of talk about this here.  Nothing like 
getting several telescopes out in the hands of competent people (i.e other 
than me) to get the problem solved.  I hope you all can see the merit of 
getting together and sharing the effort of writing proper code.

What I don't want is for you to get discouraged looking at this mess.  I 
hope you will save your energy for the real data analysis that will come 
later.  OK, there is some good data on these disks, and it is worth 
rescuing it if we can.  It will not be the ultimate that we can achieve if 
we get all the right stuff in the right place.  Nothing would please me 
more than to analyze this data, determine say a 0.04 mag error, and to come 
to the realization that this could be reduced to 0.01 mag if we recorded a, 
b, c, ... in the .fts header.

You might consider doing my "exercise for the student" to get a start on 
this.  Just pick a few dark frames, and by hand (looking at the log file) 
match them up with temperatures.  This is not even so easy.  Well, I just 
looked.  There is local time in the logfile, and UT in the fits header, so 
you can match the logfile entry to the fits image.

OK, I assume that this is what you are trying to do.  One problem is that 
the format changes because I am writing junk format for tests.  I keep 
hoping that someone will write something better and ask me to use it.

I apologize for not working through your formats below.  I would just have 
to go to the QBasic code and figure it out.  I think you can do that.  If 
you are not familiar with the QBasic commands, it should not take long to 
learn them as they are standard Microsoft.  I can help if you need it.

OK, I have been looking at the log file.   The file names have an A, B, C, 
D, and X in them when they are first written.  RawToFits discards this and 
puts some of the stuff in the .fts header.  But the ABCDX are lost.  Ted 
wrote RawToFits before this format was invented.  Somehow Arne has lost the 
X when he wrote the log file.  So there are entries with A, B, C, D, and 
then one with no letter label line.  This is the "X File".  The "X File" is 
the dark file.   A, B, C, and D are in that order the files written after 
the "X File".

OK, you know where the "X Files"  (I love writing this) are.  They are 
listed in the .txt file on each disk.  You can get to about the right place 
in the log file by converting the UT in the .fts file name to my local 
time.  Don't ask me how to do this, but someone here will know.  Now you 
can look at the log file and look for an A, B, C, D, or nothing.  Now you 
are lined up.  After that, the log file lines should match up with the 
images.  So to me, the log file appears to have sets of 5 exposures, it is 
just that one line is missing from the "X File".

BTW, I think you can take the difference between the first two "X Files" 
and use it for the whole run.  The clock is pretty constant.  The .txt file 
tells you the first "X File".  Reading the .fts header give you the time it 
was taken.  The ABCD files are labeled after the "X File".  So I think you 
have everything you need to attach times to files.

OK, I think I have now sort of answered your first question.

As to your second question.  NO!  I just build cameras and give them 
away.  It is up to you all to figure out how to write software to make them 
work.  ;^)

Actually, if you can do it, it might be useful to read the .fts images and 
the logfile and insert the logfile data into the .fts header.  Possibly as 
a subroutine that others can use since it is not obvious that it is worth 
rewriting the file just to fix the header.

I am sending this to the list since all should be interested.

Tom Droege

At 03:05 PM 12/8/00 -0700, you wrote:

>Hey Tom,
>
>What is an ADU unit?  Just curious.
>
>Looking at the log files produced by the two basic programs I have (a07.bas
>and a08m2.bas), I'm finding that I cannot match entries to pictures.  The
>column definitions in the below data are as follows:
>
>1       - Time difference from FITS header and when the camera was "Closed"
>2       - File name
>3       - Epoch seconds of exposure
>4       - "Closed" index which matched (ie the first line matched the 321st
>Closed line in the logfile)
>5       - Exposure time
>6       - An unsuccessful attempt at determine dark frames
>7       - Time the picture was complete
>
>As you can see, the first group (a07.bas) doesn't have pictures for all the
>"Closed" keywords, and the second set (a08m2.bas) doesn't have enough
>"Closed" keywords for the pictures.  I am basing these claims off of column
>4 - the index, and how it skips in the first set and repeats in the second.
>I'm after two things from you.  Am I decoding the log files correctly?  And
>if so, can you modify the program to get the data for the missing picture
>(for future logfiles only of course).  I think I'm doing this correctly as
>when I read a08m2, the pictures seem to be taken in sets of 5 - 1 dark 4
>exposures, and yet the logfile has sets of 4 exposures...
>
>My next task will be to modify my software to get the camera temprature to
>put into the fits header (along with dark/normal as determined by
>STDEV/MEAN).
>
>  51 h3r1805.548 969260885 321 173 0 01:08:05-2000/09/18
>  47 h3r1805.551 969261119 322 178 0 01:11:59-2000/09/18
>  62 h3r1805.557 969261626 324 173 0 01:20:26-2000/09/18
>  62 h3r1805.559 969261856 325 174 0 01:24:16-2000/09/18
>  72 h3r1805.565 969262365 327 174 0 01:32:45-2000/09/18
>  73 h3r1805.568 969262593 328 174 0 01:36:33-2000/09/18
>  90 h3r1805.574 969263116 330 173 0 01:45:16-2000/09/18
>  84 h3r1805.577 969263350 331 178 0 01:49:10-2000/09/18
>  88 h3r1805.583 969263858 333 173 0 01:57:38-2000/09/18
>  90 h3r1805.585 969264086 334 173 0 02:01:26-2000/09/18
>
>  24 h3r1853.567 973413430 483 120 0 01:37:10-2000/11/05
>  23 h3r1853.569 973413605 484 120 0 01:40:05-2000/11/05
>  22 h3r1853.571 973413780 485 120 0 01:43:00-2000/11/05
>158 h3r1853.573 973413780 485 120 0 01:43:00-2000/11/05
>  73 h3r1853.575 973414185 486 121 1 01:49:45-2000/11/05
>  18 h3r1853.578 973414360 487 120 0 01:52:40-2000/11/05
>  17 h3r1853.580 973414535 488 120 0 01:55:35-2000/11/05
>  19 h3r1853.582 973414710 489 120 0 01:58:30-2000/11/05
>156 h3r1853.584 973414710 489 120 0 01:58:30-2000/11/05
>  76 h3r1853.586 973415115 490 121 1 02:05:15-2000/11/05
>
>Thanks,
>Rob