RE: Pipeline and Introduction

["Knowles, Richard (CORP, Consultant)" <Richard.Knowles@corporate.ge.com>]

Tom,

Thanks!  This is extremely helpful.  When I return next week I will
set about doing this.  I've read and reread Michael's note and I get more
from it each time I do.  As I said, this is new territory for me so 
I've not doubt that I've read the answers to all my questions but I just
haven't the knowledge to back it up to tell me that I have <grin>.  I've
also drilled into the TN's and Email but there is so much there that
it will be a while to fully assimilate it.

Question - are the bias files for the V and I cameras not used or is
there another step to do the same with them as the Dark and Flats?

Just to give another sampling point on pipeline runtimes, I had 
fired off the pipeline last night to do the Disk 19 images using
the Dark and flat files from Michael's distro (which as you point out do 
not match up with the new data) and the pipeline ran for around 5 hours
before it died trying to create the final calibrated star list.  This is
on my Duron 900 with only 64M of memory.  Once I have a flow established
I'll add more memory and see if it cuts the runtimes.

Onward and upward!

Rich



-----Original Message-----
From: Tom Droege [mailto:tdroege2@earthlink.net]
Sent: Wednesday, March 20, 2002 11:16 PM
To: rknowles@woh.rr.com; tass@listserv.wwa.com
Subject: RE: Pipeline and Introduction


Rich and Maceij,

You both need this answer so I will answer to the list to get you 
started.  But recognize that it is the blind leading the blind here.  I 
only know that I did this successfully, sort of.  Note that the layout of 
the images has changed since the sample data that Michael included on the 
test disk.  This data was taken in a 2043 x 2037 format.  The data of DS20 
was taken in a 2064 x 2037 format.  You get to worry if Michael has done 
everything correctly so that this is picked up by the program.  The .fits 
header is correct.  There is another problem.  Michael picks out a covered 
pixel for part of the computation.  I think the right one to use has 
changed with the newer format.  You want to use the pixels before the 16 
overscann pixels at the end of the line.  Roughly 2040-2049 but you better 
look at them and pick out which ones to use.

I assume that you have read all 21 pages of Michael's excellent write up.

OK, the reason I sent you Disk 1 and Disk 2 is so that you could make 
proper darks and flats for this data set.  .  Disk 1 is a set of: 40 flat 
files, 20 dark files, and 20 bias files for the V camera.  Disk two has the 
same set for the I camera.

The dark files actually say bias files in the fits header.  You have to do 
an edit on the dark files to change "bias" to "dark" or Michael's program 
will not find them.  They were improperly labeled in the fits header when I 
took them.

The idea of a dark image is that it is taken at the same exposure length as 
the sky (called "object") images.
The same holds true for the flats on the disk.  These were taken at 100 
seconds as were the darks and the object images.

Now you run Michael's pipeline with parameters in the markiv_driver.param 
file set to:

do_setup
do_makelist
do_make_dark

Note that you have to follow the instructions which involves editing the 
make_list script to tell it the directories where you have put the files.

I think that is all you need.  Now run the pipeline.  It should generate 
the master dark for V and I and put them in the directory that you have set
up.

Now that you have a dark you can make a flat.  This time the .fits images 
properly say "flat" in the fits header.  Unfortunately Michael's program 
was set up to make flats from object images - those with lots of stars in 
them.  This is done by taking the median of a lot of images.  One hopes 
that some of the images have blank sky in every position for this to 
work.  But disk one and two have real flats made with a light box.  This 
should be better.  But again to let Michael's canned pipeline to work, it 
is looking for "object" in the fits header.  You have either to change what 
Michael is looking for in the script (preferred if you know how) or edit 
the .fits headers of the flats to say "object" instead of flat.  Again, I 
was able to do this with an edit script.

Now set up markiv_driver.param so that it has 1 s for:

do_setup
do_make_list
do_make_flat

Run it and it should put a master flat set in the appropriate file.

Now you are ready to put a set of real sky images in the input directory 
and run with the the markiv_driver.param set with the appropriate 
ones.  This time you don't need to make a dark or a flat as you have 
already made them.  Note that I suspect that you could do everything at 
once if set up properly.  I did it a step at a time looking at the results 
so I would be sure I understood what it was doing.

When I ran a full disk set 56 v and i images with a 1GHz Pentium III it 
took 3 hours to process the images to the .cal file.

I very much recommend that you read all of Michael's excellent 
documentation.  However, you don't have to understand it all to get it to 
work.  The one page e-mail message from Michael to me gives short 
instructions that should work with the pre-compiled set on disk 0.  I think 
I put this on disk 0.  The e-mail message is titled "CD is on the way".

Hope this helps.

Michael can comment if I have got it right.  I am just doing this from 
memory.  I muddled through and if I can do it, you computer experts can do 
it.  As L'l Abner would say "Any fool can do it, Ah did it!".

Tom Droege

At 09:18 PM 3/20/02 -0500, you wrote:
>Tom,
>
>I've started on disk19.  After copying the .fits files into the input
>directory
>I moved all of the output files (including the master_Dark, etc. files out
>of
>the output directory as I read in Michael's document that they are created
>by the
>pipeline.
>
>The popeline however fails, stating that it needs the master dark files.
>
>My question is, are they created or is this a step I have to do prior to
>running
>the pipeline?
>
>I copied the master files back and the pipeline is running, but I'm worried
>that I'm
>missing something here.
>
>Rich
>
>-----Original Message-----
>From: Tom Droege [mailto:tdroege2@earthlink.net]
>Sent: Wednesday, March 20, 2002 5:34 PM
>To: rknowles@woh.rr.com
>Subject: RE: Pipeline and Introduction
>
>
>Rich,
>
>I am not sure which version of the MR pipeline you are using.  The one I am
>using was precompiled for me by Michael.  So if I mark the steps to take
>with a 1 it runs to completion and gives calibrated stars and a measure of
>how well the process went.  I am working with the .cal file which is a list
>of calibrated measurements with combined v and I measurements in one .cal
>file line.   At this point, the positions have been matched up to a few arc
>seconds.  I think the .cal file comes several stages later than the .clt.
>
>Don't expect any real magnitudes below 7 or 8.  These are always saturated
>and so get deleted from the processing stream, I hope.
>
>Tom
>
>
>At 06:10 AM 3/20/02 -0500, you wrote:
> >Tom,
> >
> >Thanks for the note.  My next step was to actually identify the stars
> >in the field which wasn't too difficult.  On hira2011797 the bright one
was
> >zeta virgonis.  The internal RA/DEC numbers in DS9 were off a little but
> >it got me close enough to where I was able to identify the nearby stars.
> >
> >The bright one I was using as a locator, Zeta Virgonis is ~2.3 mag, and
by
> >that time I was pretty convinced that I was truly looking at raw order
> >of magnitude.  Is the raw magnitude linear enough to where one could
> >extrapolate
> >magnitudes from it?
> >
> >I'm still struggling with the different coordinate systems, between WCS
and
> >the
> >others so it'll take some practice and further experimentation.
> >
> >As I understand it then, the last step in the pipeline is the creation of
> >the *.clt
> >files where the correlation between the position and the relative
>magnitudes
> >of
> >the V and I filtered channels is rolled up?  It doesn't look like there
is
> >any matching
> >being done between the catalog and the actual stars then - which I
presume
> >is a further
> >step yet?
> >
> >Rich
> >
> >
> >
> >-----Original Message-----
> >From: Tom Droege [mailto:tdroege2@earthlink.net]
> >Sent: Tuesday, March 19, 2002 11:59 PM
> >To: rknowles@woh.rr.com
> >Subject: Re: Pipeline and Introduction
> >
> >
> >Rich,
> >
> >Others can answer this better than I.  But I believe raw is a purely
> >arbitrary scale.  It is then corrected later to the true magnitude.  So
raw
> >can be many mags off, I think.  I think the pipeline write up covers
this.
> >
> >Note that you should be able to look at a frame with DS9 and a star atlas
> >and get the position corrected so that you can estimate where a star
> >is.  Then you can get its mag from something like Uranometria 2000 to get
> >an idea of where you are.  That is what I do.  Others look everything up
on
> >the net.  I am just learning how to do this.
> >
> >Don't worry about asking questions like this, many don't know this much
and
> >can learn from your questions.
> >
> >Remember I am the hardware guy, I just learned to spell Linux.
> >
> >Tom
> >
> >At 10:16 PM 3/19/02 -0500, you wrote:
> >
> > >In looking at the collation file for hra2011797 I noticed that the
>minimum
> > >magnitude is 16.701.  This is termed as the Raw Magnitude, and I'm
> >wondering
> > >how seriously to take this number.  The point is, when I look at the
>frame
> > >with DS9, there seems to be some pretty respectably bright stars there
>and
> >I
> > >wonder whether these numbers are right.  Checking my results against
what
> >is
> > >in the distro CD they match up (for this frame at least) and in reading
> >Mike
> > >Richmond's web page, nothing jumps out at me to suggest the magnitude
> >number
> > >is 'gospel'.  Anyone have experiences that might help?