Re: Some Success

[Stupendous Man <richmond@a188-l009.rit.edu>]

  Tom writes:

> It appears that I got Michael's pipeline to run through to conclusion.  A 
> real success for me.  Thank you Michael!
  ...
> OK this gives me the .cal file.  In this case I processed the 56 images 
> from one exposure sequence.  This gives me 160,000 or so star 
> measurements.  I have taken enough of my Linux course to sort this 
> list.  This sort of puts the stars together.  But if one sorts on RA and 
> then Dec. there is the obvious problem that the data from two stars at the 
> same RA is mixed together.

  I've written a Perl script which reads the ".cal" file and writes
an output file in which all the measurements of a single star are
consecutive.  It also calculates the mean of each star's measurements,
and the standard deviation from the mean, and some other stuff.

  You can find a copy of the script, called "calc_gammas_cal.pl" at

        http://spiff.rit.edu/tass/calc_gammas_cal.pl

One uses it like so:

        sort -n +1 M*.cal | calc_gammas_cal.pl  radecfile 

  In other words, one first sorts the .cal file by RA, and then
sends that sorted file as input to the "calc_gammas_cal.pl" script.
One could also create a separated, sorted copy of the M*.cal files,
like so:

        sort -n +1 M*.cal > sorted.cal
        calc_gammas_cal.pl radecfile < sorted.cal

  The "radecfile" argument in these commands is the name of an 
auxiliary file one must create first.  This auxiliary file must
be an ASCII text file with one line per image.  Each line should
have four words:

        filename   JD    RA    Dec

where
           JD  is the Julian Date of the exposure
           RA  is the Right Ascension of the center of the image (degrees)
           Dec is the Declination of the center of the image (degrees)

  Yes, this information _is_ in the "make_list.out" file, it's true;
but the RA and Dec in that file are based on the FITS header keywords,
and are not precise.  The "calc_gammas_cal.pl" script uses these
precise positions to determine how far from the center of each image
an individual stellar measurement was made; it includes the distance
away from the center in its output file, so that one can plot residuals
versus position.  

  (You may recall that I mentioned in a previous posting that I was
planning to modify the pipeline so that it wrote extra information
into the "make_list.out" file.  Well, now you know one of the pieces
of information I want to write!)

  One can make up this "radecfile" by hand, if one figures out exactly
where the telescope was pointing for each image.  If one runs the pipeline
at its default verbosity, one can find the required information: the
"astrom" step prints out the precise central (RA, Dec) for each image
as it makes the astrometric solution.   I've written a second Perl
script, "grab_pos.pl", to read this information from the pipeline 
output and put it into an ASCII file by itself.  You can find this
second, very simple script, at

         http://spiff.rit.edu/tass/grab_pos.pl

  
  So, to recap, if you want to avoid writing programs to do it yourself,
you could

        a) download the two Perl scripts onto your machine

        b) run the pipeline at verbosity level "2" or higher,
                saving the pipeline output into a file, say, "pipeline.out"

                    tclsh < cmd.in >& pipeline.out

                The "photom" stage of the pipeline will create one or
                more calibrated output files, with names like
                "Mhra2011797.cal"

        c) run the "grab_pos.pl" Perl script on the pipeline output
                to create a file with precise (RA, Dec) positions

                    grab_pos.pl < pipeline.out > radec.out

        d) run the "calc_gammas_cal.pl" script to place all observations
                of a star into consecutive lines of a big output file
 
            sort -n +1 M*.cal | calc_gammas_cal.pl radec.out > stats.dat


  The output of the final step is an ASCII text file with lines that
have 13 words each.  There is one line per star per V/I pair of images
(i.e. if a star was detected in both V and I images, for 5 pairs of images,
then there will be 5 lines in this output file for that star)

       col 1: number of measurements of the star
       col 2: mean value of all V-band measurements
       col 3: difference (V mag in this pair - mean V mag in all pairs)
       col 4: mean value of estimated uncertainty in V magnitude
       col 5: standard deviation of V-band magnitudes from their mean
       col 6: mean value of all I-band measurements
       col 7: difference (I mag in this pair - mean I mag in all pairs)
       col 8: mean value of estimated uncertainty in I magnitude
       col 9: standard deviation of I-band magnitudes from their mean
       col 10: (RA  of this star - central RA of image)  in V for this pair
       col 11: (Dec of this star - central Dec of image) in V for this pair
       col 12: (RA  of this star - central RA of image)  in I for this pair
       col 13: (Dec of this star - central Dec of image) in I for this pair
       
  Note that each time the pipeline measures the magnitude of a star, it
estimates the uncertainty of this measurement.  These uncertainty estimates
are averaged in columns 4 and 8.  On the other hand, by comparing the 
magnitude measurements in different images, one can determine the ACTUAL
uncertainty in the measurements: these are placed in columns 5 and 9.
If the software is doing the right thing, then the ESTIMATES of uncertainty
ought to be close to the ACTUAL uncertainties.  In Tech Note 76, 

         http://a188-l009.rit.edu/tass/technotes/tn0076.html#scatter

you can see that on one set of Mark IV images, these two methods do yield
comparable values, except for the brightest stars (due to saturation of
the CCD).  

  The point of all this is to be able to plot quantity "y" vs. "x"
and look for systematic errors or trends.

  Good luck.

                                                Michael Richmond