Re: External Queue

[Chris Albertson <chris@topdog.pas1.logicon.com>]

Nicholas Beser wrote:
> 
> Chris Albertson wrote:
> 
> >
> > The only problem here is that if mira, your data reduction computer,
> > is slow.  (I think you said it is a SPARC 2.Which is kind of like a
> > loaded 486 but with better I/O and graphics.)
> 
> It's more like a low end Pentium (60Mhz). I timed the conversion process last
> night. It took about 13 seconds to convert a FITS file to raw, convert the raw to
> pgm, create a 1/4 scale pgm, and then compress both pgm files to jpeg. I also wrote
> a small routine (parsename) that examined the file name, determined that it was CCD
> 0,1, or 2 and then copied them to the file name latest_ccd#.jpg or
> latest_ccd#.s.jpg and placed them in the web image directory. While the script
> worked fine running it at mira, it had problems running the parsename routine from
> tass. I think it might have worked better if I wrote it in perl.
> 
> I also have 5 of these rather slow sparc stations. So it would not be unreasonable
> to have one dedicated to each sensor (parallel processing).
> 
> The issue will be the automated star list generation. If it can be completed in 12
> minutes or less per channel, then we could have the lists available in real time as
> well. I have access to more sparc 2's so if I add two more, I could run them with a
> processing time of 24 minutes. I don't think it would be that slow.

I think we can reduce a Mk III image in 12 minutes even on a SPARC 2.  It will
be
different with the Mk IV if you are doing 100 second exposures.  Then we will
have
two 2K x 2K images ever couple minutes or about one per minute.  I doubt a SPARC
2
could keep up.  We will need to queue them for processing

There is a simple way to implement a queue.  Much simpler then the Idea
I proposed yesterday that used the UNIX printing facility.  Have the 
real-time controller dump the FITS files in (say) /DATA.  NFS Mount 
/DATA to all the SPARCs. Now run a script like this on each SPARC

  while (1)
    foreach FF ( /DATA/0* )
      process $FF
      mv $FF /PROCESSED/$FF
    end
  
    sleep 15
  end

The Linux PC now only has to run a one line script that moves the
file into /DATA.  It is importent to use "mv" and not "cp" because
you do not want half written files appearing in /DATA

Now if the images come off the camera at one per minute and if the SPARC
takes five minutes to process an image, no problem, all that happens is
the /DATA directory still has FITS files in it in the morning.  The
processing maybe continues 'till noon.  The above script would only pick
out "channel 0" files.  If you had 5 SPARCs each could run a script with
some fancy wild card that matches 1/5 of the FITS files like
"foreach fits ( /DATA/*.??[01] )" on the first SPARC and
( /DATA/*.??[23] ) on the second and so on.  This part is
easy to set up.  The hard part is writing the script I called 
"processes".  It's job is to extract data from an image and move that
data to some on-line database.  I still like the evolutionary approach
We can improve the system a little a a time. 


-- 
--Chris Albertson

  chris@topdog.logicon.com                Voice:  626-351-0089  X127
  Logicon RDA, Pasadena California          Fax:  626-351-0699