Re: [TASS] New Data Handling Paradigms?

[Stupendous Man <richmond@A188-L009.RIT.EDU>]

  Alan envisions new ideas for running the Mark IV cameras ...

> User 1 could receive *just* for the objects of interest, a series of reduced
> instrumental observations and calibration data (from nightly short exposure
> "seeing" frames?), by email! Full images do *not* need to be transmitted.

  Certainly this is possible, but it appears to me that enabling this
capability may entail a full reduction of the entire frame (and, by
extension, all frames taken during the night) into a calibrated star
list.  I suspect that it will be difficult for camera operators who
produce megabytes of star lists to discard most of the data and keep
just the bits relevant to user requests.

  The problem is human nature: once one has gone through the whole
process of reducing and analyzing all the data, it's hard to throw
90% of it away :-/

> If, at the end of say, one week of observing, the camera operator ran a
> small utility to scan the FITS headers of all recent images, and this small
> "where pointed & when" file was routinely sent to a central database, the
> MK4 camera image coverage would be available in a timely, public and useful
> fashion. This central database could also know where to direct users making
> special image file requests, which may not be the actual camera site. (See
> C) below).

  This idea I like very much.  It's likely that the software Chris and
Peter are writing to drive the cameras will (optionally) create a
list of the camera exposures, so this may come "for free".  Of course,
keeping it in a database and letting people search through it will
take effort.

> The paradigm shifts in thinking that seem to be required, (IMHO) are:
>
> A) Stop thinking about a large central database containing *all* objects,
> and to start thinking about storing/merging only that centralised data that
> is absolutely essential to the perceived goals of TASS. In other words, a
> full object catalog may not be necessary. If only the data rows for first
> cut variables were stored in a central variable star database, a significant
> reduction in storage capacity ensues. Constant stars are not even presented
> to this database/table/catalog.

  This _is_ a big shift in thinking.  I applaud Alan for being able
to come up with it, and I look forward to hearing what other people
think about it.

> B) Provide good public awareness of Mk4 camera activity by setting up a
> central register of this activity, which is the output of the small utility
> mentioned above. (In fact this could also be done for all Mk3 image files in
> retrospect.)

  This isn't difficult to do.  I _thought_ that an existing tool at
the Rochester WWW site,

       http://a188-L009.rit.edu/tass/www_scripts/tass_location.html

already did this for Mark III data, but perhaps it lacks some feature
Alan has in mind.

> C) In all of the above, I have been assuming that a single *camera operator*
> does all of the work. This is not the only way to go! Using Tom's figures
> below as the output of an average site, (instead of Arne's somewhat higher
> rate) we have: 2 CDs / night x six good nights a month. A local group, with
> say, six interested "computers" (in the pre-1940s sense), could reduce 2
> disks (or more) each a month with ease. As Tom has pointed out before, snail
> mail has a very good data transfer rate when CDs are used.

  This looks like a sticky point: in my opinion, it's relatively simple
to _understand_ these two modes of operation:

      1. each camera operator reduces all his data
      2. a single person reduces all data from all cameras

(although I'm not saying that it's easy to succeed in either mode, it's
just easy to figure out who is responsible for what); on the other hand,
if the persons responsible for reducing the data aren't operators
or a single human, then I foresee the possibility of communication
problems ...


> -If distributed processing was used, reduction software would probably need
> to be kept at the same version number at all "computer" sites, to maintain
> some data standards.

  I suspect that this would be a bigger problem than most of us expect...

> The fundamental question behind my persistence on this subject is this: Will
> an extension of current practices cater for all potential users in the most
> efficient way?

  Undoubtedly not.  Thanks to Alan for starting this discussion.

                                           Michael Richmond