Re: Image Storage

[Tass Mailing List <tass@mail.alembic.net>]

Date: Thu, 04 Jan 2007 19:18:42 -0600
From: Thomas F. Droege <droege@fastmail.fm>
To: tass@tass-survey.org
Subject: Re: Image Storage

Hi Doug,

OK, step one is to load all 2000 CD/DVDs into a db.

What is step two?  Who does step two?  Do we process the disks as they
are loaded?  Who writes the code that does this?

Just trying to understand the proposed process.

Possibly with a published paper, someone can get a grant to do this??

I am willing to make a bunch of representative disk copies to get the
process started if someone wants to do it.

Tom Droege

On Thu, 4 Jan 2007 16:43:20 -0800 (PST), "Tass Mailing List"
<tass@mail.alembic.net> said:
>
>
> Date: Thu, 04 Jan 2007 17:38:55 -0500
> From: Doug Welch <welch@physics.mcmaster.ca>
> To: Tass Mailing List <tass@mail.alembic.net>
> Subject: Re: Image Storage
>
>
> Hi Tom,
>
> Well - I have to disagree. To paraphrase Cortex "Load them all and let
> the computer sort it out." For instance, one could ingest the pipelined
> data and then update the image record in the db (not the header) -
> identified by JD+filter or some other ruse - with the number of stars
> measured. Finally, do a pass through to disable all images records in
> the db which do not pass some minimum production standard.
>
> My basic point is that any workaround which requires someone to be on
> call to load CD/DVDs will prevent extensive use of the images and
> photometry from TASS. The dark-subtraction and flat-fielding are
> single-time operations. Once done, they stay done and the images can be
> used from then on.
>
> Cheers,
> Doug
>
> Tass Mailing List wrote:
> >
> > Date: Thu, 04 Jan 2007 13:40:39 -0600
> > From: Thomas F. Droege <droege@fastmail.fm>
> > To: tass@tass-survey.org
> > Subject: Image Storage
> >
> > I think the last thing we want to do is to store my archive disks on a
> > big server somewhere.  The attached jan_2_3.png is data from the last
> > two nights. It is an example of real data.  Some data was taken between
> > 0 and 200 degrees in ra.  You will notice big holes.  This is because
> > there was a near full moon on these two nights.  I just start taking
> > images at dusk, and run until dawn taking images.  About 2000 images are
> > stored on the 6 DVDs that I wrote.  Note that there is no data from tom1
> > after the moon passed.  This is because tom1 failed to find the home
> > limit switch in the bright moonlight and just died.  My guess is that
> > about 70% of these images are junk.  The pipeline just throws out images
> > where the sky background from the moon is too large, or where the
> > background is too irregular (clouds), or where it does not come to an
> > astrometric solution.
> >
> > Do we really want to store images on a public archive that are 70% junk?
> > Further, these are not dark subtracted or flat fielded images.  The
> > data to do this is on the disk, as are the .list files.
> >
> > Seems to me a better plan would be to call up the .list files and look
> > at the data they contain in order to decide whether to load an image.
> > The first thing to look at is the "bad" flag.  This means that it does
> > not meet very broad criteria for good data.  Next one can look at the
> > sky background level.  This will be high for images taken at dawn and
> > dusk.  Next we might look at the star count for this image.  One might
> > require that an image contain a high percentage of the possible star
> > count to be loaded.  Many other things come to mind. Note also that the
> > .list file contains the astrometric solution for the center of the
> > image.  The fits header just contains my guess to get the solution
> > started. The loading process might want to update the .fits header with
> > the correct values.
> >
> > This holds true for the current data, 2005, 2006, 2007.  Data from 2003
> > and 2004 has other problems.  First it was written as dark subtracted
> > and flat fielded images.  This solves one problem.  But the images for a
> > night are spread over a number of CDs.  I will have to look at some of
> > the disks, but I think they just contained images at the start.  Later I
> > saved other things if I had room.
> >
> > I think it requires some discussion before we jump ahead and load disks
> > into a file.  What do you all think?  Just because we only load
> > relatively good images to the file does not mean that the archive disks
> > are not available to a determined researcher.  The process of loading
> > the disks should produce a comprehensive catalog of all the images.
> > Someone may want to look at "moon" data which is full of streaks and
> > other funny artifacts when a star was doing something really interesting
> > on that night.
> >
> > Tom Droege
> >
> >
> > ------------------------------------------------------------------------
> >
>
>
-- 
  Thomas F. Droege
  droege@fastmail.fm