Re: Image Storage

[Tass Mailing List <tass@mail.alembic.net>]

Date: Fri, 05 Jan 2007 21:29:48 +1000
From: Peter Stewart <peterstewart67@optusnet.com.au>
To: Tass Mailing List <tass@mail.alembic.net>
Subject: Re: Image Storage

Hi Tom,
as a long time lurker on this list, not a participant, I don't have any
say. But as an amateur fiddler with all things astro, I'd love to have
ready access to the images taken by the TASS project. 70% may be junk,
however they are tagged as junk, and people accessing the database could
discount them if they wish.

You have data, and who can tell at this point how somebody in the future
may make use of that data.... even the "junk" data. If there is a server
that is ready to host these images, and you post them up with all the
information as to how they were obtained, any known problems etc, then I
think that should be done.  Surely that is the point of science, make
the information available, and people will use it, normally in ways you
cannot predict. Let the users of the data work out how to process them,
but make them available in the raw form.

On the purely practical side, it would be a lot easier to index 200,000
images on one server that spread across thousands of disks stacked on
shelves.

Anyway Tom, keep up the great work, it is great to see that you are
still hammering away at the project

Regards,

Peter Stewart,
Brisbane, Australia.
> Date: Thu, 04 Jan 2007 13:40:39 -0600
> From: Thomas F. Droege <droege@fastmail.fm>
> To: tass@tass-survey.org
> Subject: Image Storage
>
> I think the last thing we want to do is to store my archive disks on a
> big server somewhere.  The attached jan_2_3.png is data from the last
> two nights. It is an example of real data.  Some data was taken between
> 0 and 200 degrees in ra.  You will notice big holes.  This is because
> there was a near full moon on these two nights.  I just start taking
> images at dusk, and run until dawn taking images.  About 2000 images are
> stored on the 6 DVDs that I wrote.  Note that there is no data from tom1
> after the moon passed.  This is because tom1 failed to find the home
> limit switch in the bright moonlight and just died.  My guess is that
> about 70% of these images are junk.  The pipeline just throws out images
> where the sky background from the moon is too large, or where the
> background is too irregular (clouds), or where it does not come to an
> astrometric solution.
>
> Do we really want to store images on a public archive that are 70% junk?
> Further, these are not dark subtracted or flat fielded images.  The
> data to do this is on the disk, as are the .list files.
>
> Seems to me a better plan would be to call up the .list files and look
> at the data they contain in order to decide whether to load an image.
> The first thing to look at is the "bad" flag.  This means that it does
> not meet very broad criteria for good data.  Next one can look at the
> sky background level.  This will be high for images taken at dawn and
> dusk.  Next we might look at the star count for this image.  One might
> require that an image contain a high percentage of the possible star
> count to be loaded.  Many other things come to mind. Note also that the
> .list file contains the astrometric solution for the center of the
> image.  The fits header just contains my guess to get the solution
> started. The loading process might want to update the .fits header with
> the correct values.
>
> This holds true for the current data, 2005, 2006, 2007.  Data from 2003
> and 2004 has other problems.  First it was written as dark subtracted
> and flat fielded images.  This solves one problem.  But the images for a
> night are spread over a number of CDs.  I will have to look at some of
> the disks, but I think they just contained images at the start.  Later I
> saved other things if I had room.
>
> I think it requires some discussion before we jump ahead and load disks
> into a file.  What do you all think?  Just because we only load
> relatively good images to the file does not mean that the archive disks
> are not available to a determined researcher.  The process of loading
> the disks should produce a comprehensive catalog of all the images.
> Someone may want to look at "moon" data which is full of streaks and
> other funny artifacts when a star was doing something really interesting
> on that night.
>
> Tom Droege
>