Re: Questions about placing TASS images into on-line database

[Tass Mailing List <tass@mail.alembic.net>]

Date: Wed, 16 Jan 2008 08:17:53 -0500
From: arne <arne@aavso.org>
To: Tass Mailing List <tass@mail.alembic.net>
Subject: Re: Questions about placing TASS images into on-line database

Tass Mailing List wrote:
> 
> Date: Tue, 15 Jan 2008 09:43:39 -0500
> From: Michael Richmond <richmond@stupendous.cis.rit.edu>
> To: tass@mail.alembic.net
> Cc: mwrsps@rit.edu
> Subject: Questions about placing TASS images into on-line database
> 
>
>   I visited Tom recently and acquired copies of all the TASS
> images taken over the period 2003-2007.  Some are raw, some are
> reduced.  The total volume is about 3 TB.
>
>   I am thinking about reducing the raw images, then placing
> all the corrected images into some sort of archive which is
> available to users via a web page, with some facility for
> searching and displaying the images.  As an enhanced goal,
> it might be nice to try to co-add images in some places where
> many exist.
>
>   I can handle the correction of the raw images, and probably
> the creation of co-added images (if that becomes reasonable).
> However, I don't have much experience with such a large data
> volume.  Can others on the mailing list help?
>
>   What I'd like to know is:
>
>      - what sort of hardware would you recommend for the job?
>
>      - should I simple purchase a unit which can store all the
>             information (like an NAS), or try to build one
>             by purchasing separately drives, an enclosure,
>             and perhaps a controller card?
>
>      - hardware or software RAID?  I'm leaning towards software,
>             but seek the voices of experience
>
>      - any suggestions for the software to connect a database
>             to a web browser?  Note that this database
>             will be read-only
>
>   Suggestions?  Comments?  Guesses at an overall cost?  I welcome
> all input.  If we start to go into really gory details, it might
> make sense to take the discussion off the E-mail list and conduct
> it via private messages, but let's see if that becomes necessary.
> 
We run a pair of 3.5TB RAID5 systems based on 500GB hard drives.
This is primarily FITS image storage, so similar to what you want to
do.  This was the best break point for pricing; such a system will
cost about $4K today.  Going 7TB by using 1TB drives will add
another $1-2K.  Be sure to purchase a spare drive for maintenance.
Talk to Aaron if you want the details of the actual systems.

The best way to handle FITS images, IMHO, is to extract the headers
and build a database around their keywords.  Then lossless compress
the images (we've found bzip2 to work best) and store them on the
file system.  Let people download the compressed images and expand
them on their local system.  Don't try to provide display or analysis
functions.  You will have enough problem building the header database
and the GUI access.

My feeling would be that hosting the images at RIT would be the best
choice, but if you run into problems, let me know and we could probably
handle another computer here.
Arne