RE: Storage breakthrough?

[Thomas Droege <tdroege2@earthlink.net>]

Robert,

I use 9 computers to process the data stream from the 3 dual telescopes.

3 are used to take and store the night's data.

6 are used to process the data.  The scheme is to break the night's run
down the middle and process one telescope with two computers.

It takes about 6 hours to process the data from one night.

The telescopes are started at dusk, and at dawn programs are started that
copy the resulting images from the 3 data taking computers to the 6
processing computers. 

This scheme has several advantages.  The raw data can sit on the data
access computers until the next clear night.  This means there is always 24
hours to complete the processing.  Since there are several processing
computers, if one dies, it's work can be transferred to one of the other
computer.  I also keep a hot running spare which is loaded with the
pipelines so that it can take on the job if a computer fails.  The
processing computers are mostly 1600 to 2600 Athlons.  The data taking
computers need only 400 MHz or so capacity but they need ISA slots.  These
are getting hard to find though I just found a modern 2600 Athlon that I am
saving as a spare.  The data taking computers run Windows 98 while the
others run linux.  I will switch over to linux everywhere as soon as I work
out how to do it.

I seem to be losing a computer every couple of months.  

The results are written to the processing computer and two other computers
on the net to try to avoid loss of processed data.  The dark subtracted and
flat fielded images are saved to CD.  Just writing CDs takes several hours.
I will believe in other media when it has been proven to last for 10 years
or so.  This will take 10 years and I doubt I will be around to make the
changeover. 

As I bring two more telescopes on line, I am adding to the CPU list.

Tom Droege


> [Original Message]
> From: Robert J. Bradbury <bradbury@aeiveos.com>
> To: TASS <tass@listserv.wwa.com>
> Date: 8/26/2004 9:11:30 AM
> Subject: Storage breakthrough?
>
>
> You may be interested in the development of holographic
> data storage devices by Optware [1,2].  The media may be a bit
> pricey due to the dichroic layer but at 200-1000 GB/disk
> it will probably be worth it.
>
> At least in a couple of years it looks like one will be able
> to store the TASS database on a much smaller stack of "CDs".
> It would also appear likely that libraries are going to get
> significantly smaller...
>
> On a different topic... can someone send me information
> on the current data reduction hardware and CPU load
> requirements.
>
> Robert
>
> 1. http://www.optware.co.jp/english/what_040823.htm
> 2. http://slashdot.org/article.pl?sid=04/08/25/1639224
>
>