Re: Databases, was Data Processing Switch

[Robert Creager <Robert_Creager@LogicalChaos.org>]

On Mon, 2 Jun 2003 16:56:25 -0700 (PDT)
Chris Albertson <chrisalbertson90278@yahoo.com> said something like:

> 
> Rob,
> 
> So, what kind of performance do you see, How long to import 
> 1E7 observations

Depends.  If importing all measurements (i.e. all stars, not those just seen both bands), I get from 6000 to 9000 per second (using COPY).  I've just recently added a step to only import stars which match in both bands, and that slows me down to about 1 image set per 10 to 30 seconds (around 100 per second tops), but I've not yet investigated ways to improve that dB performance (I believe I can).

I've just started trying to trim what goes in, as the UPDATE I run to calculate the magnitude from color, zero point and instrumental magnitude was taking over 1 hour.  When I dropped the number of stars and bumped up some memory usage in the postgresql backend, the update gets done in 1 to 2 minutes, so I'm leaning the route of triming the data to what gets seen in both bands.  I just have to figure out the best way to do that.

The dB is running in a dual AMD 2600+ with 2Gb RAM and 36Gb 10K SCSI disk (although it will be migrated to an 200Gb ATA disk later).  I've been able to run through the entire pipeline (reduce, import, calculate, update) in around 6 hours on this machine (guesstimate, as I usually reduce, then watch the dB stage).

I'm slowly learning about how to tune a dB system.

> 
> I exchanged some e-mail with Michael Sallman and it looks like
> he could without to much effort point his very nice looking
> Java based web page at a replacement backend DBMS.  What you
> are doing could make a nice replacement.  All that would be
> required is to define a "view" that emulates his table.

That would be interesting to try.  I'm contemplating getting wireless up here to handle making my dB server 'public' (satellite has no outbound to speak of), and it would be nice to not have to do all that nice GUI work ;-)

> 
> Do you have this on a CVS server yet.

On my local machine...  I don't use CVS enough to do well with it.  If only I could get ClearCase here at home.  I'm thinking your're thinking SourceForge?

Cheers,
Rob

-- 
O_

PGP signature