[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Lots of data

--- "Creager, Robert S" <CreagRS@LOUISVILLE.STORTEK.COM> wrote:
> 
> Hey folks,
> 
> I started importing lots of data last night.  It was easier than I
> thought
> to re-import the same data over and over and over... again. 

I don't know the details of Postgresq's quadtree index algorithm
but with identical data I suspect a kind of unnatural tree will
be built.  I like your idea of adding an offset to the whole
frame I'd mix small (arc seconds) offsets with large (degrees)
offsets.  This way your database will simulate a real one.  My
guess is index tree depth has an effect on timing.  Using identical
data may create trees that look more like linear lists (one legged 
trees) but maybe not as they claim to use a "balanced quad tree"

keep data so you can plot processing time vs.
size of database.  It would be good to know the shape of the
curve. linear, log or whatever.  In fact, I think the shape
is of more interest then the absolute value.

On another subject, if all you want is a large database why use
such a slow method to populate it?  I would just start a
COPY and then write random data.  This typically goes at
500 rows per second.  In fact you could run this at the same
time as your current test.

One more question:
What command line options do you use for Postgresql?  Specifically
the -B and -F (buffers and fsync) options.  Actually the fact that
you are CPU bound ssys you have these about right as CPU bound is
where you want to be.
  

> Currently I'm
> set up to import the data twice (520k stars total), then crunch it,
> then
> repeat 240 times (yeah, right).  Currently, I'm literally just
> reusing the
> same data, with a random filename (one of my unique parameters). But
> I'm not
> sure if this is the right way.
<snip>

=====
Chris Albertson 
  Home:   310-376-1029  chrisalbertson90278@yahoo.com
  Cell:   310-990-7550
  Office: 310-336-5189  Christopher.J.Albertson@aero.org

__________________________________________________
Do You Yahoo!?
Send FREE Valentine eCards with Yahoo! Greetings!
http://greetings.yahoo.com