[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: new tech note


OK, some comments.  I went through this procedure, asking for
comments, for the Mk III database.  Got lots of good comments.
I'll pass them on plus some things I learned.  I used Postgresql
also.  Seems everyone here like it.

1) The orientation of the chip "belongs" to an __image__ not a
telescope.  Telescops are likely to be modified, fixed, moved
or whatever.  Also IMO the best way to descibe the orientation
is to copy the data direct from the FITS header. 
The other option is to assign each telescope a new serial number
whenever some part is swapped out,  You'll never keep track.
Best to just remember the FITS keywords from the image.
The pipeline __should__ (must?) pass the image header through
so that it shows up in the starlist header.

2) Again, where you say "for each detector".  It will be easyer to
keep this data for each image.  (quick calculation:  10 year
project lifetime, 20 cameras, 300 nights per year, 100 images
per night = 6 million images = no big deal.  6M row tables are
easy and we will not be at 6M for ten years.

3) Again, in place of "for each night" use "for each frame".  I'd
keep the information on processing along with the dataset.
There is always the option of deleting data, re-proceesing it and
adding it back in.  Say we find a bug in version 2.3 or the
data reduction software.  We will want to delete all data proceesed
by the software. 

So, given my comments above.  I'd have a table called "FRAMES"
that lists eveything we know about a frame.  Which telescope
was used, what software was used, clouds cover, the filter
the image scale and so on.  Also each frame is given an ID.

Now, where you say "for each star", I think you want to say
"for each observation of an object"  because we will likely
see the same star many times.  OK, next I think you need a
"FRAME ID" to be stored along with each row.  Now you can trace
every observation back to the camera and FITS image file.

Not done yet.

Next we need a table called "stars" or "catalog" or "index".
This has one row for ever object TASS has ever seen.  (That
is, one row for each physical star.)  We assign a arbitrary
ID to each start (use 1, 2, 3, or a name made up from the
location.  But I like 1, 2, 3 because the exact location will
change as we average more observations or as we "split" 
dobles.) Call it "TASS Catalog ID".

Going back to the table called "for each observation of an
object"  We need to assign a "TASS Catalog ID" to each
observation.  Now the "catalog" will contain summary statistics
for each star while the "observations" table contains the
data points that make up a light curve.

++++++

The algorithm I used to process a star list first added one row
to the "frames" table, just copy the file header.  next for
each row in the starlist I added it to the big "observation"
table.  Next I had to asign a "TASS Catalog ID".  So I looked
in the catalog for a close match.  If none was found I figured
this was the first time this star was seen an added a new
catalog entry.  If I found a match I'd recompute the summary
statistics for that star.

The above was not easy to implement and I still don't like the
way I did it.  A run could take many hours.  I would make a
frame a "transaction" in DBMS terms.

I also had a set of "standard" catalogs.  Get as many as you
want.  Tycho, variable star catalogs and so on.  The more the
better.  Now for each catalog you make a "Cross ID Table"
that has two colums listing matches between the Tycho ID and
the TASS Catalog ID.  

With the above you can see that with one SQL query I could 
quicky make a light curve for any object I could name or even
querry for stars with a summary statistic within some range.
like "SELECT ... where ColorIndex between 2.0 and 3.0;"






--- "Creager, Robert S" <CreagRS@LOUISVILLE.STORTEK.COM> wrote:
> 
> Michael has kindly put a new tech note up on the server:
> http://www.tass-survey.org/tass/technotes/tn0080.html
> 
> This is an attempt to document, in one location, what is required to
> be kept
> in a database for long term viability of the data.  This first pass
> of the
> document is just reduced data, gathered mainly from information Arne
> posted
> and Chris implemented.  I will update and add new section(s) when
> enough
> ideas present themselves and are agreed upon (or at least make
> sense).  I
> hope you all will take a look, make comments and the like.
> 
> Cheers,
> Rob
> 
> Robert Creager
> Senior Software Engineer
> ATS Library Engineering
> 303.673.2365 V
> 303.661.5379 F
> 888.912.4458 P
> StorageTek
> INFORMATION made POWERFUL
> 
> 
> 


=====
Chris Albertson 
  Home:   310-376-1029  chrisalbertson90278@yahoo.com
  Cell:   310-990-7550
  Office: 310-336-5189  Christopher.J.Albertson@aero.org

__________________________________________________
Do You Yahoo!?
Great stuff seeking new owners in Yahoo! Auctions! 
http://auctions.yahoo.com