[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: Data sets over time
> Tom would still have to figure out which CDs to pull from the
> list of filenames. Is this a big problem? I don't know.
>
> Writting a script is not hard - gettig the requiremnets nailed
> down is hard.
This is the hard part. If we actually want the images, then I presently have
to search over 100 cake boxes, (I just counted them) search through typically
40 or so CDs, and look at each one to read the time of day range and the JD.
Then load the CD and guess which file to view the fits header from it's
label. OK, this can be extracted from the make_list.out file so one could
have a program to check the CD directory against some master make_list.out
file to see if it contains any desirable file.
One would still have to look at every CD. But it is hard to look at a CD and
decide if it might contain a wanted file. OK, they are stored by TOM, month
and year. Best to let a program do the search since it won't make people
type mistakes. So each disk has to be loaded. While at it I figure we might
as well load all the images so that we can extract interesting image sets.
OK, I agree that we can easily make a big make_list.out file. We just have to
load 25 CDs and collect the make_list files into one big file. This can then
be searched for any image area of interest. So it is easy to get the list of
wanted images specifying an area of the sky.
Getting a particular sub set of images (If we ever want to do it) is the hard
job. If we choose to do it once then we might as well load all the images to
hard disk so that we can extract arbitrary sets.
What I was thinking was that it might be useful to be able to extract the
images from various parts of the sky for tests. We have the index, it is the
collection of make_list.out files. What is lacking is a big set of images
stored on hard drives that we could extract using the make_list.out file.
This is what I was requesting help to set up. Note that there is more images
than will fit on one disk. So someone (I am incompetent to do this) needs to
set up how the .fits files would be put in directories on several hard
drives. I imagine something like the present make_list.out file with an
appended location for the image file. Given this, even I could do the kind
of searches that I think are needed with gawk.
It is not obvious that we want to do this at all, or that any use will be
worth the effort. The assumption is that the processing will do something
different from the present pipeline or else everything could be extracted
from the .cal files.
There is at least the use of looking up an image with a problem to see if an
airplane or a satellite was going by.
Tom Droege
On Wednesday 15 September 2004 02:49 pm, Chris Albertson wrote:
> I agree with Rob. I hope the headers contain usful data.
> But isn't this already done? I thought it was.
>
> What's needed is a table with one row for each FITS file. The
> data in that row would descibe the image and would include the
> vield of view and time of day and oher date derived from the
> headers.
>
> Then there is another table that contains photometric data points
> that were produced by the pipeline. I assume each data point has
> a "pointer" the the above table so you can tell from where that
> point came from.
>
> Databse designs dating from the Mk III era contained at least both of
> the above tables.
> Given that nothing has changed, to make the time series
> you first select the set of points that are within the RA,DEC boundry
> and times of interrest (Say the "galatic center in the last two years')
> and then you ask the databse for the unique list of filename
> pointers for those points. So, the existing database should be able
> to supply the list of FITS files needed.
>
> Tom would still have to figure out which CDs to pull from the
> list of filenames. Is this a big problem? I don't know.
>
> Writting a script is not hard - gettig the requiremnets nailed
> down is hard.
>
> --- Robert Creager <Robert_Creager@LogicalChaos.org> wrote:
> > When grilled further on (Wed, 15 Sep 2004 12:56:06 -0500),
> >
> > "Thomas Droege" <tdroege2@earthlink.net> confessed:
> > > To set up a data base, the only way to get the information is to
> >
> > read each
> >
> > > fits header from each disk. So this involves loading 2500 or so
> >
> > CDs,
> >
> > > reading the headers and making up the information for the data
> >
> > base. While
> >
> > > at it, one might as well invest in a stack of hard disks and load
> >
> > the
> >
> > > images. Then one could extract images from such a file. There are
> >
> > a some
> >
> > > of tests I would like to do by processing at one time all the
> >
> > images
> >
> > > containing a particular star or region of the sky.
> >
> > Are the headers updated after the reductions are done, or are the
> > headers real,
> > real accurate? If not, then doing this will not be of much value. I
> > suspect
> > Michael R. can answer this question?
> >
> > Cheers,
> > Rob
> >
> > --
> > 12:18:54 up 10 days, 23 min, 4 users, load average: 2.38, 2.28,
> > 2.20
> > Linux 2.6.5-02 #8 SMP Mon Jul 12 21:34:44 MDT 2004
> >
> >
> > ATTACHMENT part 2 application/pgp-signature
>
> =====
> Chris Albertson
> Home: 310-376-1029 chrisalbertson90278@yahoo.com
> Cell: 310-990-7550
> Office: 310-336-5189 Christopher.J.Albertson@aero.org
> KG6OMK
>
>
>
> _______________________________
> Do you Yahoo!?
> Declare Yourself - Register online to vote today!
> http://vote.yahoo.com