New FITS->Postgres converter program

[Chris Albertson <chrisja@jps.net>]

I wrote something that maybe someone else can use.  We are storing
the processed TASS data in a PostgreSQL database.  I thought it
would be nice to also store as many "standard" catalogs as possible
in the same database.  This should allow us to correlate our data
with those other catalogs.  I have a set of the ADC CD-ROMs these
contain hundreds of catalogs of not only photometry and
astrometery but spectra, radio sources and you name it.  The ADC's
coverage is broad if not 100% up to date.  I thought it would be
good to import the entire set of catalogs into the Postgres database.
With disk space selling for $25/gigabyte why not?
For more info about these CDs see
http://adc.gsfc.nasa.gov/adc/adc_other_media.html

I've completed the first step:  Written a program which will
automatically import a FITS format table into a Postgres table.
The program is written in C and will certainly run on any UNIX/
Linux computer and likely can be compiled on an MS Windows (win32)
system as well.  Get program at
ftp://ftp.logicon.com/open/fits2pg-0.1.0.tgz 
I'll move it to my web page once my ISP gets it's act back together.

Here is the README file:

+++++++++++++++++++++++++++ README FILE +++++++++++++++++++++++++++
I wrote this program because I wanted to import most of the FITS
data catalogs contained NASA's ADC CD-ROMS (NASA Goddard Space Flight
Center, Astronomical Data Center, NSSDC "Selected Astronomical
Catalogs" Volumes 1 through 3) into a PostgreSQL DBMS.  I did not want
to have to type in the SQL commands to create a table then read in the
data for each file as there are a great many files each in it's own
unique format.  This program will look at a FITS tabular file and
determine the number of columns it contains, the data type and heading
of each column.  It will then create a Postgres Table with analogous
characteristics.  Finally it copies the data from the FITS table to the
newly created database able.  Any number of FITS table files may be
specified on the command line or a "wild card" may be used (i.e. *.fits)
to specify the files to be processed.  In this way an entire directory
or entire CD-ROM may be processed at once.

This is the first release of this program.  It has been tested only
under Linux on an Intel Pentium (200Mhz, 96MB RAM).  It should run on
any computer that supports both Postgres and the "cfitsio" library.
Any Linux/UNIX machine should have no problem and with later versions
of Postgres, the MS Windows (win32) platform is supported.  On my test
machine it is not fast.  A typical 20 column table is processed at a 
rate of about 700 rows per minute.  With some effort this could be 
improved by a factor of 5x or more by switching from the use of multiple
SQL INSERT statements to a single COPY in statement. 

This program may have use other than as a way to import the ADC CD-ROMs.
I'd like to see us agree to write all tabular TASS Mk IV data products
in FITS Table format.  If this is done then the "database import
problem" is solved by this program. 


To build this fits2pg unpack the tar file change to the source
directory and type "make depend" followed by "make".  If you
want you can (as root) also type "make install" to copy the executable
to where it belongs.

To run the program you will need to enter at least a couple command
line options.  For a quick summary of the options use the "-H" or
"--help" option.  Here is a sample output:

------------------ SAMPLE ------------------------------
[chris@rabbit]$ ./fits2pg --help
usage: fits2pg [option ...] [FitsFileName ...]

  -H, --help             Will print this message
  -V, --version          Will print the version ID
  -d, --dbname=NAME      Name of the database to be used
  -h, --host=HOSTNAME    Name of machine running Postgres server
  -m, --msglevel=NUMBER  0=quiet, 1=verbose, 2=debug
  -p, --prepend=P_STRING table = P_STRING+basename(fitsfilename)+
  -a, --append=A_STRING          A_STRING

Following the options may be any number of filenames.  These must
contain valid FITS format ASCII or binary format tables.

fits2pg will create a Postgresql table for each FITS table.  One
table per FITS file.  Column names and data types are presevered

----------------- END SAMPLE ------------------------------

Notice that the Postgres table name is derived from the FITS 
filename by first dropping any extemtion (.fts, .fit,...) and
then prepending or appending a user supplied string if such a
string is given.  The "-d" option is always required.  If the
"-h" optionis not supplied fits2pg will asume that the Postgres
database server is running on the local machine.  If that is 
notthe case use "-h".  The default message level is "verbose".

The program fits2pg is free software; you can redistribute it 
and/or modify it under the terms of the GNU General Public 
License as published by the Free Software Foundation; either 
version 2, or (at your option) any later version.  The full
text of this license is contained in the file called "COPYING"
that should have supplied along with this README file. 


If you have any comments, or bug reports send e-mail

   --Chris Albertson             home: chrisja@jps.net        
     Redondo Beach, California   work: chris@topdog.logicon.com