Re: Common and uncommon goals and software

[Tom Droege <tdroege@veriomail.com>]

Dear Robert and all,

I am a good German.  There is nothing that would please me more that a 
bunch of telescopes lined up in a neat row, and a line of computers all 
lined up behind them crunching the data in sync.  (Actually this is a lie, 
as anyone who has seen my workroom can attest.  I would like to see things 
working neatly in rows, but I am afraid that I can always find something 
more important than neatness.)

I can't do this by myself.  So I have to turn to others.  I would not want 
people working on this project that would take orders and line up in neat 
rows.  The most probable result of finding such a group is that they would 
efficiently process the wrong task.

So I have to live with the real people, like yourself, that I find on the 
net.  I am not the leader of tass.  I hope there is no leader.  If one 
springs up I will probably try to kill him/her.

I am operating on the "Field of Dreams" concept of management.  If you 
build a bunch of telescopes someone will do something useful with them.  My 
only control is to build them so that they are not good for much besides a 
survey.

So I will only speak for myself.  I hope to be able to copy someone's 
pipeline that is doing something I think useful.  Then I will shove my data 
through it.

I know this seems strange.  I would rather have a bunch of good people 
doing all different things than some mediocre people doing a directed 
project.  The real hope is that we will sort of get together.  I think we 
will.  If only half of us join up, then it will still be a powerful 
project.  About half of the Mark IIIs that got sent out produced a useful 
amount of data.  They would probably still be working if the Mark IV 
project had not killed the incentive to run them.  I hope to do better with 
the Mark IVs, but we shall see.

This written by the non astronomer Tom

Rob, thanks for your result on Disk 16.  I have 60 more disks like it 
sitting on my desk.  We could team up and produce a nice paper.  I could 
easily run for a year running this machine right where it is covering the 
same band of sky over and over.  If you have already set up a pipeline that 
reduces this data to star lists, we could just run it through and produce 
material that would generate a nice list of variable stars.  It would be 
still better if someone would join up with us to provide guidance for the 
photometry so that it would be as good as it can be.  I don't know how to 
do this, but there are several on the list that do.

As I build the hardware, I want to push some data through the system.  What 
say, out there who wants to join up with me and start some data 
analysis.  Rob, will you join up?  Then we just need an astronomer to guide 
the photometry.  I am not talking about the ultimate data reduction.  Just 
look at the -1 to +3 (roughly) band where I have already collected a lot of 
data and try to push it through to an output.

Note that if the pipeline is set up and proven to work, I am willing to do 
the dog work of processing the data disks.  I would propose a simple goal:

1) Process the data disks to star lists with best effort photometry.
2) Merge the star lists so that we have all the measurements of the same 
star grouped together as good as we can do it.
3) Search the star lists for obvious variables.
4) Produce plots of the results from 3)
5) Make a first pass at classification of what we find.  I have someone in 
mind to work on this if we get this far.
6) Write an IAPPP paper.

OK, this is not a "final solution".  It would be a nice project.  We would 
learn a lot.  It would make a nice IAPPP paper.  I propose this as a 
non-astronomer project, but I asking for astronomer guidance.  What 
say?   I am itching to start processing the data that is piling up.

Tom Droege

At 12:16 PM 11/3/00 -0700, Creager, Robert S wrote:

>Hello all,
>
>I've a question which will either beat me into submission, or something
>else.  When folks are talking about data reductions, the thread seems to
>ends up about how everyone is doing something different, and if a data
>reduction pipeline is set up, it's possible that only the person who set it
>up will use it.
>
>Shouldn't there be an effort to generate a generic data reduction pipeline
>which everyone runs on the data taken from their camera?  This pipeline
>would be completely automated once set up, such that it could be run from a
>cron job (Linux) or from the windows scheduler.  This would only burden a
>CPU, and not the person running the camera, leaving that person to do the
>data reductions specific to their goals.  Presumably this data would end up
>in some kind of database (ASCII flat file, PostgreSQL, Oracle...).  If
>anyone had access to a machine with the storage/bandwidth necessary to be a
>central database server, syncing could also be automated.  Otherwise,
>sending out CD's with database information every now and then would
>accomplish the same with minimal user intervention.
>
>"The non-astronomer" Rob
>
>PS - I ran SPS2 on my Sparc Ultra 5 (with parameter data from the SPS
>validation information), and it crunched all the data on disk 16 in 8494
>seconds (80 images at 106 seconds per image).  The log file indicates about
>60 seconds between pictures (is this data download?), so the crunching is
>near real time (for SPS at least) on a reasonably fast computer.