[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: Introduction and some comments ;-)
I've been wanting to work that way for some time. It seems however
most people on this list are used to the "one person, one program"
programming model. The only way anything gets done using that method
is if there is a very smart person with much time on his hands.
That combination is uncommon. You get more done when you can put
the skills of several people together.
I'd like to use a model from some successfull free softwre project.
Some good, well run projects, IMO, are the Postgresql DBMS, the KDE
project and others. What they have i common is use of good tools
for version control, intergroup communication and testing and have
a limited structure at least a method for decision making. All you
really need is a CVS server, e-mail list, bug tracking system and
an FTP/web site. www.sourcefourge.net provifes this to anyone who
will sign up.
You also need people wanting to work this way.
You also need well stated (written) goals. I'll propose a few:
1) It is not enough to have a working pileline. We need a turnkey
system that can be used by software non-experts.
2) It should have multiple internal stable interfaces to facilitate
swaping parts in and out without effects the non-swapped parts.
2a) one example: All astrometric catalogs used by the system
will have a standard format.
Yes, I do agree a Databse of some kind is the end result.
Starlists are an intermediate result
--- Mark Pitts <PITTS@shands.ufl.edu> wrote:
> Tom said:
> >I actually think that there is a good chance that you or someone
> else could
> >put together an analysis group. I get lots of mail from people who
> want to
> >join a collaborative effort. If you announce one, they might sign
> on.
>
> Tom and all,
>
> If you build it, they will come? ;-)
>
> I wondered what sort of response I'd be stirring up with that post!
> =)
>
> The reason I suggest a collaborative effort is that it seems no one
> person seems to possess everything necessary to pull off a
> large-scale pipeline. It would take time (!), the requisite computer
> skills, a good understanding of the raw data, appropriate computer
> hardware and software, a good understanding of the photometric
> transformations necessary to reduce the data accurately, a good
> understanding of the hoped-for end result, and probably a couple of
> other things I haven't thought of.
>
> As a learning tool, I'm preparing to have a go at getting the Michael
> Richmond pipeline running on one of my Linux boxes. I am NOT looking
> forward to it (nothing against you, Michael!). But the thought of a
> multiple-day hair-pulling session trying to download and build the
> requisite packages from source, editing configuration files, tweaking
> this, cursing that, copying files from CD's into the (hopefullly!)
> right directories, etc etc ad nauseum is simply not an appealing
> prospect. It's also not very practical if the goal is to process
> (and actually use) terabytes of data eventually.
>
> This thread started with the idea of a "SETI like" mechanism for
> processing TASS data. In my opinion, the key to SETI's success is
> that, for the participant, installing and running the SETI software
> is remarkably simple,and, as far as I can tell, nearly impossible to
> screw up. I have always believed that the study of engineering
> genius is the study of simple solutions to complex problems.
>
> When I think about this, I have this vision in my head of a database
> containing, for a given piece of the sky, the original raw data, the
> intermediate products of the reduction process, and the end results -
> normalized such that a comparison over time is as close to
> "apples-to-apples" as we can come. The data could be queried for
> analysis, either directly or in an automated fashion. Have a look at
> the MAST DSS query form for a *rough* idea of what I mean.
> (http://stdatu.stsci.edu/cgi-bin/dss_form) Storing the data in this
> fashion leaves the door open to all kinds of possibilities you won't
> find from a stack of CD's or a collection of files in a Linux
> filesystem. It has the *key* benefit of simplifying the data
> handling, making a "lights-out" reduction pipeline easier to
> implement. It also makes multiple passes of the reduction pipeline
> much easier to accomplish in a consistent fashion. It simplifies a
> great many things.
>
> Now, I have a Linux server with 384MB of RAM sitting 'round doing
> nothing, and just yesterday I purchased a 60GB hard drive for the
> express purpose of installing it in that server to support my
> experimentation with the MR pipeline. I also obtained an evaluation
> copy of DB2 for Linux, with an eye toward experimenting with a
> database for the TASS data.
>
> However, there is no way I can go much further than that by myself.
> I'm really good at dealing with systems, but I'm a novice at this
> application. Hence the need for a collaborative approach. To be
> honest, I don't think it's going to be all that difficult. As Chris
> pointed out, much of what we need can be created by copying and
> modifying readily available modules.
>
> What are your thoughts? Who's interested in embarking on such a
> journey with me?
>
> Mark
>
>
>
>
=====
Chris Albertson
Home: 310-376-1029 chrisalbertson90278@yahoo.com
Cell: 310-990-7550
Office: 310-336-5189 Christopher.J.Albertson@aero.org
__________________________________________________
Do You Yahoo!?
Great stuff seeking new owners in Yahoo! Auctions!
http://auctions.yahoo.com