Re: Harnessing Unused Computer Power Over A Network

[Tom Droege <droege@wwa.com>]

Paul,

We don't really have a problem for which this is appropriate.  At present,
on a 100MHz Pentium it takes me only 44 seconds to process a file that is
taken in 15 minutes.  Even running a triplet the processing time is 1/7 of
the data acquisition time.   It seems better to process the data first
before moving it anywhere.  

The challenge is learning how to run the data base.  Now that we have 10
million or so measurements we can start facing the real problems.    

An interesting problem is to consider making the data base distributed.

Tom Droege  

At 02:22 PM 5/7/98 +0200, you wrote:
>Hello,
>
>Using a "farm" of machines to solve a large problem has been with us for
>quite long. Two main products, both PD, are: pvm and mpi . They are 
>available through netlib.
>
>But the question is: can you use one or the other for TASS? 
>The answer depends on:
>
>- can you split the work in small, more or less independant pieces?
>- what are the relations between these pieces?
>- how much data are to be transfered to and from the remote machine 
>  (input and output) ?
>- how much computation is needed for every one piece?
>
>The ideal case is when the amount of transfered data is very small and
>the computation time quite long. Then even a slow email through phone
>lines can be used. If to the contrary you have a small computation time
>on a large data set, then a symetric multiprocessor is probably the
>only acceptable solution. The DES Challange, the finding of new larger 
>prime numbers (or Mersens prime numbers etc) are good example of the 
>first case (widely distributed network). In the past, we ran Monte-Carlo
>simulations in galactic dynamic involving more than 30 Sun and Vax stations
>situated 500 Km apart, and linked through 64 kbps phone lines.
>
>In the extreme case where all small pieces are not related to each other,
>then an other PD product can be used: NQS (Network Queuing System), whose
>goal is to "balance" the load between many different processors on a network.
>But again the computation should be significant compared to the transfer 
>of data. 
>
>With NQS, every station has the same status, any station can
>accept work form any station, and the balancing takes into account the 
>relative power of each one.
>
>With pvm and mpi, the work is distributed from one central station to
>the various "slaves".
>
>In both cases, the number of stations involved is essentially unlimited.
>
>As a simplified, dedicated version of NQS, look at crack, the PD program
>used to crack password files (the goal really is to find weak passwords).
>The work is also distributed to a set of machines using rsh only and 
>according to their relative power. Data are passed to other machines 
>using files. The data transfer is very small compared to computation 
>time. Tass could develop a similar scheme.
>
>Best regards and whishes for your fascinating project!
>                                                          Paul
>
>    ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>    +  Paul Bartholdi                 Paul.Bartholdi@obs.unige.ch  +
>    +  Observatoire de Geneve    tel  +41 22 755 2611              +
>    +  CH-1290 Sauverny          fax  +41 22 755 3983              +
>    +  Switzerland               www  http://obswww.unige.ch/      +
>    ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
>
>
>