RE: TASS image compression

[Thomas Droege <tdroege2@earthlink.net>]

There was a lot of discussion on this as is summarized by Herb Johnson in
TN-55.  My conclusion at the time was that we were unlikely to find a
compression scheme that would allow sending any significant number of files
over the internet.  Since then the bandwidth available to most of us on the
internet has increased by a factor of 10 or so.  However, the amount of
data that I am creating has increased also by an order of magnitude.  The
note says "a ten to one compression would allow putting 4-5 nights on one
CD."  Now a ten to one compression would allow putting 4-5 nights on 10
CDs.  I have written 198 CDs already this month.  

Mailing CDs has a very wide bandwith.  There is a large cable delay, but
this is usually acceptable since we are not trying to do much in real time.

I suggest that there are many things of more interest to tass than work on
compression.  Further, we are not down to looking at subtle effects.  Who
knows what lossy compression will do the problems that we are presently
researching.

Tom Droege


> [Original Message]
> From: Robert J. Bradbury <bradbury@aeiveos.com>
> To: TASS <tass@listserv.wwa.com>
> Date: 9/14/2004 9:55:05 AM
> Subject: TASS image compression
>
>
> I have been looking at the problem of image compression
> of the TASS data as Tom has indicated to me that there
> has been some discussion of this in the past.
>
> However what I find in the mail archives seems to be
> five or more years old.
>
> So I've decided to document and experiment a little bit
> with the data and programs which may be available.
>
> For documentation, see: [1].
>
> Worth noting is that the FITSPRESS and compFITS program
> sources seem to be unavailable [if anyone has these or
> knows who might please let me know].
>
> So, working with the HCOMPRESS program from STSCI it
> looks like TASS images can be compressed...
>
> With scale factors of 1, 2, 4, 8, 16, 32, 64 & 128,
> the hcompress approach (actually fcompress script) reduces
> the normal TASS FITS image from 8,415,350 bytes (~8 MB)
> to a fractional size as follows:
>  1: 0.48, 2: 0.46, 4: 0.41, 8: 0.35, 16: 0.29, 32: 0.23,
>  64: 0.16, 128: 0.10
>
> So, using a scale compression factor of 128 one reduces
> the size of the images by about an order of magnitude.
> The question then becomes where, if one uses a potentially
> lossy compression strategy, a reduction in the image information
> begins to significantly impact the accuracy of the data.
> (As the discussions I've seen would tend to suggest that we
> don't have a good feel for the sources of and/or long term values
> and trends of errors in the raw data.)
>
> But there has to be some level at which semi-lossy compression
> is not going to negatively impact the data coming out of the
> reduction pipeline.  Of course, when one is trying to deal
> with difficult cases (identification of binaries, etc.) one
> may want to return to the original data -- but for most of
> what one wants to do the question is can the data be dealt
> with in a compressed (noise-reduced?) format?
>
> Robert
>
> 1. http://www.aeiveos.com/~bradbury/Astronomy/AIC.html
> [If anyone has any good references to be added to this page
> please send them to me.]
>
>