Re: [S] Problem reading large files.

Cande V Ananth (ananthcv@epi.umdnj.edu)
Fri, 16 Oct 1998 08:33:10 -0400


> From Andreas.Krause@genedata.com Fri Oct 16 08:22:56 1998
> Date: Fri, 16 Oct 1998 14:12:36 +0200
> To: andrey@utstat.toronto.edu
>
> Hi.
> As I have the same equipment, Splus 3.4 on an SGI, I tried your example.
> x <- rnorm (100000)
> write (x, "x", n=1)
> unix.time (xx <- scan ("x"))
> [1] 4.25000000 0.09000003 4.00000000 0.00000000 0.00000000
>
> So it takes 4.3 seconds to read in the data on our machine without changing the
> object size.
> Reading in 500,000 values took 24 seconds and I had to set
> options (object.size=1E+10)
> Splus didn't occupy more than 15 Meg in memory.

The same exercise took 5.54 and 27.85 seconds, respectively, for
reading in the data sets with 100,000 and 500,000 observations.
I'm running S-Plus 3.4 on the UNIX.

Obviously, it all depends on the available RAM on the system, and
the allocated swap space. As I had indicated to Dr Feuerverger
(andrey@utstat.toronto.edu) in a personal communication, I believe
it all has to do with the amount of swap space that's available on
the system. Please correct me if I'm wrong, or if there are other
(more elegant) ways of handling this problem.

--
Cande V. Ananth
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Assistant Professor                       Tel: (732) 235-7940
Div of Epidemiology and Biostatistics     Fax: (732) 235-6627
Dept of OB/GYN and Reproductive Sciences  ananthcv@EPI.UMDNJ.EDU
UMDNJ-Robert Wood Johnson Medical School  http://www.rwj-obgyn.umdnj.edu/
125 Paterson St, New Brunswick NJ 08901      homepages/Ananth/ananth.html   
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-----------------------------------------------------------------------
This message was distributed by s-news@wubios.wustl.edu.  To unsubscribe
send e-mail to s-news-request@wubios.wustl.edu with the BODY of the
message:  unsubscribe s-news