Re: [S] Problem reading large files.

Andreas Krause (Andreas.Krause@genedata.com)
Fri, 16 Oct 1998 14:12:36 +0200


Hi.
As I have the same equipment, Splus 3.4 on an SGI, I tried your example.
x <- rnorm (100000)
write (x, "x", n=1)
unix.time (xx <- scan ("x"))
[1] 4.25000000 0.09000003 4.00000000 0.00000000 0.00000000

So it takes 4.3 seconds to read in the data on our machine without changing the
object size.
Reading in 500,000 values took 24 seconds and I had to set
options (object.size=1E+10)
Splus didn't occupy more than 15 Meg in memory.

andrey@utstat.toronto.edu wrote:
>
> Hi!
> I am running S-PLUS Version 3.4 Release 1 for Silicon Graphics Iris, IRIX 5.3.
> i have a large ascii file, over 100,000 lines long, one number on each line:
> 0.766007
> 0.916291
> 0.924875
> 0.924710
> 0.914394
> 0.042228 etc etc...
> When I try to read this in:
> my.vector _ scan("my.file") I get the error message:
> Error in scan....Cannot allocate 5242880 bytes: options("object.size")
> is 5000000: see options help file...
>
> So I did:
> options(object.size=10000000) and even much larger numbers,
> but s-plus keeps complaining; e.g.:
>
> Cannot allocate 167772160 bytes:
> options("object.size") is 100000000: see options help file
> and so on.
>
> If I use much higher than 10000000, it seems to jam our shared UNIX facility.
>
> 100,000 is not that large, so it should be possible to read this in,
> convert it to a matrix, etc.
>
> Any suggestions?
>
> Thanks in advance,
> Andrey Feuerverger / Dept Statistics / Univ or Toronto
> andrey@utstat.toronto.edu
>
> -----------------------------------------------------------------------
> This message was distributed by s-news@wubios.wustl.edu. To unsubscribe
> send e-mail to s-news-request@wubios.wustl.edu with the BODY of the
> message: unsubscribe s-news

-- 
Dr. Andreas Krause, GeneData AG, Postfach 254, CH-4016 Basel
mailto:Andreas.Krause@genedata.com   http://www.genedata.com
-----------------------------------------------------------------------
This message was distributed by s-news@wubios.wustl.edu.  To unsubscribe
send e-mail to s-news-request@wubios.wustl.edu with the BODY of the
message:  unsubscribe s-news