MemoryError when list append... plz help

[bearophileHUGS]

above sim is floating type.
s.append is totally coducted 60,494,500 times.
but this code raise MemoryError.
My computer has 4G RAM.
i think it's enough. but it doesn't...

from array import array
a = array("d", [0.0]) * 60494500

This requires about 477 MB to me.
Note that I have not used "append" that slows down code a lot and
wastes memory, that's what I mean with "clean".
If you really need the (i,j) part too (and you may not need it), then
you can create two more "parallel arrays". Also a numpy array may be
better for such large amount of data.

Bye,
bearophile

[James Mills]

On Wed, Dec 31, 2008 at 4:17 PM, James Mills

I have no idea how many bytes of memory
storing each element of a list consumes
let alone each float object, but I assure you
it's not going to be anywhere near that of
60494500 4-bytes spaces (do floats in C
normally consume 4 bytes) in C.

Just creating a list of this many elements
consumes 60% of my desktops RAM.
YOu _SHOLD NOT_ be doing this.

L = list(xrange(N))
len(L)

60494500

L[10000]

10000

ps stats:

32351 jmills 20 0 962m 610m 1356 S 0.0 60.6 0:03.36 python

It takes several seconds to create this list
and several minutes for the Python interpreter
to shutdown and deallocate all the memory.

--JamesMills

[Gabriel Genellina]

En Wed, 31 Dec 2008 06:34:48 -0200, Steven D'Aprano

((i, j), sim)
where sim is a float and i and j are ints. How much memory does each of
those take?

sys.getsizeof( ((0, 1), 1.1) )

32

(On Windows, 32 bits, I get 36)

So each entry requires 32 bytes. 60 million times 32 bytes = almost 2GB
alone. Plus the list itself will require (approximately) between 230MB
and 460MB just for the pointers.

That was just the size of the "outer" tuple; you have to add the size of
each element too. First one is another 2-item tuple (36 bytes too) plus
its elements (two integers, 12 bytes each). Second element is a float and
takes 16 bytes. Total: 112 bytes per item; the final size may be a bit
smaller because some objects may be shared (e.g. small integers)

[Steven D'Aprano]

======================
s=[]
....
s.append(((i,j),sim))
======================
above sim is floating type.
s.append is totally coducted 60,494,500 times. but this code raise
MemoryError.
My computer has 4G RAM.
i think it's enough. but it doesn't...

Your computer might have 4GB, but how much memory can Python allocate?
What operating system are you using?

Each time you are appending to the list, you append a tuple:

((i, j), sim)

where sim is a float and i and j are ints. How much memory does each of
those take?

sys.getsizeof( ((0, 1), 1.1) )

32


So each entry requires 32 bytes. 60 million times 32 bytes = almost 2GB
alone. Plus the list itself will require (approximately) between 230MB
and 460MB just for the pointers.

What are you expecting to do with this enormous list, and why do you need
it all at once?

[Francesco Bochicchio]

======================
s=[]
....
s.append(((i,j),sim))
======================
above sim is floating type.
s.append is totally coducted 60,494,500 times.
but this code raise MemoryError.
My computer has 4G RAM.
i think it's enough. but it doesn't...
So, i've tested below code.
======================
a=[]
i=0
a.append(i)
i+=1
======================
but this code raise also MemoryError.
How can i resolve this problem?
please, help...
Regards,

If you _really_ have to store so many numbers in memory (hint: if you
are processing them sequentially, you don't need to store all them - use
generators instead) then you may have better luck using mmap module to
create a huge file-memory object, that you can access both as a file and
as a list, and put numbers in it after packing/unpacking with struct.

memory = mmap.mmap(-1, 60494500*4)

... memory[offset*4:(offset+1)*4] = struct.pack( "%f", f )
... return struct.unpack( "f", memory[offset*4:(offset+1)*4] )[0]

memory_put(12, 3.14 )
memory_get(12)

3.1400001049041748

Ciao
------
FB

[BON]

======================
s=[]
for i in range(11000-1):
for j in range(i+1, 11000):
....
s.append(((i,j),sim))
======================
above sim is floating type.
s.append is totally coducted 60,494,500 times.
but this code raise MemoryError.

My computer has 4G RAM.
i think it's enough. but it doesn't...

So, i've tested below code.
======================
a=[]
i=0
while i<60494500 :
a.append(i)
i+=1
======================
but this code raise also MemoryError.

How can i resolve this problem?
please, help...

Regards,

[James Mills]

(Sorry for top posting):

You are mad! Why on God's earth would you want
to create a list containing 60 MILLION elements ?

What is the use case ? What are you solving ?

You may have 4G of ram, but I very seriously
doubt you have 4G of ram available to Python.

I have no idea how many bytes of memory
storing each element of a list consumes
let alone each float object, but I assure you
it's not going to be anywhere near that of
60494500 4-bytes spaces (do floats in C
normally consume 4 bytes) in C.

--JamesMills

--
-- "Problems are solved by method"

======================
s=[]
....
s.append(((i,j),sim))
======================
above sim is floating type.
s.append is totally coducted 60,494,500 times.
but this code raise MemoryError.
My computer has 4G RAM.
i think it's enough. but it doesn't...
So, i've tested below code.
======================
a=[]
i=0
a.append(i)
i+=1
======================
but this code raise also MemoryError.
How can i resolve this problem?
please, help...
Regards,
--
View this message in context: http://www.nabble.com/MemoryError-when-list-append...-plz-help-tp21227745p21227745.html
Sent from the Python - python-list mailing list archive at Nabble.com.
--
http://mail.python.org/mailman/listinfo/python-list