mmap resizing macosx unix

[David Pratt]

Hi. I have been experimenting with mmap recently. I determined how to
read and write properly from it and so search and replace on large
files. The problem I am having is with replaces that are larger than
the mmap. In this instance I need to

* rewind
* resize the mmap to accomodate the text
* move some part of the text to a new location on the mmap so the new
text does not overwrite the old
* write the replacement text

When I try to use resize it gives me the following error;

SystemError: mmap: resizing not available--no mremap()

I pass size to mmap to begin with based on my filesize

size = os.path.getsize(filename)

m = mmap.mmap(f.fileno(), size)

I am attempting to give it new size by doing:

m.resize(size + size_diff)

size_diff is difference between original and what will be new file
size after the insertion.

Can someone provide some hint on resizing or its this a python bug?
Many thanks

Regards
David

[Philip Semanchuk]

Post by David Pratt
Hi. I have been experimenting with mmap recently. I determined how
to read and write properly from it and so search and replace on
large files. The problem I am having is with replaces that are
larger than the mmap. In this instance I need to
* rewind
* resize the mmap to accomodate the text
* move some part of the text to a new location on the mmap so the
new text does not overwrite the old
* write the replacement text
When I try to use resize it gives me the following error;
SystemError: mmap: resizing not available--no mremap()

Hi David,
Based on experience with my posix_ipc module, ISTR that (a) resizing
of the mmapp-ed segment happens via a call to ftruncate() (or
os.truncate() in Python land) and (b) OS X only supports one call to
this per segment; subsequent calls fail. Keep in mind that I was
dealing with shared memory. A regular mmapped file might resize just
fine so point (b) might be irrelevant for you.

HTH
Philip

[David Pratt]

Hi Phillip. I appreciate your reply. I think perhaps I will need to
create a new mmap as a work around and write to locations of the
second mmap based on my regex searches in the first. I should have
said I am using 2.5.4 as well to be clear. I am wondering if I should
recommend change to documentation for mmap if this is an issue for
OSX. I have vmware installed so might try on CentOS to see what
happens there. Many thanks

Regards,
David

Post by Philip Semanchuk

Post by David Pratt
Hi. I have been experimenting with mmap recently. I determined how
to read and write properly from it and so search and replace on
large files. The problem I am having is with replaces that are
larger than the mmap. In this instance I need to
* rewind
* resize the mmap to accomodate the text
* move some part of the text to a new location on the mmap so the
new text does not overwrite the old
* write the replacement text
When I try to use resize it gives me the following error;
SystemError: mmap: resizing not available--no mremap()

Hi David,
Based on experience with my posix_ipc module, ISTR that (a) resizing
of the mmapp-ed segment happens via a call to ftruncate() (or
os.truncate() in Python land) and (b) OS X only supports one call to
this per segment; subsequent calls fail. Keep in mind that I was
dealing with shared memory. A regular mmapped file might resize
just fine so point (b) might be irrelevant for you.
HTH
Philip
--
http://mail.python.org/mailman/listinfo/python-list

[David Pratt]

For sake of documenting for list, I ended up opening file a second
time with 'a', padding it to extend its size (previous size +
additional bytes to accommodate the insertion), closing file, open
file with 'r+', open a second mmap with with new size, moving text to
new location, then inserting the replacement text. Not quite as smooth
as doing a resize(), moving text and inserting if this had worked.

Regards,
David

Post by Philip Semanchuk

Post by David Pratt
Hi. I have been experimenting with mmap recently. I determined how
to read and write properly from it and so search and replace on
large files. The problem I am having is with replaces that are
larger than the mmap. In this instance I need to
* rewind
* resize the mmap to accomodate the text
* move some part of the text to a new location on the mmap so the
new text does not overwrite the old
* write the replacement text
When I try to use resize it gives me the following error;
SystemError: mmap: resizing not available--no mremap()

Hi David,
Based on experience with my posix_ipc module, ISTR that (a) resizing
of the mmapp-ed segment happens via a call to ftruncate() (or
os.truncate() in Python land) and (b) OS X only supports one call to
this per segment; subsequent calls fail. Keep in mind that I was
dealing with shared memory. A regular mmapped file might resize
just fine so point (b) might be irrelevant for you.
HTH
Philip
--
http://mail.python.org/mailman/listinfo/python-list