bug#38345: 27.0.50; Permanent increase in memory consumption after opening images (or pdfs)

classic Classic list List threaded Threaded
28 messages Options
12
Reply | Threaded
Open this post in threaded view
|

bug#38345: 27.0.50; Permanent increase in memory consumption after opening images (or pdfs)

Ihor Radchenko
Recently, I have noticed that my emacs tends to increase its memory
usage up to over 1.5Gb after I open a large number of pdf documents. It
is understandable since some of the pdfs can be quite large. However,
the problem is that the memory does not seem to be released even after I
close all the pdf buffers.

First, I though that the issue is with pdf-tools and reported my problem
there (https://github.com/politza/pdf-tools/issues/177). However, the
memory consumption problem appears to happen even if I just open images.

I monitored emacs (emacs -Q) memory usage with
https://github.com/dkogan/memory_leak_instrumentation

If I open a large (>250) photos using the code below, the memory usage
increases permanently even after I close all the image buffers and wait
for some time.

#+begin_src emacs-lisp
(setq large-file-warning-threshold nil)
(setq image-cache-eviction-delay 5)
(find-file "~/Tosort/pictures&photos/*.jpg" 'wild)
(mapc #'kill-buffer (seq-filter (apply-partially #'string-match ".+.jpg$") (mapcar #'buffer-name (buffer-list))))
(clear-image-cache t)
(garbage-collect)
#+end_src

The attached is memory consumption plot I got after running M-x
eval-buffer on the code above several times (each run correspond to the
memory consumption peak on the plot).

Regards,
Ihor

in GNU Emacs 27.0.50 (build 1, x86_64-pc-linux-gnu, X toolkit, cairo version 1.16.0)
 of 2019-11-21 built on yantar92-laptop
Repository revision: 1805b4cc08c8f9668f0f43f5b157d91b9c757e88
Repository branch: master
Windowing system distributor 'The X.Org Foundation', version 11.0.12005000
System Description: Gentoo/Linux




Reply | Threaded
Open this post in threaded view
|

bug#38345: Acknowledgement (27.0.50; Permanent increase in memory consumption after opening images (or pdfs))

Ihor Radchenko
Forgot to attach the memory consumption plot.
Here is goes.


images2.png (28K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

bug#38345: 27.0.50; Permanent increase in memory consumption after opening images (or pdfs)

Lars Ingebrigtsen
In reply to this post by Ihor Radchenko
Ihor Radchenko <[hidden email]> writes:

> Recently, I have noticed that my emacs tends to increase its memory
> usage up to over 1.5Gb after I open a large number of pdf documents. It
> is understandable since some of the pdfs can be quite large. However,
> the problem is that the memory does not seem to be released even after I
> close all the pdf buffers.

If you say

  M-: (clear-image-cache)

is the memory released?

--
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no



Reply | Threaded
Open this post in threaded view
|

bug#38345: 27.0.50; Permanent increase in memory consumption after opening images (or pdfs)

Eli Zaretskii
In reply to this post by Ihor Radchenko
> From: Ihor Radchenko <[hidden email]>
> Date: Sat, 23 Nov 2019 22:37:44 +0800
>
> Recently, I have noticed that my emacs tends to increase its memory
> usage up to over 1.5Gb after I open a large number of pdf documents. It
> is understandable since some of the pdfs can be quite large. However,
> the problem is that the memory does not seem to be released even after I
> close all the pdf buffers.
>
> First, I though that the issue is with pdf-tools and reported my problem
> there (https://github.com/politza/pdf-tools/issues/177). However, the
> memory consumption problem appears to happen even if I just open images.
>
> I monitored emacs (emacs -Q) memory usage with
> https://github.com/dkogan/memory_leak_instrumentation
>
> If I open a large (>250) photos using the code below, the memory usage
> increases permanently even after I close all the image buffers and wait
> for some time.
>
> #+begin_src emacs-lisp
> (setq large-file-warning-threshold nil)
> (setq image-cache-eviction-delay 5)
> (find-file "~/Tosort/pictures&photos/*.jpg" 'wild)
> (mapc #'kill-buffer (seq-filter (apply-partially #'string-match ".+.jpg$") (mapcar #'buffer-name (buffer-list))))
> (clear-image-cache t)
> (garbage-collect)
> #+end_src
>
> The attached is memory consumption plot I got after running M-x
> eval-buffer on the code above several times (each run correspond to the
> memory consumption peak on the plot).

I see no attachment here, but isn't this because Emacs never returns
memory to the system when it frees it?



Reply | Threaded
Open this post in threaded view
|

bug#38345: 27.0.50; Permanent increase in memory consumption after opening images (or pdfs)

Ihor Radchenko
In reply to this post by Lars Ingebrigtsen
> If you say
>
>   M-: (clear-image-cache)
>
> is the memory released?

The command is called in the code I provided.
When I call it manually, I can see that some part of memory is being
released. However, it is just a small fraction.

Best,
Ihor

Lars Ingebrigtsen <[hidden email]> writes:

> Ihor Radchenko <[hidden email]> writes:
>
>> Recently, I have noticed that my emacs tends to increase its memory
>> usage up to over 1.5Gb after I open a large number of pdf documents. It
>> is understandable since some of the pdfs can be quite large. However,
>> the problem is that the memory does not seem to be released even after I
>> close all the pdf buffers.
>
> If you say
>
>   M-: (clear-image-cache)
>
> is the memory released?
>
> --
> (domestic pets only, the antidote for overdose, milk.)
>    bloggy blog: http://lars.ingebrigtsen.no




Reply | Threaded
Open this post in threaded view
|

bug#38345: 27.0.50; Permanent increase in memory consumption after opening images (or pdfs)

Ihor Radchenko
In reply to this post by Eli Zaretskii
> I see no attachment here, but isn't this because Emacs never returns
> memory to the system when it frees it?

Sorry, I forgot to send the attachment in the first message. I have sent
a followup, but also attaching here just in case.

What you say about Emacs not returning memory sounds like very very
strange behaviour unless I misunderstand something. Does it mean that if
I have emacs running as daemon and open a few hundreds of heavy pdfs
during, say, a week, it will keep all the memory allocated for those pdfs
(which is several Gb, at least)? If so, I don't think that Emacs should
do it.

Regards,
Ihor





Eli Zaretskii <[hidden email]> writes:

>> From: Ihor Radchenko <[hidden email]>
>> Date: Sat, 23 Nov 2019 22:37:44 +0800
>>
>> Recently, I have noticed that my emacs tends to increase its memory
>> usage up to over 1.5Gb after I open a large number of pdf documents. It
>> is understandable since some of the pdfs can be quite large. However,
>> the problem is that the memory does not seem to be released even after I
>> close all the pdf buffers.
>>
>> First, I though that the issue is with pdf-tools and reported my problem
>> there (https://github.com/politza/pdf-tools/issues/177). However, the
>> memory consumption problem appears to happen even if I just open images.
>>
>> I monitored emacs (emacs -Q) memory usage with
>> https://github.com/dkogan/memory_leak_instrumentation
>>
>> If I open a large (>250) photos using the code below, the memory usage
>> increases permanently even after I close all the image buffers and wait
>> for some time.
>>
>> #+begin_src emacs-lisp
>> (setq large-file-warning-threshold nil)
>> (setq image-cache-eviction-delay 5)
>> (find-file "~/Tosort/pictures&photos/*.jpg" 'wild)
>> (mapc #'kill-buffer (seq-filter (apply-partially #'string-match ".+.jpg$") (mapcar #'buffer-name (buffer-list))))
>> (clear-image-cache t)
>> (garbage-collect)
>> #+end_src
>>
>> The attached is memory consumption plot I got after running M-x
>> eval-buffer on the code above several times (each run correspond to the
>> memory consumption peak on the plot).
>
> I see no attachment here, but isn't this because Emacs never returns
> memory to the system when it frees it?
--
Ihor Radchenko,
PhD,
Center for Advancing Materials Performance from the Nanoscale (CAMP-nano)
State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China
Email: [hidden email], [hidden email]

images2.png (28K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

bug#38345: 27.0.50; Permanent increase in memory consumption after opening images (or pdfs)

Eli Zaretskii
> From: Ihor Radchenko <[hidden email]>
> Cc: [hidden email]
> Date: Sat, 23 Nov 2019 23:18:21 +0800
>
> What you say about Emacs not returning memory sounds like very very
> strange behaviour unless I misunderstand something. Does it mean that if
> I have emacs running as daemon and open a few hundreds of heavy pdfs
> during, say, a week, it will keep all the memory allocated for those pdfs
> (which is several Gb, at least)? If so, I don't think that Emacs should
> do it.

It depends on how the memory is allocated.  Memory allocated for
buffers gets returned to the OS when those buffers are killed.  But
memory allocated via malloc AFAIK gets grafted into the program's
address space, and when it is freed, it is left in the program's
address space, free to be used by that program for any further
allocation -- but is not returned to the system.  So if you look at
the program's address space, you will think it only ever grows,
especially if you allocate a lot of memory before releasing the first
allocation.

At least that's what I think happens on modern platforms.



Reply | Threaded
Open this post in threaded view
|

bug#38345: 27.0.50; Permanent increase in memory consumption after opening images (or pdfs)

Ihor Radchenko
> memory allocated via malloc AFAIK gets grafted into the program's
> address space, and when it is freed, it is left in the program's
> address space, free to be used by that program for any further
> allocation -- but is not returned to the system.  So if you look at
> the program's address space, you will think it only ever grows,
> especially if you allocate a lot of memory before releasing the first
> allocation.

Hmm. Following an article by Dima Kogan [1], it appears to me that free()
should de-allocate memory in the program's address space (or at least
memory drop should be visible in the plot I got, since I used the same
memory debugging tools)

Best,
Ihor

[1] http://notes.secretsauce.net/notes/2015/10/05_memory-leak-debugging-tools.html


Eli Zaretskii <[hidden email]> writes:

>> From: Ihor Radchenko <[hidden email]>
>> Cc: [hidden email]
>> Date: Sat, 23 Nov 2019 23:18:21 +0800
>>
>> What you say about Emacs not returning memory sounds like very very
>> strange behaviour unless I misunderstand something. Does it mean that if
>> I have emacs running as daemon and open a few hundreds of heavy pdfs
>> during, say, a week, it will keep all the memory allocated for those pdfs
>> (which is several Gb, at least)? If so, I don't think that Emacs should
>> do it.
>
> It depends on how the memory is allocated.  Memory allocated for
> buffers gets returned to the OS when those buffers are killed.  But
> memory allocated via malloc AFAIK gets grafted into the program's
> address space, and when it is freed, it is left in the program's
> address space, free to be used by that program for any further
> allocation -- but is not returned to the system.  So if you look at
> the program's address space, you will think it only ever grows,
> especially if you allocate a lot of memory before releasing the first
> allocation.
>
> At least that's what I think happens on modern platforms.




Reply | Threaded
Open this post in threaded view
|

bug#38345: 27.0.50; Permanent increase in memory consumption after opening images (or pdfs)

Eli Zaretskii
> From: Ihor Radchenko <[hidden email]>
> Cc: [hidden email]
> Date: Sun, 24 Nov 2019 00:04:17 +0800
>
> Hmm. Following an article by Dima Kogan [1], it appears to me that free()
> should de-allocate memory in the program's address space (or at least
> memory drop should be visible in the plot I got, since I used the same
> memory debugging tools)

It may be useful to try memory mapping tools that show which part(s)
of a program "own" specific regions of memory.  It could be that the
memory growth is due to some library Emacs uses to show images, for
example.  Those tools usually also show the memory in the free pool of
the program's address space, so you should be able to see what's going
on when you call clear-image-cache.



Reply | Threaded
Open this post in threaded view
|

bug#38345: 27.0.50; Permanent increase in memory consumption after opening images (or pdfs)

Ihor Radchenko
> It may be useful to try memory mapping tools that show which part(s)
> of a program "own" specific regions of memory.  It could be that the
> memory growth is due to some library Emacs uses to show images, for
> example.  Those tools usually also show the memory in the free pool of
> the program's address space, so you should be able to see what's going
> on when you call clear-image-cache.

I tried to follow EmacsWiki
(https://www.emacswiki.org/emacs/EmacsMemoryDebugging).

The largest entry in smaps:

1. Before opening images

562e4cf38000-562e4d395000 rw-p 00000000 00:00 0                          [heap]
Size:               4468 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Rss:                4148 kB
Pss:                4148 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:      4148 kB
Referenced:         4148 kB
Anonymous:          4148 kB
LazyFree:              0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:                0 kB
THPeligible: 0
VmFlags: rd wr mr mw me ac

2. After opening images with all the buffers open

562e4cf38000-562e945a5000 rw-p 00000000 00:00 0                          [heap]
Size:            1169844 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Rss:             1169500 kB
Pss:             1169500 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:   1169500 kB
Referenced:      1169500 kB
Anonymous:       1169500 kB
LazyFree:              0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:                0 kB
THPeligible: 0
VmFlags: rd wr mr mw me ac

3. After killing all the image buffers

562e4cf38000-562e945a5000 rw-p 00000000 00:00 0                          [heap]
Size:            1169844 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Rss:             1169692 kB
Pss:             1169692 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:   1169692 kB
Referenced:      1169692 kB
Anonymous:       1169692 kB
LazyFree:              0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:                0 kB
THPeligible: 0
VmFlags: rd wr mr mw me ac

4. After calling (clear-image-cache t) and (garbage-collect)

562e4cf38000-562e945a5000 rw-p 00000000 00:00 0                          [heap]
Size:            1169844 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Rss:             1169692 kB
Pss:             1169692 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:   1169692 kB
Referenced:      1169692 kB
Anonymous:       1169692 kB
LazyFree:              0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:                0 kB
THPeligible: 0
VmFlags: rd wr mr mw me ac

The full info from smaps and output from pmap are attached:
*0.txt ::  Before opening images
*open.txt :: With all the image buffers
*kill.txt :: After killing all the image buffers
*clear.txt :: After clearing cache and garbage collecting

I also attempted to use perf, but I cannot understand much of
information from there and I was not able to make the scripts I use work
with the output. If you need, I can also generate perf output (may take
some time).

Regards,
Ihor








Eli Zaretskii <[hidden email]> writes:

>> From: Ihor Radchenko <[hidden email]>
>> Cc: [hidden email]
>> Date: Sun, 24 Nov 2019 00:04:17 +0800
>>
>> Hmm. Following an article by Dima Kogan [1], it appears to me that free()
>> should de-allocate memory in the program's address space (or at least
>> memory drop should be visible in the plot I got, since I used the same
>> memory debugging tools)
>
> It may be useful to try memory mapping tools that show which part(s)
> of a program "own" specific regions of memory.  It could be that the
> memory growth is due to some library Emacs uses to show images, for
> example.  Those tools usually also show the memory in the free pool of
> the program's address space, so you should be able to see what's going
> on when you call clear-image-cache.

pmap0.txt (46K) Download Attachment
pmap-open.txt (47K) Download Attachment
pmap-kill.txt (47K) Download Attachment
pmap-clear.txt (47K) Download Attachment
smaps0.txt (414K) Download Attachment
smaps-open.txt (422K) Download Attachment
smaps-kill.txt (422K) Download Attachment
smaps-clear.txt (422K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

bug#38345: 27.0.50; Permanent increase in memory consumption after opening images (or pdfs)

Eli Zaretskii
> From: Ihor Radchenko <[hidden email]>
> Cc: [hidden email]
> Date: Sun, 24 Nov 2019 01:33:29 +0800
>
> The full info from smaps and output from pmap are attached:
> *0.txt ::  Before opening images
> *open.txt :: With all the image buffers
> *kill.txt :: After killing all the image buffers
> *clear.txt :: After clearing cache and garbage collecting

I'm not an expert, but AFAIK "anon" means buffer memory.

I hope someone more knowledgeable will chime in, though.



Reply | Threaded
Open this post in threaded view
|

bug#38345: 27.0.50; Permanent increase in memory consumption after opening images (or pdfs)

Eli Zaretskii
In reply to this post by Ihor Radchenko
> From: Ihor Radchenko <[hidden email]>
> Date: Sat, 23 Nov 2019 22:37:44 +0800
>
> If I open a large (>250) photos using the code below, the memory usage
> increases permanently even after I close all the image buffers and wait
> for some time.

For some background on this, see

  https://unix.stackexchange.com/questions/53447/does-free-unmap-the-memory-of-a-process
  https://stackoverflow.com/questions/1421491/does-calling-free-or-delete-ever-release-memory-back-to-the-system

I think what you see is just normal behavior of memory allocation in
glibc.



Reply | Threaded
Open this post in threaded view
|

bug#38345: 27.0.50; Permanent increase in memory consumption after opening images (or pdfs)

Ihor Radchenko
Thanks for the links. I got to know more about memory management now.
Now, it is clear why the memory is consumption is so high when I open a
bunch of images at the same time. However, it still does not explain my
observations during usual workflow.

For my real usage, I open pdfs (or images) one by one most of the time
and kill the buffers periodically.
Then, if the memory is freed upon closing a pdf buffer, I expect it to
be reused for opening a new buffer. Even though memory consumption is
expected to grow in this case, it should be in order of the largest
image I open (times maximum number of image buffers open at the same
time). But it is not what I see.

I did a small test by modifying my earlier lisp code to open and close
the same image list sequentially:

#+begin_src emacs-lisp
(dolist (file (directory-files "~/Tosort/pictures&photos/" 'full ".*jpg"))
  (find-file file)
  (mapc #'kill-buffer (seq-filter (apply-partially #'string-match ".+.jpg$") (mapcar #'buffer-name (buffer-list)))))
#+end_src

The resulting memory usage graph is attached.

What we can see is that the memory is indeed growing (as expected).
Moreover, the memory consumption does not increase as much as if we open
all the images together. However, the final heap size appears to be over
400Mb (from smaps), which is almost half of what was observed with all
the images open at the same time. Since the largest .jpg file I have in
the folder is just around 5.5Mb, 400Mb sounds strange for me. Googling
on memory consumption issues, I found that there might be some memory
fragmentation problem happening [1].

P.S. Were there any attempts to implement garbage collection for emacs
in C code? I found an article [2] showing that using an actual GC may speed
up an application in comparison with malloc/free approach.


[1] https://stackoverflow.com/a/9069474/9196985
[2] https://www.linuxjournal.com/article/6679

Regards,
Ihor




Eli Zaretskii <[hidden email]> writes:

>> From: Ihor Radchenko <[hidden email]>
>> Date: Sat, 23 Nov 2019 22:37:44 +0800
>>
>> If I open a large (>250) photos using the code below, the memory usage
>> increases permanently even after I close all the image buffers and wait
>> for some time.
>
> For some background on this, see
>
>   https://unix.stackexchange.com/questions/53447/does-free-unmap-the-memory-of-a-process
>   https://stackoverflow.com/questions/1421491/does-calling-free-or-delete-ever-release-memory-back-to-the-system
>
> I think what you see is just normal behavior of memory allocation in
> glibc.


images-seq.png (29K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

bug#38345: 27.0.50; Permanent increase in memory consumption after opening images (or pdfs)

Eli Zaretskii
> From: Ihor Radchenko <[hidden email]>
> Cc: [hidden email]
> Date: Tue, 26 Nov 2019 23:21:11 +0800
>
> I did a small test by modifying my earlier lisp code to open and close
> the same image list sequentially:
>
> #+begin_src emacs-lisp
> (dolist (file (directory-files "~/Tosort/pictures&photos/" 'full ".*jpg"))
>   (find-file file)
>   (mapc #'kill-buffer (seq-filter (apply-partially #'string-match ".+.jpg$") (mapcar #'buffer-name (buffer-list)))))
> #+end_src
>
> The resulting memory usage graph is attached.
>
> What we can see is that the memory is indeed growing (as expected).
> Moreover, the memory consumption does not increase as much as if we open
> all the images together. However, the final heap size appears to be over
> 400Mb (from smaps), which is almost half of what was observed with all
> the images open at the same time.

Does it help to call garbage-collect after killing each buffer and
before visiting the next image file?

> P.S. Were there any attempts to implement garbage collection for emacs
> in C code?

I don't understand the question, sorry.  GC in Emacs is implemented in
C, not in Lisp.  If you mean to use a garbage-collecting memory
allocator, then we once had that only for buffer text, but nowadays
system malloc is good enough for that, so we no longer use it.  Doing
that for "normal" C code is not easy because it generally requires
double indirection for accessing data through pointers.



Reply | Threaded
Open this post in threaded view
|

bug#38345: 27.0.50; Permanent increase in memory consumption after opening images (or pdfs)

Ihor Radchenko
> Does it help to call garbage-collect after killing each buffer and
> before visiting the next image file?

No. It makes no difference.

> I don't understand the question, sorry.  GC in Emacs is implemented in
> C, not in Lisp.  If you mean to use a garbage-collecting memory
> allocator, then we once had that only for buffer text, but nowadays
> system malloc is good enough for that, so we no longer use it.  Doing
> that for "normal" C code is not easy because it generally requires
> double indirection for accessing data through pointers.

Yes, I meant garbage collecting malloc. I don't know much about this
topic. Just stumbled on an article saying that it can speed up
performance.

Regards,
Ihor


Eli Zaretskii <[hidden email]> writes:

>> From: Ihor Radchenko <[hidden email]>
>> Cc: [hidden email]
>> Date: Tue, 26 Nov 2019 23:21:11 +0800
>>
>> I did a small test by modifying my earlier lisp code to open and close
>> the same image list sequentially:
>>
>> #+begin_src emacs-lisp
>> (dolist (file (directory-files "~/Tosort/pictures&photos/" 'full ".*jpg"))
>>   (find-file file)
>>   (mapc #'kill-buffer (seq-filter (apply-partially #'string-match ".+.jpg$") (mapcar #'buffer-name (buffer-list)))))
>> #+end_src
>>
>> The resulting memory usage graph is attached.
>>
>> What we can see is that the memory is indeed growing (as expected).
>> Moreover, the memory consumption does not increase as much as if we open
>> all the images together. However, the final heap size appears to be over
>> 400Mb (from smaps), which is almost half of what was observed with all
>> the images open at the same time.
>
> Does it help to call garbage-collect after killing each buffer and
> before visiting the next image file?
>
>> P.S. Were there any attempts to implement garbage collection for emacs
>> in C code?
>
> I don't understand the question, sorry.  GC in Emacs is implemented in
> C, not in Lisp.  If you mean to use a garbage-collecting memory
> allocator, then we once had that only for buffer text, but nowadays
> system malloc is good enough for that, so we no longer use it.  Doing
> that for "normal" C code is not easy because it generally requires
> double indirection for accessing data through pointers.




Reply | Threaded
Open this post in threaded view
|

bug#38345: 27.0.50; Permanent increase in memory consumption after opening images (or pdfs)

Juri Linkov-2
In reply to this post by Ihor Radchenko
> Thanks for the links. I got to know more about memory management now.
> Now, it is clear why the memory is consumption is so high when I open a
> bunch of images at the same time. However, it still does not explain my
> observations during usual workflow.

Please see also the recent discussion in
https://debbugs.gnu.org/38187#95



Reply | Threaded
Open this post in threaded view
|

bug#38345: 27.0.50; Permanent increase in memory consumption after opening images (or pdfs)

Ihor Radchenko
> Please see also the recent discussion in
> https://debbugs.gnu.org/38187#95

Thanks.

Similar to that discussion, I tried to force clearing the image cache
after killing each image buffer.

#+begin_sec emacs-lisp
(dolist (file (directory-files "~/Tosort/pictures&photos/" 'full ".*jpg"))
  (find-file file)
  (mapc #'kill-buffer (seq-filter (apply-partially #'string-match ".+.jpg$") (mapcar #'buffer-name (buffer-list))))
  (garbage-collect)
  (clear-image-cache t))
#+end_src

The resulting memory consumption graph is attached.
The memory increase almost disappeared (remaining heap size becomes
~40Mb in comparison ~400Mb in the version with just garbage collect).

Just calling (clear-image-cache) after cycling over opening/killing
the image buffers still results in ~400Mb (it has no effect, basically).

The above result is confusing since the all the code I tried to run so
far had (setq image-cache-eviction-delay 5). Since, cycling over all the
images usually took >1min, cache clearing supposed to happen at least
several times during opening/killing the image buffers.

To understand further, I tried to disable automatic cache clearing
completely (setq image-cache-eviction-delay 5000). The resulting memory
consumption is ~400Mb, but a few tens of Mb higher in comparison with
(setq image-cache-eviction-delay 5).

It appears to me that clearing image cache has some problem when there
is a relatively large number of images in the cache already.


Regards,
Ihor




Juri Linkov <[hidden email]> writes:

>> Thanks for the links. I got to know more about memory management now.
>> Now, it is clear why the memory is consumption is so high when I open a
>> bunch of images at the same time. However, it still does not explain my
>> observations during usual workflow.
>
> Please see also the recent discussion in
> https://debbugs.gnu.org/38187#95


images-seq-flush.png (30K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

bug#38345: 27.0.50; Permanent increase in memory consumption after opening images (or pdfs)

Lars Ingebrigtsen
Ihor Radchenko <[hidden email]> writes:

> It appears to me that clearing image cache has some problem when there
> is a relatively large number of images in the cache already.

When Emacs frees memory, it's not normally returned to the OS.  Is that
what you're seeing?

--
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no



Reply | Threaded
Open this post in threaded view
|

bug#38345: 27.0.50; Permanent increase in memory consumption after opening images (or pdfs)

Ihor Radchenko
> When Emacs frees memory, it's not normally returned to the OS.  Is that
> what you're seeing?

It's indeed not returned to the OS (as from previous discussion), which
is fine as soon as emacs can reuse the owned memory.
When I clear the image cache after closing every single image buffer,
emacs does seem to reuse the owned memory.
However, when the cache is cleared less frequently, it does not seem to
be the case - emacs just claims more memory from the system.
What I saw from my last test is that there is no significant difference
in the final memory consumption if I don't clear the image cache or
clear it every 5 seconds (around every 10-20 opened/killed images).

Best,
Ihor

Lars Ingebrigtsen <[hidden email]> writes:

> Ihor Radchenko <[hidden email]> writes:
>
>> It appears to me that clearing image cache has some problem when there
>> is a relatively large number of images in the cache already.
>
> When Emacs frees memory, it's not normally returned to the OS.  Is that
> what you're seeing?
>
> --
> (domestic pets only, the antidote for overdose, milk.)
>    bloggy blog: http://lars.ingebrigtsen.no




Reply | Threaded
Open this post in threaded view
|

bug#38345: 27.0.50; Permanent increase in memory consumption after opening images (or pdfs)

Eli Zaretskii
In reply to this post by Ihor Radchenko
> From: Ihor Radchenko <[hidden email]>
> Cc: Eli Zaretskii <[hidden email]>, [hidden email]
> Date: Thu, 28 Nov 2019 09:38:34 +0800
>
> #+begin_sec emacs-lisp
> (dolist (file (directory-files "~/Tosort/pictures&photos/" 'full ".*jpg"))
>   (find-file file)
>   (mapc #'kill-buffer (seq-filter (apply-partially #'string-match ".+.jpg$") (mapcar #'buffer-name (buffer-list))))
>   (garbage-collect)
>   (clear-image-cache t))
> #+end_src
>
> The resulting memory consumption graph is attached.
> The memory increase almost disappeared (remaining heap size becomes
> ~40Mb in comparison ~400Mb in the version with just garbage collect).
>
> Just calling (clear-image-cache) after cycling over opening/killing
> the image buffers still results in ~400Mb (it has no effect, basically).
>
> The above result is confusing since the all the code I tried to run so
> far had (setq image-cache-eviction-delay 5). Since, cycling over all the
> images usually took >1min, cache clearing supposed to happen at least
> several times during opening/killing the image buffers.

As I explained elsewhere, unless you call clear-image-cache from Lisp,
the only place we do that automatically is when the number of
redisplay cycles since last time the image cache was cleared becomes
greater than 101.  This, no matter how low is the value of
image-cache-eviction-delay, it will have no effect until we've done
101 redisplay cycles.  And your loop above does just one.



12