bug#44666: 28.0.50; malloc-info: Emacs became not responsive, using hard disk all time

classic Classic list List threaded Threaded
98 messages Options
12345
Reply | Threaded
Open this post in threaded view
|

bug#44666: 28.0.50; malloc-info: Emacs became not responsive, using hard disk all time

Jean Louis

Since I wish to find out what is making Emacs slow sometimes, I am
running it with this shell script:

emacs-debug.sh:

#!/bin/bash
## CDPATH I have to unset as otherwise eshell and shell do not work well
unset CDPATH
date >> /home/data1/protected/tmp/emacs-debug
emacs >> /home/data1/protected/tmp/emacs-debug 2>&1

Then if there is non-responsive problem I can do M-x malloc-info

This time computer became totally not responsive:

- using IceWM (rarely happens, almost by rule with EXWM)

- I have not invoked any special function, just small list processing
  where it had 6 elements in total. The non-responsiveness was not
  caused by this function. That is how I say by feeling.

- then I think, by feeling, swapping started or already started during
  my work.

- hardly hardly and with a lot of patience I could invoke M-x
  malloc-info

Fri Nov 13 08:40:17 EAT 2020
Fri Nov 13 19:41:22 EAT 2020
Fri Nov 13 21:51:07 EAT 2020
Fri Nov 13 23:28:16 EAT 2020
Fri Nov 13 23:28:49 EAT 2020
Fri Nov 13 23:41:47 EAT 2020
Fri Nov 13 23:42:35 EAT 2020
Fri Nov 13 23:43:32 EAT 2020
Sat Nov 14 00:22:09 EAT 2020
Sat Nov 14 00:26:32 EAT 2020
Sat Nov 14 11:47:26 EAT 2020
Sat Nov 14 11:59:16 EAT 2020
Sun Nov 15 12:38:28 EAT 2020
<malloc version="1">
<heap nr="0">
<sizes>
                                                                <size from="49" to="49" total="49" count="1"/>
  <unsorted from="257" to="257" total="257" count="1"/>
</sizes>
<total type="fast" count="0" size="0"/>
<total type="rest" count="2" size="306"/>
<system type="current" size="11470942208"/>
<system type="max" size="11470942208"/>
<aspace type="total" size="11470942208"/>
<aspace type="mprotect" size="11470942208"/>
</heap>
<heap nr="1">
<sizes>
                                                                <size from="33" to="48" total="48" count="1"/>
                                                                <size from="65" to="80" total="80" count="1"/>
</sizes>
<total type="fast" count="2" size="128"/>
<total type="rest" count="0" size="0"/>
<system type="current" size="135168"/>
<system type="max" size="135168"/>
<aspace type="total" size="135168"/>
<aspace type="mprotect" size="135168"/>
</heap>
<heap nr="2">
<sizes>
                                                                <size from="33" to="48" total="48" count="1"/>
                                                                <size from="65" to="80" total="80" count="1"/>
</sizes>
<total type="fast" count="2" size="128"/>
<total type="rest" count="0" size="0"/>
<system type="current" size="135168"/>
<system type="max" size="135168"/>
<aspace type="total" size="135168"/>
<aspace type="mprotect" size="135168"/>
</heap>
<heap nr="3">
<sizes>
                                                                <size from="33" to="48" total="48" count="1"/>
                                                                <size from="65" to="80" total="80" count="1"/>
                                                                <size from="33" to="33" total="33" count="1"/>
</sizes>
<total type="fast" count="2" size="128"/>
<total type="rest" count="1" size="33"/>
<system type="current" size="135168"/>
<system type="max" size="135168"/>
<aspace type="total" size="135168"/>
<aspace type="mprotect" size="135168"/>
</heap>
<heap nr="4">
<sizes>
                                                                <size from="33" to="48" total="48" count="1"/>
                                                                <size from="65" to="80" total="80" count="1"/>
</sizes>
<total type="fast" count="2" size="128"/>
<total type="rest" count="0" size="0"/>
<system type="current" size="135168"/>
<system type="max" size="135168"/>
<aspace type="total" size="135168"/>
<aspace type="mprotect" size="135168"/>
</heap>
<heap nr="5">
<sizes>
                                                                <size from="33" to="48" total="48" count="1"/>
                                                                <size from="65" to="80" total="80" count="1"/>
  <unsorted from="2449" to="2449" total="2449" count="1"/>
</sizes>
<total type="fast" count="2" size="128"/>
<total type="rest" count="1" size="2449"/>
<system type="current" size="135168"/>
<system type="max" size="135168"/>
<aspace type="total" size="135168"/>
<aspace type="mprotect" size="135168"/>
</heap>
<heap nr="6">
<sizes>
                                                                <size from="33" to="48" total="48" count="1"/>
                                                                <size from="65" to="80" total="80" count="1"/>
</sizes>
<total type="fast" count="2" size="128"/>
<total type="rest" count="0" size="0"/>
<system type="current" size="135168"/>
<system type="max" size="135168"/>
<aspace type="total" size="135168"/>
<aspace type="mprotect" size="135168"/>
</heap>
<heap nr="7">
<sizes>
                                                                <size from="17" to="32" total="864" count="27"/>
                                                                <size from="33" to="48" total="384" count="8"/>
                                                                <size from="65" to="80" total="160" count="2"/>
                                                                <size from="97" to="112" total="336" count="3"/>
</sizes>
<total type="fast" count="40" size="1744"/>
<total type="rest" count="0" size="0"/>
<system type="current" size="139264"/>
<system type="max" size="139264"/>
<aspace type="total" size="139264"/>
<aspace type="mprotect" size="139264"/>
</heap>
<heap nr="8">
<sizes>
                                                                <size from="17" to="32" total="832" count="26"/>
                                                                <size from="33" to="48" total="240" count="5"/>
                                                                <size from="65" to="80" total="160" count="2"/>
                                                                <size from="97" to="112" total="112" count="1"/>
                                                                <size from="113" to="128" total="128" count="1"/>
                                                                <size from="49" to="49" total="49" count="1"/>
                                                                <size from="65" to="65" total="65" count="1"/>
                                                                <size from="145" to="145" total="145" count="1"/>
                                                                <size from="193" to="193" total="193" count="1"/>
                                                                <size from="449" to="449" total="449" count="1"/>
  <unsorted from="2961" to="2961" total="2961" count="1"/>
</sizes>
<total type="fast" count="35" size="1472"/>
<total type="rest" count="6" size="3862"/>
<system type="current" size="139264"/>
<system type="max" size="139264"/>
<aspace type="total" size="139264"/>
<aspace type="mprotect" size="139264"/>
</heap>
<heap nr="9">
<sizes>
</sizes>
<total type="fast" count="0" size="0"/>
<total type="rest" count="0" size="0"/>
<system type="current" size="135168"/>
<system type="max" size="135168"/>
<aspace type="total" size="135168"/>
<aspace type="mprotect" size="135168"/>
</heap>
<total type="fast" count="87" size="3984"/>
<total type="rest" count="10" size="6650"/>
<total type="mmap" count="2" size="5341184"/>
<system type="current" size="11472166912"/>
<system type="max" size="11472166912"/>
<aspace type="total" size="11472166912"/>
<aspace type="mprotect" size="11472166912"/>
</malloc>




In GNU Emacs 28.0.50 (build 1, x86_64-pc-linux-gnu, X toolkit, cairo version 1.14.8, Xaw3d scroll bars)
 of 2020-11-14 built on protected.rcdrun.com
Repository revision: 31f94e4b1c3dc201646ec436d3e2c477f784ed21
Repository branch: master
System Description: Hyperbola GNU/Linux-libre

Configured using:
 'configure --prefix=/package/text/emacs-2020-11-14 --with-modules
 --with-x-toolkit=lucid'

Configured features:
XAW3D XPM JPEG TIFF GIF PNG RSVG CAIRO SOUND GPM DBUS GSETTINGS GLIB
NOTIFY INOTIFY ACL GNUTLS LIBXML2 FREETYPE HARFBUZZ M17N_FLT LIBOTF ZLIB
TOOLKIT_SCROLL_BARS LUCID X11 XDBE XIM MODULES THREADS JSON PDUMPER
LCMS2

Important settings:
  value of $LC_ALL: en_US.UTF-8
  value of $LANG: de_DE.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Lisp Interaction

Minor modes in effect:
  gpm-mouse-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Load-path shadows:
None found.

Features:
(shadow sort hashcash mail-extr emacsbug message rmc puny dired
dired-loaddefs rfc822 mml easymenu mml-sec epa derived epg epg-config
gnus-util rmail rmail-loaddefs auth-source cl-seq eieio eieio-core
cl-macs eieio-loaddefs password-cache json map text-property-search
time-date subr-x seq byte-opt gv bytecomp byte-compile cconv mm-decode
mm-bodies mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader
cl-loaddefs cl-lib sendmail rfc2047 rfc2045 ietf-drums mm-util
mail-prsvr mail-utils t-mouse term/linux disp-table tooltip eldoc
electric uniquify ediff-hook vc-hooks lisp-float-type mwheel term/x-win
x-win term/common-win x-dnd tool-bar dnd fontset image regexp-opt fringe
tabulated-list replace newcomment text-mode elisp-mode lisp-mode
prog-mode register page tab-bar menu-bar rfn-eshadow isearch timer
select scroll-bar mouse jit-lock font-lock syntax facemenu font-core
term/tty-colors frame minibuffer cl-generic cham georgian utf-8-lang
misc-lang vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms
cp51932 hebrew greek romanian slovak czech european ethiopic indian
cyrillic chinese composite charscript charprop case-table epa-hook
jka-cmpr-hook help simple abbrev obarray cl-preloaded nadvice button
loaddefs faces cus-face macroexp files window text-properties overlay
sha1 md5 base64 format env code-pages mule custom widget
hashtable-print-readable backquote threads dbusbind inotify lcms2
dynamic-setting system-font-setting font-render-setting cairo x-toolkit
x multi-tty make-network-process emacs)

Memory information:
((conses 16 52575 6366)
 (symbols 48 7259 1)
 (strings 32 18937 1368)
 (string-bytes 1 616804)
 (vectors 16 8986)
 (vector-slots 8 116851 8619)
 (floats 8 22 260)
 (intervals 56 196 0)
 (buffers 992 11))

--
Thanks,
Jean Louis
⎔ λ 🄯 𝍄 𝌡 𝌚



Reply | Threaded
Open this post in threaded view
|

bug#44666: 28.0.50; malloc-info: Emacs became not responsive, using hard disk all time

Eli Zaretskii
> From: Jean Louis <[hidden email]>
> Date: Sun, 15 Nov 2020 17:55:09 +0300
>
> Sun Nov 15 12:38:28 EAT 2020
> <malloc version="1">
> <heap nr="0">
> <sizes>
>        <size from="49" to="49" total="49" count="1"/>
>   <unsorted from="257" to="257" total="257" count="1"/>
> </sizes>
> <total type="fast" count="0" size="0"/>
> <total type="rest" count="2" size="306"/>
> <system type="current" size="11470942208"/>
> <system type="max" size="11470942208"/>
> <aspace type="total" size="11470942208"/>
> <aspace type="mprotect" size="11470942208"/>
> </heap>

This basically says you have 11GB in the heap, but there are no
details.  So I'm not sure how this could help us make any progress.

Thanks.



Reply | Threaded
Open this post in threaded view
|

bug#44666: 28.0.50; malloc-info: Emacs became not responsive, using hard disk all time

Jean Louis
* Eli Zaretskii <[hidden email]> [2020-11-16 19:12]:

> > From: Jean Louis <[hidden email]>
> > Date: Sun, 15 Nov 2020 17:55:09 +0300
> >
> > Sun Nov 15 12:38:28 EAT 2020
> > <malloc version="1">
> > <heap nr="0">
> > <sizes>
> >        <size from="49" to="49" total="49" count="1"/>
> >   <unsorted from="257" to="257" total="257" count="1"/>
> > </sizes>
> > <total type="fast" count="0" size="0"/>
> > <total type="rest" count="2" size="306"/>
> > <system type="current" size="11470942208"/>
> > <system type="max" size="11470942208"/>
> > <aspace type="total" size="11470942208"/>
> > <aspace type="mprotect" size="11470942208"/>
> > </heap>
>
> This basically says you have 11GB in the heap, but there are no
> details.  So I'm not sure how this could help us make any progress.

I was thinking that command would tell you something.

There was nothing special. I have 4 GB memory and 8 GB swap. There was
no special program running, just XTerm and Emacs.

I would like to find out why is Emacs taking that memory, but I am
unable.

Now I am running it with ulimit, but I am unsure if that ulimit
command really works as manual pages says it sometimes does not work.

#!/bin/bash
unset CDPATH
ulimit -m 3145728
date >> /home/data1/protected/tmp/emacs-debug
emacs >> /home/data1/protected/tmp/emacs-debug 2>&1

If there is nothing to be done with this bug, we can close.

You could suggest me on what to put attention to find out what is
going on.



Reply | Threaded
Open this post in threaded view
|

bug#44666: 28.0.50; malloc-info: Emacs became not responsive, using hard disk all time

Eli Zaretskii
> Date: Mon, 16 Nov 2020 19:17:35 +0300
> From: Jean Louis <[hidden email]>
> Cc: [hidden email]
>
> * Eli Zaretskii <[hidden email]> [2020-11-16 19:12]:
> > > From: Jean Louis <[hidden email]>
> > > Date: Sun, 15 Nov 2020 17:55:09 +0300
> > >
> > > Sun Nov 15 12:38:28 EAT 2020
> > > <malloc version="1">
> > > <heap nr="0">
> > > <sizes>
> > >        <size from="49" to="49" total="49" count="1"/>
> > >   <unsorted from="257" to="257" total="257" count="1"/>
> > > </sizes>
> > > <total type="fast" count="0" size="0"/>
> > > <total type="rest" count="2" size="306"/>
> > > <system type="current" size="11470942208"/>
> > > <system type="max" size="11470942208"/>
> > > <aspace type="total" size="11470942208"/>
> > > <aspace type="mprotect" size="11470942208"/>
> > > </heap>
> >
> > This basically says you have 11GB in the heap, but there are no
> > details.  So I'm not sure how this could help us make any progress.
>
> I was thinking that command would tell you something.

It tells something, I just don't yet know what that is.

> If there is nothing to be done with this bug, we can close.

No, closing is premature.  I've merged this bug with 3 other similar
ones, and we are discussing this issue with glibc malloc experts.



Reply | Threaded
Open this post in threaded view
|

bug#44666: 28.0.50; malloc-info: Emacs became not responsive, using hard disk all time

Jean Louis
* Eli Zaretskii <[hidden email]> [2020-11-17 10:04]:
> > If there is nothing to be done with this bug, we can close.
>
> No, closing is premature.  I've merged this bug with 3 other similar
> ones, and we are discussing this issue with glibc malloc experts.

If bug is merged, do I just reply on this email?

My emacs-uptime now is 19 hours, and I can see 4819 MB swapping
according  to symon-mode

I have not get number of buffers, I tried to delete it and there is no
change. User processes are below. I have not finished this session and
so I am prematurely sending the file
emacs.strace-2020-11-18-14:42:59-Wednesday which may be accessed here
below on the link. I could not copy the file fully through eshell probably
because if I do copy through eshell the strace becomes longer and
longer and copy never finishes. So I have aborted the copy, file may
not be complete. It is also not complete for reason that session is
not finished.

strace is here, 13M download, when unpacked it is more than 1.2 GB.
https://gnu.support/files/tmp/emacs.strace-2020-11-18-14:42:59-Wednesday.lz

When finishing this email swapping reported is 4987 MB and I know by
experience it will come to system being not usable.

              total        used        free      shared  buff/cache   available
Mem:        3844508     3575720      119476       37576      149312       55712
Swap:       8388604     4820656     3567948

$ htop shows

8399 VIRT memory for emacs and 3211M RES memory for emacs

  admin 30586  4.5 88.1 Nov 18 50:52 emacs
  admin 30584  0.9  0.0 Nov 18 10:20 strace -o emacs.strace-2020-11-18-14:42:59-Wednesday emacs
  admin  5542  0.1  0.1 Nov 17 02:13 icewm --notify
  admin 15914  0.0  0.4  07:26 00:02 mutt
  admin  5584  0.0  0.0 Nov 17 00:09 /usr/bin/dbus-daemon --fork --print-pid 5 --print-address 7 --session
  admin 17639  0.0  0.0  09:42 00:00 emacsclient -c /home/data1/protected/tmp/mutt-protected-1001-15914-94772654077392443
  admin  8410  0.0  0.0 Nov 18 00:05 /usr/lib/at-spi2-core/at-spi2-registryd --use-gnome-session
  admin 17023  0.0  0.1  08:35 00:00 /bin/bash --noediting -i
  admin 21322  0.0  0.0 Nov 18 00:00 /usr/bin/festival
  admin 28366  0.0  0.0 Nov 18 00:00 /bin/bash
  admin  8408  0.0  0.0 Nov 18 00:00 /usr/bin/dbus-daemon --config-file=/usr/share/defaults/at-spi2/accessibility.conf --nofork --print-address 3
  admin  5541  0.0  0.0 Nov 17 00:00 icewmbg
  admin 28038  0.0  0.0 Nov 18 00:00 /usr/lib/dconf/dconf-service
  admin  8429  0.0  0.0 Nov 18 00:00 /usr/lib/GConf/gconfd-2
  admin 29399  0.0  0.0  00:18 00:00 /usr/local/bin/psql -U maddox -h localhost -P pager=off rcdbusiness
  admin  5426  0.0  0.0 Nov 17 00:00 -bash
  admin 14932  0.0  0.0 Nov 18 00:00 /usr/bin/aspell -a -m -d en --encoding=utf-8
  admin  8403  0.0  0.0 Nov 18 00:00 /usr/lib/at-spi2-core/at-spi-bus-launcher
  admin  5501  0.0  0.0 Nov 17 00:00 /bin/sh /usr/bin/startx
  admin  5523  0.0  0.0 Nov 17 00:00 xinit /home/data1/protected/.xinitrc -- /etc/X11/xinit/xserverrc :0 vt1 -keeptty -auth /tmp/serverauth.Tvh06SZQdP
  admin  5528  0.0  0.0 Nov 17 00:00 sh /home/data1/protected/.xinitrc
  admin  5540  0.0  0.0 Nov 17 00:00 icewm-session
  admin  5579  0.0  0.0 Nov 17 00:00 dbus-launch --autolaunch=9459754a0df54d1465edf14d5b0bfe99 --binary-syntax --close-stderr
  admin 30582  0.0  0.0 Nov 18 00:00 /bin/bash /home/data1/protected/bin/emacs-debug.sh




Reply | Threaded
Open this post in threaded view
|

bug#44666: 28.0.50; malloc-info: Emacs became not responsive, using hard disk all time

Jean Louis
In reply to this post by Eli Zaretskii
* Eli Zaretskii <[hidden email]> [2020-11-17 10:04]:
> No, closing is premature.  I've merged this bug with 3 other similar
> ones, and we are discussing this issue with glibc malloc experts.

I have now finished the session as it became unbearable. I could not
switch from one Window Manager workspace to other WM
workspace. Swapping grew over 5.3 GB.

After finishing session memory usage came back to normal and I can
start new session.

The link for strace file that I have sent in the previous email has
been updated and is now finished as session has been finished.



Reply | Threaded
Open this post in threaded view
|

bug#43389: 28.0.50; Emacs memory leaks using hard disk all time

Eli Zaretskii
In reply to this post by Jean Louis
> Date: Thu, 19 Nov 2020 09:59:44 +0300
> From: Jean Louis <[hidden email]>
> Cc: [hidden email]
>
> * Eli Zaretskii <[hidden email]> [2020-11-17 10:04]:
> > > If there is nothing to be done with this bug, we can close.
> >
> > No, closing is premature.  I've merged this bug with 3 other similar
> > ones, and we are discussing this issue with glibc malloc experts.
>
> If bug is merged, do I just reply on this email?

No, it's better to reply to bug#43389 (I've redirected the discussion
now), and please keep the other addressees on the CC list, as they are
not subscribed to the bug list, I believe.

> My emacs-uptime now is 19 hours, and I can see 4819 MB swapping
> according  to symon-mode
>
> I have not get number of buffers, I tried to delete it and there is no
> change. User processes are below. I have not finished this session and
> so I am prematurely sending the file
> emacs.strace-2020-11-18-14:42:59-Wednesday which may be accessed here
> below on the link. I could not copy the file fully through eshell probably
> because if I do copy through eshell the strace becomes longer and
> longer and copy never finishes. So I have aborted the copy, file may
> not be complete. It is also not complete for reason that session is
> not finished.
>
> strace is here, 13M download, when unpacked it is more than 1.2 GB.
> https://gnu.support/files/tmp/emacs.strace-2020-11-18-14:42:59-Wednesday.lz

I've looked at that file, but couldn't see any smoking guns.  It shows
that your brk goes up and up and up until it reaches more than 7GB.
Some of the requests come in groups, totaling about 5MB, not sure why
(these groups always follow a call to timerfd_settime, which seems to
hint that we are setting an atimer for something).  However, without
time stamps for each syscall, it is hard to tell whether these series
of calls to 'brk' are indeed made one after the other, nor whether
they are indeed related to something we use atimers for, because it is
unknown how much time passed between these calls.

I think you should try using the malloc tracing tools pointed to here:

  https://debbugs.gnu.org/cgi/bugreport.cgi?bug=43389#158

Also, next time your vsize is several GBytes, please see how much do
your buffers take, by evaluating this form:

 (let ((size 0))
   (dolist (buffer (buffer-list) size)
     (setq size (+ size (buffer-size buffer)))))




Reply | Threaded
Open this post in threaded view
|

bug#43389: 28.0.50; Emacs memory leaks using hard disk all time

Jean Louis
* Eli Zaretskii <[hidden email]> [2020-11-19 17:38]:
> I think you should try using the malloc tracing tools pointed to here:
>
>   https://debbugs.gnu.org/cgi/bugreport.cgi?bug=43389#158

When running for long time Emacs will crush at certain point of time
as my hard disk get full as /tmp is just about 2 gigabytes. I did not
understand Carlos how to change the location for files.



Reply | Threaded
Open this post in threaded view
|

bug#43389: 28.0.50; Emacs memory leaks using hard disk all time

Eli Zaretskii
> Date: Fri, 20 Nov 2020 06:16:26 +0300
> From: Jean Louis <[hidden email]>
> Cc: [hidden email], [hidden email], [hidden email],
>   [hidden email], [hidden email], [hidden email]
>
> * Eli Zaretskii <[hidden email]> [2020-11-19 17:38]:
> > I think you should try using the malloc tracing tools pointed to here:
> >
> >   https://debbugs.gnu.org/cgi/bugreport.cgi?bug=43389#158
>
> When running for long time Emacs will crush at certain point of time
> as my hard disk get full as /tmp is just about 2 gigabytes. I did not
> understand Carlos how to change the location for files.

Carlos, could you please help Jean to direct the traces to a place
other than /tmp?



Reply | Threaded
Open this post in threaded view
|

bug#43389: 28.0.50; Emacs memory leaks using hard disk all time

Jean Louis
* Eli Zaretskii <[hidden email]> [2020-11-20 03:11]:

> > Date: Fri, 20 Nov 2020 06:16:26 +0300
> > From: Jean Louis <[hidden email]>
> > Cc: [hidden email], [hidden email], [hidden email],
> >   [hidden email], [hidden email], [hidden email]
> >
> > * Eli Zaretskii <[hidden email]> [2020-11-19 17:38]:
> > > I think you should try using the malloc tracing tools pointed to here:
> > >
> > >   https://debbugs.gnu.org/cgi/bugreport.cgi?bug=43389#158
> >
> > When running for long time Emacs will crush at certain point of time
> > as my hard disk get full as /tmp is just about 2 gigabytes. I did not
> > understand Carlos how to change the location for files.
>
> Carlos, could you please help Jean to direct the traces to a place
> other than /tmp?

I am now following this strategy here:
https://github.com/jemalloc/jemalloc/wiki/Use-Case%3A-Leak-Checking

I have run emacs -Q for very short time, with:

MALLOC_CONF=prof_leak:true,lg_prof_sample:0,prof_final:true \
LD_PRELOAD=/package/lib/jemalloc/lib/libjemalloc.so.2 emacs -Q

and there are PDF files generated. I also wish to mention that I use 2
dynamic modules, one is emacs-libpq and other emacs-libvterm if that
influences overall.

You may know easier how to interpret those files and may spot
something. This Emacs session was running just a minute or something.

https://gnu.support/files/tmp/2020-11-22/jeprof.26889.0.f.heap
https://gnu.support/files/tmp/2020-11-22/jeprof.26889.0.f.heap.pdf
https://gnu.support/files/tmp/2020-11-22/jeprof.26915.0.f.heap
https://gnu.support/files/tmp/2020-11-22/jeprof.26915.0.f.heap.pdf
https://gnu.support/files/tmp/2020-11-22/jeprof.26918.0.f.heap
https://gnu.support/files/tmp/2020-11-22/jeprof.26918.0.f.heap.pdf
https://gnu.support/files/tmp/2020-11-22/jeprof.26921.0.f.heap
https://gnu.support/files/tmp/2020-11-22/jeprof.26921.0.f.heap.pdf
https://gnu.support/files/tmp/2020-11-22/jeprof.26922.0.f.heap
https://gnu.support/files/tmp/2020-11-22/jeprof.26922.0.f.heap.pdf
https://gnu.support/files/tmp/2020-11-22/jeprof.26923.0.f.heap
https://gnu.support/files/tmp/2020-11-22/jeprof.26923.0.f.heap.pdf
https://gnu.support/files/tmp/2020-11-22/jeprof.26924.0.f.heap
https://gnu.support/files/tmp/2020-11-22/jeprof.26924.0.f.heap.pdf
https://gnu.support/files/tmp/2020-11-22/jeprof.26925.0.f.heap
https://gnu.support/files/tmp/2020-11-22/jeprof.26925.0.f.heap.pdf
https://gnu.support/files/tmp/2020-11-22/jeprof.26931.0.f.heap
https://gnu.support/files/tmp/2020-11-22/jeprof.26931.0.f.heap.pdf

I am now running new session and will have maybe quite different data
after hours of run.

Jean



Reply | Threaded
Open this post in threaded view
|

bug#43389: 28.0.50; Emacs memory leaks using hard disk all time

Carlos O'Donell-2
On 11/22/20 3:16 PM, Eli Zaretskii wrote:

>> Date: Sun, 22 Nov 2020 22:52:14 +0300
>> From: Jean Louis <[hidden email]>
>> Cc: [hidden email], [hidden email], [hidden email],
>>   [hidden email], [hidden email], [hidden email]
>>
>> I am now following this strategy here:
>> https://github.com/jemalloc/jemalloc/wiki/Use-Case%3A-Leak-Checking
>
> That uses a different implementation of malloc, so I'm not sure it
> will help us.

Correct, that is a different malloc implementation and may have
completely different behaviour for your given workload. That is
not to say that it isn't viable solution to try another allocator
that matches your workload. However, in this bug we're trying to
determine why the "default" configuration of emacs and glibc's
allocator causes memory usage to grow.

We want to run the glibc malloc algorithms because that is the
implementation under which we are observing the increased memory
pressure. The tracer I've suggested will get us an API trace
that we can use to determine if it is actually API calls that
are causing an increase in the memory usage or if it's an
algorithmic issue. It is not always obvious to see from the
API calls, but having the trace is better than not.

--
Cheers,
Carlos.




Reply | Threaded
Open this post in threaded view
|

bug#43389: 28.0.50; Emacs memory leaks using hard disk all time

Jean Louis
In reply to this post by Jean Louis
* Eli Zaretskii <[hidden email]> [2020-11-22 23:17]:

> > Date: Sun, 22 Nov 2020 22:52:14 +0300
> > From: Jean Louis <[hidden email]>
> > Cc: [hidden email], [hidden email], [hidden email],
> >   [hidden email], [hidden email], [hidden email]
> >
> > I am now following this strategy here:
> > https://github.com/jemalloc/jemalloc/wiki/Use-Case%3A-Leak-Checking
>
> That uses a different implementation of malloc, so I'm not sure it
> will help us.

It will not help if you are able to interpret the PDF reports and you
do not see anything helpful. If you do interpret those PDF reports
please tell me as such could be useful to find possible causes or find
other issues in Emacs.

Does this here tells you anything?
https://gnu.support/files/tmp/2020-11-22/jeprof.26889.0.f.heap.pdf

Does this add module isra.0 inside tells you anything?
https://gnu.support/files/tmp/2020-11-22/jeprof.26922.0.f.heap.pdf

I am using dynamic modules like vterm and libpq, can that influence
memory or create memory leaks?

What is tst_post_reentrancy_raw, is that something that eats memory?

I am still running this session with jemalloc and I wish to see if
anything will happen that blocks the work similar how it blocks with
the normal run. This helps slightly in determination. As if run of
Emacs with jemalloc does not cause problems one time, maybe 2-5 times
or 10 times, that may be deduce problem to standard malloc and not
Emacs.

Then in the next session I will try again the tools as described and
submit data.

To help me understand, do you think problem is in Emacs or in glibc
malloc?



Reply | Threaded
Open this post in threaded view
|

bug#43389: 28.0.50; Emacs memory leaks using hard disk all time

Eli Zaretskii
On November 23, 2020 10:11:22 AM GMT+02:00, Jean Louis <[hidden email]> wrote:

> * Eli Zaretskii <[hidden email]> [2020-11-22 23:17]:
> > > Date: Sun, 22 Nov 2020 22:52:14 +0300
> > > From: Jean Louis <[hidden email]>
> > > Cc: [hidden email], [hidden email], [hidden email],
> > >   [hidden email], [hidden email],
> [hidden email]
> > >
> > > I am now following this strategy here:
> > >
> https://github.com/jemalloc/jemalloc/wiki/Use-Case%3A-Leak-Checking
> >
> > That uses a different implementation of malloc, so I'm not sure it
> > will help us.
>
> It will not help if you are able to interpret the PDF reports and you
> do not see anything helpful. If you do interpret those PDF reports
> please tell me as such could be useful to find possible causes or find
> other issues in Emacs.

Granted, I looked at the reports before writing that response.  I don't see there anything related to Emacs code.

> Does this here tells you anything?
> https://gnu.support/files/tmp/2020-11-22/jeprof.26889.0.f.heap.pdf

It says that most of memory was allocated by a subroutine of jemalloc.  As I'm not familiar with how jemalloc works, I see no way for us to draw any significant conclusions from that.

> Does this add module isra.0 inside tells you anything?

AFAIU, it's some internal jemalloc midule.

> I am using dynamic modules like vterm and libpq, can that influence
> memory or create memory leaks?

I have no idea, but I don't think I see any of their functions in these reports.

> What is tst_post_reentrancy_raw, is that something that eats memory?

I don't know.  It's something internal to jemalloc.

> I am still running this session with jemalloc and I wish to see if
> anything will happen that blocks the work similar how it blocks with
> the normal run. This helps slightly in determination. As if run of
> Emacs with jemalloc does not cause problems one time, maybe 2-5 times
> or 10 times, that may be deduce problem to standard malloc and not
> Emacs.

The glibc malloc is the prime suspect anyway.  I don't really believe Emacs had such a glaring memory leak.  So trying different malloc implementations is from my POV waste of time at this stage.

> Then in the next session I will try again the tools as described and
> submit data.
>
> To help me understand, do you think problem is in Emacs or in glibc
> malloc?

I suspect the problem is in how we use glibc's malloc -- there are some usage patterns that cause glibc to be suboptimal in its memory usage, and I hope we will find ways to fine tune it to our needs.

But that is just a guess, and so I wish you'd use the tools pointed out by Carlos, because they are the most efficient way of collecting evidence that might allow us to make some progress here.

We have the attention of the best experts on the issue; let's use their attention and their time as best as we possibly can.

TIA




Reply | Threaded
Open this post in threaded view
|

bug#43389: 28.0.50; Emacs memory leaks using hard disk all time

Jean Louis
In reply to this post by Eli Zaretskii
The session I was running with jemalloc memory leak logging is
finished now.

Just the same thing happened. It started getting slower and slower.

In the IceWM window manager I have visual representation of memory
usage and that is how I get feeling, there is also tooltip telling me
that more and more memory is used. When it starts to swap like 3 GB
then I turn on symon-mode and in Emacs I see more and more swapping.

The heap file is here 24M, maybe not needed for review:
https://gnu.support/files/tmp/2020-11-23/jeprof.23826.0.f.heap

Visualization is here 20K PDF file:
https://gnu.support/files/tmp/2020-11-23/jeprof.23826.0.f.heap.pdf

Do you see anything interesting inside that should tell about memory leaks?

Jean




Reply | Threaded
Open this post in threaded view
|

bug#43389: 28.0.50; Emacs memory leaks using hard disk all time

Jean Louis
In reply to this post by Jean Louis
* Carlos O'Donell <[hidden email]> [2020-11-23 06:35]:

> On 11/19/20 10:16 PM, Jean Louis wrote:
> > * Eli Zaretskii <[hidden email]> [2020-11-19 17:38]:
> >> I think you should try using the malloc tracing tools pointed to here:
> >>
> >>   https://debbugs.gnu.org/cgi/bugreport.cgi?bug=43389#158
> >
> > When running for long time Emacs will crush at certain point of time
> > as my hard disk get full as /tmp is just about 2 gigabytes. I did not
> > understand Carlos how to change the location for files.
>  
> The glibc malloc tracer functionality can be adjusted with environment
> variables.
>
> Example:
>
> MTRACE_CTL_VERBOSE=1 MTRACE_CTL_FILE=./ls.mtr LD_PRELOAD=./libmtrace.so ls
> mtrace: writing to ./ls.mtr.350802
>
> The appended PID helps keep the files distinct (and includes a sequence
> number in the event of conflict).

Alright, thank you.

My session started with it.




Reply | Threaded
Open this post in threaded view
|

bug#43389: 28.0.50; Emacs memory leaks using hard disk all time

Jean Louis
In reply to this post by Jean Louis
* Eli Zaretskii <[hidden email]> [2020-11-22 23:17]:

> > Date: Sun, 22 Nov 2020 22:52:14 +0300
> > From: Jean Louis <[hidden email]>
> > Cc: [hidden email], [hidden email], [hidden email],
> >   [hidden email], [hidden email], [hidden email]
> >
> > I am now following this strategy here:
> > https://github.com/jemalloc/jemalloc/wiki/Use-Case%3A-Leak-Checking
>
> That uses a different implementation of malloc, so I'm not sure it
> will help us.

This is how I have run the shorter Emacs session until it got blocked:

MTRACE_CTL_VERBOSE=1 MTRACE_CTL_FILE=/home/data1/protected/tmp/mtraceEMACS.mtr LD_PRELOAD=/home/data1/protected/Programming/git/glibc-malloc-trace-utils/libmtrace.so emacs >> $DEBUG 2>&1

And here is mtrace:

https://gnu.support/files/tmp/2020-11-23/mtraceEMACS.mtr.9294.lz

I cannot run Emacs that way as something happens and Emacs get
blocked. Problem arrives with M-s M-w to search for anything on
Internet with eww. Anything blocks. And I get message:

error in process filter: Quit

after that C-g does not work, I cannot kill buffer, I cannot save the
current work or other buffers, I cannot switch from buffer to buffer
neither open any menu.

Debugging requires longer run sessions and actual work in Emacs.

This happens all the time when I run Emacs like the above example
command.

Unless there is safer way for debugging the above one is not
functional as it blocks everything and I do use incidentally or
accidentally eww in the work.

I hope that something will be visible from that mtrace.



Reply | Threaded
Open this post in threaded view
|

bug#43389: 28.0.50; Emacs memory leaks using hard disk all time

Eli Zaretskii
In reply to this post by Jean Louis
> Date: Mon, 23 Nov 2020 13:59:47 +0300
> From: Jean Louis <[hidden email]>
> Cc: [hidden email], [hidden email], [hidden email],
>   [hidden email], [hidden email], [hidden email]
>
> In the IceWM window manager I have visual representation of memory
> usage and that is how I get feeling, there is also tooltip telling me
> that more and more memory is used. When it starts to swap like 3 GB
> then I turn on symon-mode and in Emacs I see more and more swapping.

I think I described how to write an Emacs function that you could use
to watch the vsize of the Emacs process and alert you to it being
above some threshold.

> The heap file is here 24M, maybe not needed for review:
> https://gnu.support/files/tmp/2020-11-23/jeprof.23826.0.f.heap
>
> Visualization is here 20K PDF file:
> https://gnu.support/files/tmp/2020-11-23/jeprof.23826.0.f.heap.pdf
>
> Do you see anything interesting inside that should tell about memory leaks?

I'm not sure.  I think I see that you have some timer that triggers a
lot of memory allocations because it conses a lot of Lisp objects.
Whether that is part of the problem or not is not clear.

Next time when your session causes the system to swap, please type:

  M-: (garbage-collect) RET

and post here the output of that (it should be a list of numbers
whose meanings are explained in the doc string of garbage-collect).

Also, I think I asked to tell how large are your buffers by evaluation
the following (again, near the point where your session causes the
system to page heavily):

  (let ((size 0))
    (dolist (buffer (buffer-list) size)
      (setq size (+ size (buffer-size buffer)))))

It is important to have both these pieces of information from the same
session at the same time near the point where you must kill Emacs, so
that we know how much memory is actually used by your session at that
point (as opposed to memory that is "free" in the heap, but was  not
returned to the OS).

Thanks.



Reply | Threaded
Open this post in threaded view
|

bug#43389: 28.0.50; Emacs memory leaks using hard disk all time

Carlos O'Donell-2
In reply to this post by Jean Louis
On 11/23/20 8:27 AM, Jean Louis wrote:

> * Eli Zaretskii <[hidden email]> [2020-11-22 23:17]:
>>> Date: Sun, 22 Nov 2020 22:52:14 +0300
>>> From: Jean Louis <[hidden email]>
>>> Cc: [hidden email], [hidden email], [hidden email],
>>>   [hidden email], [hidden email], [hidden email]
>>>
>>> I am now following this strategy here:
>>> https://github.com/jemalloc/jemalloc/wiki/Use-Case%3A-Leak-Checking
>>
>> That uses a different implementation of malloc, so I'm not sure it
>> will help us.
>
> This is how I have run the shorter Emacs session until it got blocked:
>
> MTRACE_CTL_VERBOSE=1 MTRACE_CTL_FILE=/home/data1/protected/tmp/mtraceEMACS.mtr LD_PRELOAD=/home/data1/protected/Programming/git/glibc-malloc-trace-utils/libmtrace.so emacs >> $DEBUG 2>&1
>
> And here is mtrace:
>
> https://gnu.support/files/tmp/2020-11-23/mtraceEMACS.mtr.9294.lz
>
> I cannot run Emacs that way as something happens and Emacs get
> blocked. Problem arrives with M-s M-w to search for anything on
> Internet with eww. Anything blocks. And I get message:
>
> error in process filter: Quit

Sorry, please drop MTRACE_CTL_VERBOSE=1, as it adds output to stdout
which may affect the process if using pipes.

--
Cheers,
Carlos.




Reply | Threaded
Open this post in threaded view
|

bug#43389: 28.0.50; Emacs memory leaks using hard disk all time

arthur miller
In reply to this post by Eli Zaretskii
Eli Zaretskii <[hidden email]> writes:

>> Date: Mon, 23 Nov 2020 13:59:47 +0300
>> From: Jean Louis <[hidden email]>
>> Cc: [hidden email], [hidden email], [hidden email],
>>   [hidden email], [hidden email], [hidden email]
>>
>> In the IceWM window manager I have visual representation of memory
>> usage and that is how I get feeling, there is also tooltip telling me
>> that more and more memory is used. When it starts to swap like 3 GB
>> then I turn on symon-mode and in Emacs I see more and more swapping.
>
> I think I described how to write an Emacs function that you could use
> to watch the vsize of the Emacs process and alert you to it being
> above some threshold.
>
>> The heap file is here 24M, maybe not needed for review:
>> https://gnu.support/files/tmp/2020-11-23/jeprof.23826.0.f.heap
>>
>> Visualization is here 20K PDF file:
>> https://gnu.support/files/tmp/2020-11-23/jeprof.23826.0.f.heap.pdf
>>
>> Do you see anything interesting inside that should tell about memory leaks?
>
> I'm not sure.  I think I see that you have some timer that triggers a
> lot of memory allocations because it conses a lot of Lisp objects.
> Whether that is part of the problem or not is not clear.
>
> Next time when your session causes the system to swap, please type:
>
>   M-: (garbage-collect) RET
>
> and post here the output of that (it should be a list of numbers
> whose meanings are explained in the doc string of garbage-collect).
>
> Also, I think I asked to tell how large are your buffers by evaluation
> the following (again, near the point where your session causes the
> system to page heavily):
>
>   (let ((size 0))
>     (dolist (buffer (buffer-list) size)
>       (setq size (+ size (buffer-size buffer)))))
>
> It is important to have both these pieces of information from the same
> session at the same time near the point where you must kill Emacs, so
> that we know how much memory is actually used by your session at that
> point (as opposed to memory that is "free" in the heap, but was  not
> returned to the OS).
>
> Thanks.
For me it happends like really, really fast. Things work normally, and
then suddenly everythign freezes, and after first freeze, it takes for
every to see result of any keypress. For example video in Firefox gets
slow down to like a frame per minut or so; I can see that system is
alive, but it is impossible to type something like (garbage-collect) and
see the result; I would be sitting here for a day :-).

The only thing I can is switch to another console, and then back. By
that time Emacs process is restarted and everything is normal. I don't
use swap file at all, and I can't believe that Emacs is eating up 32 gig
or RAM either. However I can't type any command to see what it is
peeking at since everything is efefctively frozen. I have seen it at 800
meg on my machine at some time, but it is far away from 32 gig I have.



Reply | Threaded
Open this post in threaded view
|

bug#43389: 28.0.50; Emacs memory leaks using hard disk all time

Eli Zaretskii
In reply to this post by Eli Zaretskii
> From: Arthur Miller <[hidden email]>
> Cc: Jean Louis <[hidden email]>,  [hidden email],
>   [hidden email],  [hidden email],  [hidden email],
>   [hidden email],  [hidden email]
> Date: Mon, 23 Nov 2020 18:19:32 +0100
>
> > The glibc malloc is the prime suspect anyway.  I don't really believe Emacs had
> > such a glaring memory leak.
>
> This has to be something introduced fairly recently, right?

Maybe, I'm not sure.  Since we introduced the pdumper, we use malloc
somewhat differently, and OTOH glibc removed some of the malloc hooks
we used to use in versions of Emacs before 26.  In addition, glibc is
also being developed, and maybe some change there somehow triggered
this.

As you see, there's more than one factor that could possibly be
related.

> I didn't have any such problems before, but since maybe few weeks ago, I
> have also experienced heavy lockdowns of my entire OS. To the point
> where entire X11 got unresposnsive, when it happens I can't even switch
> to terminal to kill Emacs. What I do is Alt-Shift to another virtual
> linux console. I don't even need to log into the system in that console,
> I can then Alt-Shift 1 to go back to one I am logged into, and
> everything is normal. Emacs is every time restarted by systemd and
> everything is repsonsive and working as normal.
>
> This started sometime ago; and I have noticed that it happens when I was
> cleaning my disk and reading big directories in Dired (I have some with
> ~7k-10k files in them). I was using Helm to complete paths when I was
> shifting fiels and folders around. After maybe hour or so I would
> experience big slowdown. I don't have swap file on my system enabled at
> all, so I am not sure what was going, but I didn't have time to
> participate in this memory leak thing yet. I haven't experienced any
> problems since I recompiled Emacs last time, which was in 18th (last
> Wendesday). I have recompiled without Gtk this time, but I have no idea
> if it has anything to do with the issue, was just a wild shot to see if
> things are better.

If the problem is memory, I suggest to look at the system log to see
if there are any signs of that.



12345