Should use of mallopt depend on DOUG_LEA_MALLOC? (WAS: bug#38345)

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Should use of mallopt depend on DOUG_LEA_MALLOC? (WAS: bug#38345)

Noam Postavsky
[Moving to emacs-devel, since this doesn't seem to be about the bug as such.]

Eli Zaretskii <[hidden email]> writes:

>> From: Noam Postavsky <[hidden email]>
>> Cc: [hidden email],  [hidden email],  [hidden email]
>> Date: Sat, 07 Dec 2019 14:25:06 -0500
>>
>> > So does this mean modern glibc has no knobs to tailor its behavior to
>> > various needs, like set the threshold for using mmap, etc.?  IOW, no
>> > more support for the likes of mallopt?
>>
>> Looks like mallopt is still there, but Emacs' DOUG_LEA_MALLOC code
>> depends on malloc_set_state and malloc_get_state which were removed in
>> glibc 2.25 (2017).
>> https://sourceware.org/ml/libc-alpha/2017-02/msg00079.html
>
> Thanks.  Then I don't understand why our calls to mallopt are
> conditioned on DOUG_LEA_MALLOC.  What they are supposed to do should
> be good for modern glibc versions as well.

It could be just accidental, since DOUG_LEA_MALLOC (i.e.,
malloc_get_state & malloc_set_state) used to always come with mallopt.
Although it looks like most of the mallopt calls are only done if
mmap_lisp_allowed_p returns false.  mmap_lisp_allowed_p is only defined
for DOUG_LEA_MALLOC, so I'm not sure what that condition should evaluate
to if DOUG_LEA_MALLOC is not defined.

I also don't really have any idea what the mallopt calls do (that is, I
can see in the documentation that they modify such and such parameter of
the allocator, but I don't know what that means in practice: faster,
slower, less RAM, more RAM, etc).  So I hope some more clueful people will
chime in.

Reply | Threaded
Open this post in threaded view
|

Re: Should use of mallopt depend on DOUG_LEA_MALLOC? (WAS: bug#38345)

Eli Zaretskii
> From: Noam Postavsky <[hidden email]>
> Date: Fri, 13 Dec 2019 07:12:09 -0500
>
> >> Looks like mallopt is still there, but Emacs' DOUG_LEA_MALLOC code
> >> depends on malloc_set_state and malloc_get_state which were removed in
> >> glibc 2.25 (2017).
> >> https://sourceware.org/ml/libc-alpha/2017-02/msg00079.html
> >
> > Thanks.  Then I don't understand why our calls to mallopt are
> > conditioned on DOUG_LEA_MALLOC.  What they are supposed to do should
> > be good for modern glibc versions as well.
>
> It could be just accidental, since DOUG_LEA_MALLOC (i.e.,
> malloc_get_state & malloc_set_state) used to always come with mallopt.
> Although it looks like most of the mallopt calls are only done if
> mmap_lisp_allowed_p returns false.  mmap_lisp_allowed_p is only defined
> for DOUG_LEA_MALLOC, so I'm not sure what that condition should evaluate
> to if DOUG_LEA_MALLOC is not defined.

I think the fact that mmap_lisp_allowed_p is only defined when
DOUG_LEA_MALLOC is just the consequence of its being used under that
condition.  IOW, someone tried to reduce the memory footprint by
#ifdef'ing away functions that aren't use otherwise.  I see nothing in
mmap_lisp_allowed_p that is specific to Doug Lea malloc per se.

> I also don't really have any idea what the mallopt calls do (that is, I
> can see in the documentation that they modify such and such parameter of
> the allocator, but I don't know what that means in practice: faster,
> slower, less RAM, more RAM, etc).

AFAIU:

  . mallopt (M_MMAP_MAX, 0) forces malloc not to use mmap when we
    think it's sub-optimal
  . mallopt (M_TRIM_THRESHOLD, 128 * 1024) makes us use sbrk less often
  . mallopt (M_MMAP_THRESHOLD, 64 * 1024) favors mmap allocations for
    smaller objects/buffers than the default, thus making it easier
    for Emacs to release unused memory to the system
  . mallopt (M_MMAP_MAX, MMAP_MAX_AREAS) tells the library that there
    should be no limit on allocations via mmap, since Emacs is a
    single-threaded application

> So I hope some more clueful people will chime in.

Seconded.

Reply | Threaded
Open this post in threaded view
|

Re: Should use of mallopt depend on DOUG_LEA_MALLOC?

Stefan Monnier
>> I also don't really have any idea what the mallopt calls do (that is, I
>> can see in the documentation that they modify such and such parameter of
>> the allocator, but I don't know what that means in practice: faster,
>> slower, less RAM, more RAM, etc).

For those in `init_alloc_once_for_pdumper`, I can't help you.
But the ones conditionalized on `mmap_lisp_allowed_p` are used not as
optimizations but to prevent use of mmap within the malloc
implementation in the two known cases where it can break our
(unportable) assumptions:

- when we use MSB bits for the Lisp_Object tags and hence cannot use all
  of the addressable memory for Lisp objects.
  [ Hopefully, there are no platforms in this category remaining, but
  I don't know this to be the case for sure.  ]

- when we're about to do the unexec dump because the unexec code doesn't
  handle those mmap'd areas.


        Stefan


Reply | Threaded
Open this post in threaded view
|

Re: Should use of mallopt depend on DOUG_LEA_MALLOC?

Paul Eggert
On 12/13/19 7:22 AM, Stefan Monnier wrote:
> For those in `init_alloc_once_for_pdumper`, I can't help you.

Those mallopt calls shouldn't be needed.

> But the ones conditionalized on `mmap_lisp_allowed_p` are used not as
> optimizations but to prevent use of mmap within the malloc
> implementation in the two known cases where it can break our
> (unportable) assumptions:

Yes, we need to keep those only for the vanishingly small cases where we
use unexec and a Doug Lea-like malloc.

I looked through Emacs's uses of mallopt. There's a lot of cargo-cult
cruft there that should go. Plus, I expect there's a real bug with newer
glibc (so mallopt isn't called) but unexec is still used (because the
builder asked for it - a mistake in my view) and where Emacs doesn't
disable mmap properly.

Proposed patch attached. The basic idea is to use "#ifdef M_MMAP_MAX"
instead of "#ifdef DOUG_LEA_MALLOC" to decide whether to call mallopt
with M_MMAP_MAX. Plus, stop fiddling with mallopt parameters that are
problematic given that glibc malloc has mutated so much (and is likely
to mutate further).

0001-Simplify-use-of-mallopt.patch (6K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Should use of mallopt depend on DOUG_LEA_MALLOC?

Eli Zaretskii
> Cc: Noam Postavsky <[hidden email]>, [hidden email]
> From: Paul Eggert <[hidden email]>
> Date: Fri, 13 Dec 2019 13:55:36 -0800
>
> On 12/13/19 7:22 AM, Stefan Monnier wrote:
> > For those in `init_alloc_once_for_pdumper`, I can't help you.
>
> Those mallopt calls shouldn't be needed.

Can you elaborate why?

And if that's indeed true, then what is our response to reports such
as the one in https://debbugs.gnu.org/cgi/bugreport.cgi?bug=38345#71 ?
I thought a judicious use of mallopt would allow us to come close to
jemalloc, but if not, then what would?  Assuming what Ihor Radchenko
sees and is reported in that discussion is normal operation of glibc's
malloc, don't we want to be able to return memory to the OS more
eagerly than the default malloc configuration does?

> Proposed patch attached. The basic idea is to use "#ifdef M_MMAP_MAX"
> instead of "#ifdef DOUG_LEA_MALLOC" to decide whether to call mallopt
> with M_MMAP_MAX. Plus, stop fiddling with mallopt parameters that are
> problematic given that glibc malloc has mutated so much (and is likely
> to mutate further).

Thanks, but let's postpone such cleanups until after the release
branch is cut (hopefully soon).  I would like to leave the
unexec-related code in its current form for Emacs 27.1, and only fix
any bugs that will be reported against it.