bug#41194: 28.0.50; [feature/native-comp] fibn benchmark exhausts memory

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

bug#41194: 28.0.50; [feature/native-comp] fibn benchmark exhausts memory

Kévin Le Gouguec
As detailed in bug#41077, I compiled the feature/native-comp branch[1]
on a low-end-ish laptop[2], blacklisting char-fold.el since native
compilation took too much memory for this file.

As reported there, I get appreciable speedups on the benchmarks
distributed in the elisp-benchmarks ELPA package, except for one
specific test: fibn.  When native-compiled (with any value of
comp-speed), this test makes Emacs abort with the message:

> Memory exhausted--use C-x s then exit and restart Emacs

I've plotted some graphs showing fibn's memory consumption for every
value of comp-speed, as well as for emacs master[3] (where all 3
iterations run to completion):



See [4] for the methodology, source material and logs.  Out of curiosity
I've disassembled the compiled functions in fibn.el for comp-speed 2;
I'm not fluent enough in assembly to spot anything obviously wrong[5].


Let me know if there are things you would like me to try out in order to
investigate this.  


[1] Commit 92cf4bb8cc3da81f4877a734b9e9089ac4b89e85.

[2] Samsung NC10, 2GB RAM, 2GB HDD swap, on Debian Buster.

[3] Commit 9d8fc3a598090da518fcdd5c0503ed0f7faa41a9.

[4] .eln files compiled with:

    (dotimes (i 4) (rename-file (let ((comp-speed i)) (native-compile "fibn.el"))
                                (format "fibn-%d.eln" i)))

    Script used to collect measurements on branch native-comp:


    Script used to collect measurements on branch master:


    Script used to plot the graphs:


    Measurements:


    Logs for native-comp:


    Logs for master:


[5]

fibn.pdf (108K) Download Attachment
repro.sh (944 bytes) Download Attachment
master.sh (644 bytes) Download Attachment
plot.py (2K) Download Attachment
measurements.tgz (24K) Download Attachment
repro.log (5K) Download Attachment
master.log (1K) Download Attachment
elb-fibn.s (4K) Download Attachment
elb-fibn-entry.s (1K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

bug#41194: 28.0.50; [feature/native-comp] fibn benchmark exhausts memory

Andrea Corallo
Kévin Le Gouguec <[hidden email]> writes:

> As detailed in bug#41077, I compiled the feature/native-comp branch[1]
> on a low-end-ish laptop[2], blacklisting char-fold.el since native
> compilation took too much memory for this file.
>
> As reported there, I get appreciable speedups on the benchmarks
> distributed in the elisp-benchmarks ELPA package, except for one
> specific test: fibn.  When native-compiled (with any value of
> comp-speed), this test makes Emacs abort with the message:
>
>> Memory exhausted--use C-x s then exit and restart Emacs
>
> I've plotted some graphs showing fibn's memory consumption for every
> value of comp-speed, as well as for emacs master[3] (where all 3
> iterations run to completion):
>
>
>
> See [4] for the methodology, source material and logs.  Out of curiosity
> I've disassembled the compiled functions in fibn.el for comp-speed 2;
> I'm not fluent enough in assembly to spot anything obviously wrong[5].
>
>
> Let me know if there are things you would like me to try out in order to
> investigate this.  

Hi Kevin,

sorry for the very late reply I just had a look.

The following is going on: being on 32bit system you get into bignums
end this allocates memory while is looping inside `elb-fibn'.
Unfortunatelly there's no opportunity of passing through maybe_gc to
trigger garbage collection because I've not implemented the equivalent
of what is the quitcounter in the byte interpreter.

I'll come-up with a patch to generate the equivalent code of the
quitcounter when compiling for speed < 3.

Thanks for the feedback

  Andrea

PS FYI I think memory consumption during compilation (as compile time)
should be now considerably lowered (but I've still haven't measured it).

--
[hidden email]



Reply | Threaded
Open this post in threaded view
|

bug#41194: 28.0.50; [feature/native-comp] fibn benchmark exhausts memory

Andrea Corallo
Kévin Le Gouguec <[hidden email]> writes:

> Andrea Corallo <[hidden email]> writes:
>
>> sorry for the very late reply I just had a look.
>
> No need to apologize!  You've obviously been busy on other fronts.
>
>> The following is going on: being on 32bit system you get into bignums
>> end this allocates memory while is looping inside `elb-fibn'.
>> Unfortunatelly there's no opportunity of passing through maybe_gc to
>> trigger garbage collection because I've not implemented the equivalent
>> of what is the quitcounter in the byte interpreter.
>>
>> I'll come-up with a patch to generate the equivalent code of the
>> quitcounter when compiling for speed < 3.
>>
>> Thanks for the feedback
>>
>>   Andrea
>>
>> PS FYI I think memory consumption during compilation (as compile time)
>> should be now considerably lowered (but I've still haven't measured it).
>
> Alright!  I'll keep an eye on your branch and on the bugtracker, and
> recompile & run my measurements when I spot your patch.

Hi Kévin,

5a55a845a7 * Implement 'maybe_gc_or_quit' to allow correct GC in compiled Lisp
34ed9d2498 * Introduce latches

are fixing here this issue, now running fibn at speed 2 the memory
consuptions stays constant.

You'll still see the same behavior at speed 3 because there the
programmer is supposedly responsible to add manually possible GC entry
points.

I think now 32bit should be in a much better state but I hadn't time to
test it still.

Thanks!

  Andrea

--
[hidden email]



Reply | Threaded
Open this post in threaded view
|

bug#41194: 28.0.50; [feature/native-comp] fibn benchmark exhausts memory

Kévin Le Gouguec
Andrea Corallo <[hidden email]> writes:

> I think now 32bit should be in a much better state but I hadn't time to
> test it still.

Alright!  I've pulled the changes and started recompiling (without
blacklisting char-fold.el, because I'm such an optimist).  I'll followup
when compilation has finished to let you know how it went with fibn!

Thank you for your efforts.



Reply | Threaded
Open this post in threaded view
|

bug#41194: 28.0.50; [feature/native-comp] fibn benchmark exhausts memory

Kévin Le Gouguec
In reply to this post by Andrea Corallo
Andrea Corallo <[hidden email]> writes:

> 5a55a845a7 * Implement 'maybe_gc_or_quit' to allow correct GC in compiled Lisp
> 34ed9d2498 * Introduce latches
>
> are fixing here this issue, now running fibn at speed 2 the memory
> consuptions stays constant.

Alright, feedback time!

Commits:
- master: 82a632edc8b80bf16d9b9f205474bf9724b084c0
- feature/native-comp: 5a55a845a7c426e82e8a6a6d02bc4a39992871e3


The Good News
=============

- I can compile the branch without adding files to the default
  blacklist,

- compilation (-j1) now only takes less than 6 hours instead of 3 days
  (compared with 1.5 hour on master),

- at no point during compilation did the system swap; in fact, I could
  probably have risked a -j2 build,

- elisp-benchmarks doesn't show any obvious performance regression
  (… though see Bad News).

For details on memory usage during compilation, see attached memory
profile (to compare with bug#41077#28).



Few files take more than 3 minutes to compile now:

ELC+ELN emacs-lisp/comp.el      0:09:49
ELC+ELN org/org.el              0:03:39
ELC+ELN ../lisp/window.el       0:03:16
ELC+ELN emacs-lisp/bytecomp.el  0:03:16

For details on performance, see attached tables for master and
feature/native-comp, respectively (to compare with bug#41077#40).




The Bad News
============

- fibn still exhausts memory,
- now pidigits does too.



I've only tried the default comp-speed (2) so far.  I've attached the
results of M-x disassemble on the functions in fibn.el and pidigits.el.



Let me know what I can do to help diagnose this further.  Again, thank
you for the time spent on this issue!

compilation-memory.pdf (98K) Download Attachment
perfs.org (2K) Download Attachment
fibn.pdf (25K) Download Attachment
pidigits.pdf (23K) Download Attachment
elb-eliminate-digit.s (4K) Download Attachment
elb-extract-digit.s (2K) Download Attachment
elb-fibn.s (6K) Download Attachment
elb-fibn-entry.s (711 bytes) Download Attachment
elb-next-term.s (6K) Download Attachment
elb-pidigits.s (8K) Download Attachment
elb-pidigits-entry.s (3K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

bug#41194: 28.0.50; [feature/native-comp] fibn benchmark exhausts memory

Andrea Corallo
Kévin Le Gouguec <[hidden email]> writes:

> Andrea Corallo <[hidden email]> writes:
>
>> 5a55a845a7 * Implement 'maybe_gc_or_quit' to allow correct GC in compiled Lisp
>> 34ed9d2498 * Introduce latches
>>
>> are fixing here this issue, now running fibn at speed 2 the memory
>> consuptions stays constant.
>
> Alright, feedback time!
>
> Commits:
> - master: 82a632edc8b80bf16d9b9f205474bf9724b084c0
> - feature/native-comp: 5a55a845a7c426e82e8a6a6d02bc4a39992871e3
>
>
> The Good News
> =============
>
> - I can compile the branch without adding files to the default
>   blacklist,
>
> - compilation (-j1) now only takes less than 6 hours instead of 3 days
>   (compared with 1.5 hour on master),
>
> - at no point during compilation did the system swap; in fact, I could
>   probably have risked a -j2 build,
>
> - elisp-benchmarks doesn't show any obvious performance regression
>   (… though see Bad News).
>
> For details on memory usage during compilation, see attached memory
> profile (to compare with bug#41077#28).

Hi Kévin,

this is very good new thanks!  If I read correctly we are always under
about 600MB as memory consumption is this correct?

>
> Few files take more than 3 minutes to compile now:
>
> ELC+ELN emacs-lisp/comp.el      0:09:49
> ELC+ELN org/org.el              0:03:39
> ELC+ELN ../lisp/window.el       0:03:16
> ELC+ELN emacs-lisp/bytecomp.el  0:03:16
>
> For details on performance, see attached tables for master and
> feature/native-comp, respectively (to compare with bug#41077#40).
>
>
>
>
> The Bad News
> ============
>
> - fibn still exhausts memory,
> - now pidigits does too.
>
>
>
>
> I've only tried the default comp-speed (2) so far.  I've attached the
> results of M-x disassemble on the functions in fibn.el and pidigits.el.


I suspect we are still looking at a speed 3 run.  How comp-speed was set
for the test?

I ask that because for now speed 3 is written into the stone within
elisp-benchmarks.el, yeah... :/ and so has to be manually changed there.

Thanks!

  Andrea

--
[hidden email]



Reply | Threaded
Open this post in threaded view
|

bug#41194: 28.0.50; [feature/native-comp] fibn benchmark exhausts memory

Kévin Le Gouguec
Andrea Corallo <[hidden email]> writes:

> this is very good new thanks!  If I read correctly we are always under
> about 600MB as memory consumption is this correct?

Yep.  I can post the detailed data here if that's needed; here are the
10 files that took the most virtual memory (vsz):

 1. org/org.el              541.2 MB
 2. gnus/gnus-sum.el        459.3 MB
 3. emacs-lisp/comp.el      455.2 MB
 4. net/tramp.el            408.2 MB
 5. net/tramp-adb.el        390.4 MB
 6. net/tramp-sh.el         371.0 MB
 7. progmodes/cperl-mode.el 356.2 MB
 8. ../lisp/window.el       348.9 MB
 9. gnus/message.el         345.4 MB
10. net/soap-client.el      344.4 MB

And the 10 files that took the most resident memory (rss):

 1. org/org.el              361.6 MB
 2. emacs-lisp/comp.el      287.3 MB
 3. gnus/gnus-sum.el        286.7 MB
 4. net/tramp.el            236.8 MB
 5. net/tramp-adb.el        218.1 MB
 6. net/tramp-sh.el         199.0 MB
 7. progmodes/cperl-mode.el 185.0 MB
 8. ../lisp/window.el       181.6 MB
 9. gnus/message.el         174.0 MB
10. erc/erc.el              173.5 MB

> I suspect we are still looking at a speed 3 run.  How comp-speed was set
> for the test?
>
> I ask that because for now speed 3 is written into the stone within
> elisp-benchmarks.el, yeah... :/ and so has to be manually changed there.

Now that's just sneaky 😛 You're right, it was a speed 3 run.  Setting
comp-speed to 2 in elisp-benchmarks.el, I now get the attached results.


* Results
** master
#+TBLNAME: master
| test           | non-gc avg (s) | gc avg (s) | gcs avg | tot avg (s) | tot avg err (s) |
|----------------+----------------+------------+---------+-------------+-----------------|
| bubble-no-cons |         118.41 |       0.23 |       3 |      118.63 |            0.36 |
| bubble         |          46.29 |      29.34 |     388 |       75.63 |            0.36 |
| dhrystone      |          74.82 |       0.00 |       0 |       74.82 |            0.34 |
| fibn-rec       |          59.50 |       0.00 |       0 |       59.50 |            0.28 |
| fibn-tc        |         147.39 |      96.86 |    1179 |      244.26 |            2.70 |
| fibn           |         320.33 |     276.23 |    3442 |      596.56 |            5.31 |
| flet           |         112.02 |       0.00 |       0 |      112.02 |            0.29 |
| inclist        |         166.30 |       0.00 |       0 |      166.30 |            0.67 |
| listlen-tc     |          76.97 |       0.00 |       0 |       76.97 |            0.53 |
| map-closure    |          81.86 |       0.00 |       0 |       81.86 |            0.31 |
| nbody          |          28.62 |      98.53 |    1303 |      127.14 |            0.21 |
| pcase          |         110.63 |       0.00 |       0 |      110.63 |            0.29 |
| pidigits       |          61.78 |      39.46 |     469 |      101.24 |            0.70 |
|----------------+----------------+------------+---------+-------------+-----------------|
| total          |        1404.92 |     540.65 |    6786 |     1945.57 |            6.13 |
** feature/native-comp
#+TBLNAME: feature/native-comp
| test           | non-gc avg (s) | gc avg (s) | gcs avg | tot avg (s) | tot avg err (s) |
|----------------+----------------+------------+---------+-------------+-----------------|
| bubble-no-cons |          30.12 |       0.16 |       2 |       30.28 |            0.04 |
| bubble         |          18.05 |      27.91 |     352 |       45.96 |            0.33 |
| dhrystone      |          34.26 |       0.00 |       0 |       34.26 |            0.05 |
| fibn-rec       |          26.92 |       0.00 |       0 |       26.92 |            0.03 |
| fibn-tc        |          98.66 |      93.89 |    1096 |      192.55 |            1.66 |
| fibn           |         253.19 |     269.12 |    3192 |      522.31 |            2.17 |
| flet           |          22.92 |       0.00 |       0 |       22.92 |            0.04 |
| inclist        |          11.76 |       0.00 |       0 |       11.76 |            0.16 |
| listlen-tc     |          21.44 |       0.00 |       0 |       21.44 |            0.05 |
| map-closure    |          80.57 |       0.00 |       0 |       80.57 |            0.03 |
| nbody          |          15.62 |      95.66 |    1212 |      111.28 |            0.10 |
| pcase          |          17.64 |       0.00 |       0 |       17.64 |            0.03 |
| pidigits       |          59.82 |      38.86 |     436 |       98.68 |            0.39 |
|----------------+----------------+------------+---------+-------------+-----------------|
| total          |         690.98 |     525.59 |    6291 |     1216.57 |            2.79 |
** ratio
| test           | non-gc avg (s) | gc avg (s) | gcs avg | tot avg (s) |
|----------------+----------------+------------+---------+-------------|
| bubble-no-cons |           3.92 |       1.44 |     1.5 |        3.93 |
| bubble         |           2.56 |       1.05 |     1.1 |        1.64 |
| dhrystone      |           2.18 |            |         |        2.18 |
| fibn-rec       |           2.21 |            |         |        2.21 |
| fibn-tc        |           1.49 |       1.03 |    1.07 |        1.26 |
| fibn           |           1.26 |       1.03 |    1.08 |        1.14 |
| flet           |           4.89 |            |         |        4.89 |
| inclist        |           14.1 |            |         |        14.1 |
| listlen-tc     |            3.6 |            |         |         3.6 |
| map-closure    |           1.02 |            |         |        1.02 |
| nbody          |           1.83 |       1.03 |    1.07 |        1.14 |
| pcase          |           6.31 |            |         |        6.31 |
| pidigits       |           1.03 |       1.02 |    1.08 |        1.02 |
|----------------+----------------+------------+---------+-------------|
| total          |           2.03 |       1.03 |    1.08 |         1.6 |
#+TBLFM: @2$2..@>$> = if (remote(feature/native-comp,@@#$$#)!=0, remote(master,@@#$$#)/remote(feature/native-comp,@@#$$#), string("")); p3


Conclusion: I'm closing this report!  Again, thank you for your efforts.

(And thank you for this opportunity to learn about Org spreadsheets 😉)
Reply | Threaded
Open this post in threaded view
|

bug#41194: 28.0.50; [feature/native-comp] fibn benchmark exhausts memory

Andrea Corallo
Kévin Le Gouguec <[hidden email]> writes:

> Andrea Corallo <[hidden email]> writes:
>
>> this is very good new thanks!  If I read correctly we are always under
>> about 600MB as memory consumption is this correct?
>
> Yep.  I can post the detailed data here if that's needed; here are the
> 10 files that took the most virtual memory (vsz):
>
>  1. org/org.el              541.2 MB
>  2. gnus/gnus-sum.el        459.3 MB
>  3. emacs-lisp/comp.el      455.2 MB
>  4. net/tramp.el            408.2 MB
>  5. net/tramp-adb.el        390.4 MB
>  6. net/tramp-sh.el         371.0 MB
>  7. progmodes/cperl-mode.el 356.2 MB
>  8. ../lisp/window.el       348.9 MB
>  9. gnus/message.el         345.4 MB
> 10. net/soap-client.el      344.4 MB

Cool, I guess means should be buildable at -j3 with 2GB of ram.

> And the 10 files that took the most resident memory (rss):
>
>  1. org/org.el              361.6 MB
>  2. emacs-lisp/comp.el      287.3 MB
>  3. gnus/gnus-sum.el        286.7 MB
>  4. net/tramp.el            236.8 MB
>  5. net/tramp-adb.el        218.1 MB
>  6. net/tramp-sh.el         199.0 MB
>  7. progmodes/cperl-mode.el 185.0 MB
>  8. ../lisp/window.el       181.6 MB
>  9. gnus/message.el         174.0 MB
> 10. erc/erc.el              173.5 MB
>
>> I suspect we are still looking at a speed 3 run.  How comp-speed was set
>> for the test?
>>
>> I ask that because for now speed 3 is written into the stone within
>> elisp-benchmarks.el, yeah... :/ and so has to be manually changed there.
>
> Now that's just sneaky 😛

Well... we can also just call it bad :)

> * Results
> ** master
>
> #+TBLNAME: master
> | test           | non-gc avg (s) | gc avg (s) | gcs avg | tot avg (s) | tot avg err (s) |
>
> |----------------+----------------+------------+---------+-------------+-----------------|
> | bubble-no-cons |         118.41 |       0.23 |       3 |      118.63 |            0.36 |
> | bubble         |          46.29 |      29.34 |     388 |       75.63 |            0.36 |
> | dhrystone      |          74.82 |       0.00 |       0 |       74.82 |            0.34 |
> | fibn-rec       |          59.50 |       0.00 |       0 |       59.50 |            0.28 |
> | fibn-tc        |         147.39 |      96.86 |    1179 |      244.26 |            2.70 |
> | fibn           |         320.33 |     276.23 |    3442 |      596.56 |            5.31 |
> | flet           |         112.02 |       0.00 |       0 |      112.02 |            0.29 |
> | inclist        |         166.30 |       0.00 |       0 |      166.30 |            0.67 |
> | listlen-tc     |          76.97 |       0.00 |       0 |       76.97 |            0.53 |
> | map-closure    |          81.86 |       0.00 |       0 |       81.86 |            0.31 |
> | nbody          |          28.62 |      98.53 |    1303 |      127.14 |            0.21 |
> | pcase          |         110.63 |       0.00 |       0 |      110.63 |            0.29 |
> | pidigits       |          61.78 |      39.46 |     469 |      101.24 |            0.70 |
> |----------------+----------------+------------+---------+-------------+-----------------|
> | total          |        1404.92 |     540.65 |    6786 |     1945.57 |            6.13 |
> ** feature/native-comp
>
> #+TBLNAME: feature/native-comp
> | test           | non-gc avg (s) | gc avg (s) | gcs avg | tot avg (s) | tot avg err (s) |
>
> |----------------+----------------+------------+---------+-------------+-----------------|
> | bubble-no-cons |          30.12 |       0.16 |       2 |       30.28 |            0.04 |
> | bubble         |          18.05 |      27.91 |     352 |       45.96 |            0.33 |
> | dhrystone      |          34.26 |       0.00 |       0 |       34.26 |            0.05 |
> | fibn-rec       |          26.92 |       0.00 |       0 |       26.92 |            0.03 |
> | fibn-tc        |          98.66 |      93.89 |    1096 |      192.55 |            1.66 |
> | fibn           |         253.19 |     269.12 |    3192 |      522.31 |            2.17 |
> | flet           |          22.92 |       0.00 |       0 |       22.92 |            0.04 |
> | inclist        |          11.76 |       0.00 |       0 |       11.76 |            0.16 |
> | listlen-tc     |          21.44 |       0.00 |       0 |       21.44 |            0.05 |
> | map-closure    |          80.57 |       0.00 |       0 |       80.57 |            0.03 |
> | nbody          |          15.62 |      95.66 |    1212 |      111.28 |            0.10 |
> | pcase          |          17.64 |       0.00 |       0 |       17.64 |            0.03 |
> | pidigits       |          59.82 |      38.86 |     436 |       98.68 |            0.39 |
> |----------------+----------------+------------+---------+-------------+-----------------|
> | total          |         690.98 |     525.59 |    6291 |     1216.57 |            2.79 |
> ** ratio
> | test           | non-gc avg (s) | gc avg (s) | gcs avg | tot avg (s) |
> |----------------+----------------+------------+---------+-------------|
> | bubble-no-cons |           3.92 |       1.44 |     1.5 |        3.93 |
> | bubble         |           2.56 |       1.05 |     1.1 |        1.64 |
> | dhrystone      |           2.18 |            |         |        2.18 |
> | fibn-rec       |           2.21 |            |         |        2.21 |
> | fibn-tc        |           1.49 |       1.03 |    1.07 |        1.26 |
> | fibn           |           1.26 |       1.03 |    1.08 |        1.14 |
> | flet           |           4.89 |            |         |        4.89 |
> | inclist        |           14.1 |            |         |        14.1 |
> | listlen-tc     |            3.6 |            |         |         3.6 |
> | map-closure    |           1.02 |            |         |        1.02 |
> | nbody          |           1.83 |       1.03 |    1.07 |        1.14 |
> | pcase          |           6.31 |            |         |        6.31 |
> | pidigits       |           1.03 |       1.02 |    1.08 |        1.02 |
> |----------------+----------------+------------+---------+-------------|
> | total          |           2.03 |       1.03 |    1.08 |         1.6 |
>
> #+TBLFM: @2$2..@>$> = if (remote(feature/native-comp,@@#$$#)!=0, remote(master,@@#$$#)/remote(feature/native-comp,@@#$$#), string("")); p3

That's not bad for speed 2, GC and funcall are eating part of the perf,
probably with bigger function bodies we would see more delta.

>
> Conclusion: I'm closing this report!  Again, thank you for your efforts.
>
> (And thank you for this opportunity to learn about Org spreadsheets 😉)

Thanks for your inputs!

  Andrea

--
[hidden email]