[ELPA] New package: elisp-benckmarks

classic Classic list List threaded Threaded
16 messages Options
Reply | Threaded
Open this post in threaded view
|

[ELPA] New package: elisp-benckmarks

Andrea Corallo
Hi all, as suggested I'd like submit a simple ELPA package to benchmark
the elisp implementation.

The idea is that is possible to add benchmarks just deposing the new
test into the 'benchmarks' directory of the package following a name
convention for the entry point function.

Ex: 'foo.el' has to have 'foo-entry' as entry point for the test.

Calling `elisp-benchmarks-run' the whole list of benchmarks is executed
`elb-runs' times.  A summary of the results is then presented.

This is the output on my machine:
"
* Results

  | test           | non-gc avg (s) | gc avg (s) | gcs avg | tot avg (s) | tot avg err (s) |
  |----------------+----------------+------------+---------+-------------+-----------------|
  | bubble-no-cons |          16.46 |       0.05 |       4 |       16.50 |            0.50 |
  | bubble         |           6.03 |       5.85 |     480 |       11.88 |            0.19 |
  | fibn-rec       |           8.21 |       0.00 |       0 |        8.21 |            0.23 |
  | fibn-tc        |           7.34 |       0.00 |       0 |        7.34 |            0.05 |
  | fibn           |          14.00 |       0.00 |       0 |       14.00 |            0.22 |
  | inclist        |          18.22 |       0.02 |       1 |       18.23 |            0.52 |
  | listlen-tc     |           6.96 |       0.00 |       0 |        6.96 |            0.10 |
  | pidigits       |           6.65 |       6.36 |     457 |       13.01 |            0.20 |
  |----------------+----------------+------------+---------+-------------+-----------------|
  | total          |          83.87 |      12.28 |     942 |       96.15 |            0.84 |
"

Regarding the included tests I've mostly taken what I've used to test
gccemacs but I had to strip out nbody and dhrystone because these has
been ported by a colleague of mine.  My understanding is that he wants
to do the paper-works so we can probably add these later.

License-wise I think the only concern would be with pidigits.  The
original is licensed under a "Revised BSD license".

https://salsa.debian.org/benchmarksgame-team/archive-alioth-benchmarksgame/blob/master/contributed-source-code/benchmarksgame/pidigits/LICENSE

For my basic understanding this should be GPL compliant therefore should
be just sufficient to re-license it.  Please correct me if I'm wrong.

Does this make the job?  Feedback are welcome.

Andrea

--
[hidden email]

elb.patch (18K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [ELPA] New package: elisp-benchmarks

Andrea Corallo
Sorry I've misspelled the subject.  Fixed it.

Andrea
--
[hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [ELPA] New package: elisp-benchmarks

Andrea Corallo
Hi, I attach the patch git formatted.  I've also fixed two nits.

Bests

Andrea

--
[hidden email]

0001-elisp-benckmarks-New-package-initial-add.patch (19K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [ELPA] New package: elisp-benchmarks

Eric Abrahamsen-2
Andrea Corallo <[hidden email]> writes:

> Hi, I attach the patch git formatted.  I've also fixed two nits.

Thanks for this package, and I'm looking forward to trying it out! I
have one question at this point, which is that it seems somewhat
limiting to require benchmarks to be placed into this package's own
benchmarks/ directory. It seems like it would be nice to let them live
anywhere, even if that means selecting them manually, rather than
forcing a single location, as the package currently seems to do (unless
I'm missing something?). For example, it might be nice to ship other
packages with their own benchmark libraries, which would then be
selectable somehow. What do you think?

Thanks!
Eric


Reply | Threaded
Open this post in threaded view
|

Re: [ELPA] New package: elisp-benchmarks

Andrea Corallo
Eric Abrahamsen <[hidden email]> writes:

> Andrea Corallo <[hidden email]> writes:
>
>> Hi, I attach the patch git formatted.  I've also fixed two nits.
>
> Thanks for this package, and I'm looking forward to trying it out! I
> have one question at this point, which is that it seems somewhat
> limiting to require benchmarks to be placed into this package's own
> benchmarks/ directory. It seems like it would be nice to let them live
> anywhere, even if that means selecting them manually, rather than
> forcing a single location, as the package currently seems to do (unless
> I'm missing something?). For example, it might be nice to ship other
> packages with their own benchmark libraries, which would then be
> selectable somehow. What do you think?
>
> Thanks!
> Eric

Hi Eric,
that's a very good point thanks for commenting.

Actually with the current setup you could depose a benchmark in the
benchmarks folder that calls say some gnus function and define a test in
that way.  Everything would work except the 'recompile'.

I also like the idea of doing the other way around (defining benchmarks
and have them shipped with the package itself).  This would be
potentially more elegant but I guess we should then have the
infrastructure to do that in Emacs and not just in ELPA.  Is this
correct?

Maybe you have in mind a different mechanism for this?

Bests

Andrea

--
[hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [ELPA] New package: elisp-benchmarks

Eric Abrahamsen-2
Andrea Corallo <[hidden email]> writes:

> Eric Abrahamsen <[hidden email]> writes:
>
>> Andrea Corallo <[hidden email]> writes:
>>
>>> Hi, I attach the patch git formatted.  I've also fixed two nits.
>>
>> Thanks for this package, and I'm looking forward to trying it out! I
>> have one question at this point, which is that it seems somewhat
>> limiting to require benchmarks to be placed into this package's own
>> benchmarks/ directory. It seems like it would be nice to let them live
>> anywhere, even if that means selecting them manually, rather than
>> forcing a single location, as the package currently seems to do (unless
>> I'm missing something?). For example, it might be nice to ship other
>> packages with their own benchmark libraries, which would then be
>> selectable somehow. What do you think?
>>
>> Thanks!
>> Eric
>
> Hi Eric,
> that's a very good point thanks for commenting.
>
> Actually with the current setup you could depose a benchmark in the
> benchmarks folder that calls say some gnus function and define a test in
> that way.  Everything would work except the 'recompile'.

Is the expectation that we'd actually be committing benchmarks to the
benchmarks package? Or just dropping files into the installation
directory? Both approaches seem a bit awkward -- for the former, you'd
have to update the benchmark package whenever any other package updated
its benchmarks. For the latter, when we update the benchmarks package,
we'll lose any files we stuck in there manually.

> I also like the idea of doing the other way around (defining benchmarks
> and have them shipped with the package itself).  This would be
> potentially more elegant but I guess we should then have the
> infrastructure to do that in Emacs and not just in ELPA.  Is this
> correct?
>
> Maybe you have in mind a different mechanism for this?

I suppose the fanciest approach would be to do what ert does: provide a
`elb-defbenchmark' macro that registers the benchmark in some central
location, and allows the user to select which of the registered
benchmarks to run. That might take more work than you'd had in mind,
though. A simpler approach might be just to make the "working directory"
argument of the interactive commands selectable with a prefix arg. Right
now the prefix arg controls the selector -- maybe that could be a
two-part selection? First directory, then string? Or something like
that.

At the very least, `elb-bench-directory' could be let-bound in a custom
function. But making it a defconst really makes it feel like we
shouldn't be doing that.

Hope that's useful,
Eric

Reply | Threaded
Open this post in threaded view
|

Re: [ELPA] New package: elisp-benchmarks

Stefan Monnier
> Is the expectation that we'd actually be committing benchmarks to the
> benchmarks package?

I think so, yes.

> Both approaches seem a bit awkward -- for the former, you'd
> have to update the benchmark package whenever any other package updated
> its benchmarks.

Not sure what you mean by "other package".  AFAIK this is meant as
a benchmark of Elisp itself, so it shouldn't depend on anything
else than Emacs and shouldn't be affected by changes in third
party packages.


        Stefan


Reply | Threaded
Open this post in threaded view
|

Re: [ELPA] New package: elisp-benchmarks

Eric Abrahamsen-2
Stefan Monnier <[hidden email]> writes:

>> Is the expectation that we'd actually be committing benchmarks to the
>> benchmarks package?
>
> I think so, yes.
>
>> Both approaches seem a bit awkward -- for the former, you'd
>> have to update the benchmark package whenever any other package updated
>> its benchmarks.
>
> Not sure what you mean by "other package".  AFAIK this is meant as
> a benchmark of Elisp itself, so it shouldn't depend on anything
> else than Emacs and shouldn't be affected by changes in third
> party packages.

Oh, I thought this was a general facility that other ELPA packages were
supposed to be able to use, as well. If it's only for Elisp, maybe it
could just go into core?

Reply | Threaded
Open this post in threaded view
|

Re: [ELPA] New package: elisp-benchmarks

Stefan Monnier
> Oh, I thought this was a general facility that other ELPA packages were
> supposed to be able to use, as well. If it's only for Elisp, maybe it
> could just go into core?

It's meant to be a repository of benchmarks to compare Elisp performance
across different versions of Emacs, so I think it makes sense to keep it
outside of Emacs itself.


        Stefan


Reply | Threaded
Open this post in threaded view
|

Re: [ELPA] New package: elisp-benchmarks

Eric Abrahamsen-2
Stefan Monnier <[hidden email]> writes:

>> Oh, I thought this was a general facility that other ELPA packages were
>> supposed to be able to use, as well. If it's only for Elisp, maybe it
>> could just go into core?
>
> It's meant to be a repository of benchmarks to compare Elisp performance
> across different versions of Emacs, so I think it makes sense to keep it
> outside of Emacs itself.

Gotcha, that makes more sense.

Thanks,
Eric

Reply | Threaded
Open this post in threaded view
|

Re: [ELPA] New package: elisp-benchmarks

Andrea Corallo
In reply to this post by Stefan Monnier
Stefan Monnier <[hidden email]> writes:
> Not sure what you mean by "other package".  AFAIK this is meant as
> a benchmark of Elisp itself, so it shouldn't depend on anything
> else than Emacs and shouldn't be affected by changes in third
> party packages.

If understand correctly that means benchmarks can probe every piece of
Emacs core and not just 'basic' language features.  To get practical: a
test that calls the gnus infrastructure would be then acceptable right?

I'm looking forward to have more benchmarks there to look at.

Andrea

--
[hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [ELPA] New package: elisp-benchmarks

Eric Abrahamsen-2
Andrea Corallo <[hidden email]> writes:

> Stefan Monnier <[hidden email]> writes:
>> Not sure what you mean by "other package".  AFAIK this is meant as
>> a benchmark of Elisp itself, so it shouldn't depend on anything
>> else than Emacs and shouldn't be affected by changes in third
>> party packages.
>
> If understand correctly that means benchmarks can probe every piece of
> Emacs core and not just 'basic' language features.  To get practical: a
> test that calls the gnus infrastructure would be then acceptable right?

Right, I was also trying to imagine what this would look like in
practical terms. Say I want to put a change into Gnus that I hope will
result in speedups. I make a git branch, make some commits. I probably
set up a git worktree for the branch. I create an init file for that
worktree emacs (so that I can run "emacs -Q -l init.el") that just adds
the elisp-benchmarks package to the load-path. Then I start my worktree
emacs, and run the benchmarks.

I'm just thinking this through -- the above workflow is fine as far as
I'm concerned.

Eric


Reply | Threaded
Open this post in threaded view
|

Re: [ELPA] New package: elisp-benchmarks

Andrea Corallo
In reply to this post by Andrea Corallo
Andrea Corallo <[hidden email]> writes:

> Hi, I attach the patch git formatted.  I've also fixed two nits.
>
> Bests
>
> Andrea

Just a reminder that given that I don't have write access somebody will
have to push it when the review process is done (not sure where we
stand about that).

Bests

Andrea

--
[hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [ELPA] New package: elisp-benchmarks

Andrea Corallo
Andrea Corallo <[hidden email]> writes:

> Andrea Corallo <[hidden email]> writes:
>
>> Hi, I attach the patch git formatted.  I've also fixed two nits.
>>
>> Bests
>>
>> Andrea
>
> Just a reminder that given that I don't have write access somebody will
> have to push it when the review process is done (not sure where we
> stand about that).
>
> Bests
>
> Andrea
>

Hi all, in case there's no suggestions to be implemented could somebody
install the patch?

Bests
Andrea

--
[hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [ELPA] New package: elisp-benchmarks

Stefan Monnier
> Hi all, in case there's no suggestions to be implemented could somebody
> install the patch?

Installed (as an :external, tho).


        Stefan


Reply | Threaded
Open this post in threaded view
|

Re: [ELPA] New package: elisp-benchmarks

Andrea Corallo
Stefan Monnier <[hidden email]> writes:

>> Hi all, in case there's no suggestions to be implemented could somebody
>> install the patch?
>
> Installed (as an :external, tho).
>
>
>         Stefan

Cool thanks!

Andrea

--
[hidden email]