bug#29165: 26.0.90; can't use some code byte-compiled under emacs 24

classic Classic list List threaded Threaded
15 messages Options
Reply | Threaded
Open this post in threaded view
|

bug#29165: 26.0.90; can't use some code byte-compiled under emacs 24

Ken Raeburn-5

Create a file with the contents:

    (require 'cl)
    (defun* foobar (init-fun test-fun cleanup-fun
                             &optional &key
                             (test #'eql)
                             (size 65))
      "Blah blah blah"
      (list 'x init-fun test-fun cleanup-fun test size))


Use Emacs 24.3 to byte-compile the file.

Start the Emacs 26 pretest, and load the .elc file.

Type into the scratch buffer:

    (foobar 1 2 3 :test 4)

And use eval-last-sexp to try to evaluate it.  You should get a list
back.  I get an invalid-function error instead.  With the debugger
enabled, I see:

    Debugger entered--Lisp error: (invalid-function #f(compiled-function (init-fun test-fun cleanup-fun &optional &rest --cl-rest--) "Blah blah blah" #<bytecode 0x501733>))
      foobar(1 2 3 :test 4)
      eval((foobar 1 2 3 :test 4) nil)
      elisp--eval-last-sexp(nil)
      eval-last-sexp(nil)
      funcall-interactively(eval-last-sexp nil)
      call-interactively(eval-last-sexp nil nil)
      command-execute(eval-last-sexp)

It appears that the eval.c code of 26.0.90 doesn't like to see &optional
and &rest together in the argument list of a compiled function.  But in
this case, the byte compiler from version 24 does generate that.  In
version 25, only &rest is put into the list.


In GNU Emacs 26.0.90 (build 1, x86_64-pc-linux-gnu, X toolkit, Xaw3d scroll bars)
 of 2017-10-27 built on just-testing.permabit.com
Windowing system distributor 'The X.Org Foundation', version 11.0.11804000
System Description: Ubuntu 14.04.4 LTS

Recent messages:
For information about GNU Emacs and the GNU system, type C-h C-a.
Loading /tmp/foo.elc...done
Entering debugger...
Back to top level

Configured using:
 'configure --prefix=/u1/ken/install/emacs-26.0.90
 --with-x-toolkit=lucid --enable-checking --without-sound --without-gif'

Configured features:
XAW3D XPM JPEG TIFF PNG RSVG DBUS GSETTINGS NOTIFY ACL LIBSELINUX GNUTLS
LIBXML2 FREETYPE XFT ZLIB TOOLKIT_SCROLL_BARS LUCID X11

Important settings:
  locale-coding-system: nil

Major mode: Lisp Interaction

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message rmc puny seq dired
dired-loaddefs format-spec rfc822 mml mml-sec password-cache epa derived
epg epg-config gnus-util rmail rmail-loaddefs mm-decode mm-bodies
mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader sendmail
rfc2047 rfc2045 ietf-drums mm-util mail-prsvr mail-utils help-mode
easymenu cl-print byte-opt bytecomp byte-compile cconv debug cl gv
cl-loaddefs cl-lib elec-pair time-date mule-util tooltip eldoc electric
uniquify ediff-hook vc-hooks lisp-float-type mwheel term/x-win x-win
term/common-win x-dnd tool-bar dnd fontset image regexp-opt fringe
tabulated-list replace newcomment text-mode elisp-mode lisp-mode
prog-mode register page menu-bar rfn-eshadow isearch timer select
scroll-bar mouse jit-lock font-lock syntax facemenu font-core
term/tty-colors frame cl-generic cham georgian utf-8-lang misc-lang
vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms cp51932
hebrew greek romanian slovak czech european ethiopic indian cyrillic
chinese composite charscript charprop case-table epa-hook jka-cmpr-hook
help simple abbrev obarray minibuffer cl-preloaded nadvice loaddefs
button faces cus-face macroexp files text-properties overlay sha1 md5
base64 format env code-pages mule custom widget hashtable-print-readable
backquote dbusbind inotify dynamic-setting system-font-setting
font-render-setting x-toolkit x multi-tty make-network-process emacs)

Memory information:
((conses 16 98378 10227)
 (symbols 48 20669 1)
 (miscs 40 45 197)
 (strings 32 29669 1021)
 (string-bytes 1 765521)
 (vectors 16 14315)
 (vector-slots 8 495072 6314)
 (floats 8 53 146)
 (intervals 56 314 138)
 (buffers 992 12)
 (heap 1024 32780 1041))



Reply | Threaded
Open this post in threaded view
|

bug#29165: 26.0.90; can't use some code byte-compiled under emacs 24

Noam Postavsky-2
tags 29165 + notabug
quit

Ken Raeburn <[hidden email]> writes:

>     (defun* foobar (init-fun test-fun cleanup-fun
>                              &optional &key

> Use Emacs 24.3 to byte-compile the file.
>
> Start the Emacs 26 pretest, and load the .elc file.

>     Debugger entered--Lisp error: (invalid-function #f(compiled-function (init-fun test-fun cleanup-fun &optional &rest --cl-rest--) "Blah blah blah"

> It appears that the eval.c code of 26.0.90 doesn't like to see &optional
> and &rest together in the argument list of a compiled function.

Yes, see Bug#24912 and Bug#24913.

> But in this case, the byte compiler from version 24 does generate
> that.  In version 25, only &rest is put into the list.

It's not the byte compiler specifically, the defun* macro (aka cl-defun)
expands &key into &rest, so the expansion has &optional &rest.  At any
rate, using &optional is meaningless there, so the solution is just to
remove it.

We should perhaps put something about throwing error on '&option &rest'
into NEWS though.




Reply | Threaded
Open this post in threaded view
|

bug#29165: 26.0.90; can't use some code byte-compiled under emacs 24

Drew Adams
> It's not the byte compiler specifically, the defun* macro (aka cl-defun)
> expands &key into &rest, so the expansion has &optional &rest.  At any
> rate, using &optional is meaningless there, so the solution is just to
> remove it.
>
> We should perhaps put something about throwing error on '&option &rest'
> into NEWS though.

I don't understand.  In Common Lisp it is perfectly correct
to use both &optional and &rest.  If Emacs's emulation of
CL's `defun' is defective in this regard, is that not a bug?

https://www.cs.cmu.edu/Groups/AI/html/cltl/clm/node64.html

Or are you saying that there was a bug in Emacs 24's handling
of the CL `defun' emulation, where it expanded &key into
&rest, producing multiple &optional or &rest, and that now
this has been fixed - hence Emacs 24 byte-code does not work
in Emacs 26?

If you do add something to NEWS, please make clear just what
the problem is/was.  Does Emacs now correctly emulate CL
`defun' wrt &* keywords?  Is it an error to repeat &optional
or &rest?  If not, what is the behavior in that case?

Whatever the explanation is, please make it very clear in
NEWS.  Thx.



Reply | Threaded
Open this post in threaded view
|

bug#29165: 26.0.90; can't use some code byte-compiled under emacs 24

Noam Postavsky-2
On Mon, Nov 6, 2017 at 9:24 AM, Drew Adams <[hidden email]> wrote:

>> We should perhaps put something about throwing error on '&option &rest'
>> into NEWS though.
>
> I don't understand.  In Common Lisp it is perfectly correct
> to use both &optional and &rest.

What's rejected is (&optional &rest other-vars), whereas (&optional
var1 &rest other-vars) is okay. Does CL accept the first form (and if
yes, what does it mean)? I couldn't tell from the page you linked to.



Reply | Threaded
Open this post in threaded view
|

bug#29165: 26.0.90; can't use some code byte-compiled under emacs 24

Drew Adams
> >> We should perhaps put something about throwing error on '&option &rest'
> >> into NEWS though.
> >
> > I don't understand.  In Common Lisp it is perfectly correct
> > to use both &optional and &rest.
>
> What's rejected is (&optional &rest other-vars), whereas (&optional
> var1 &rest other-vars) is okay. Does CL accept the first form (and if
> yes, what does it mean)? I couldn't tell from the page you linked to.

CL accepts a single variable after &rest. And there must be
a variable after &optional.  (&optional foo &rest bar) is OK.

(&optional &rest foo) is not OK.
(&optional foo &rest bar toto titi) is not OK.



Reply | Threaded
Open this post in threaded view
|

bug#29165: 26.0.90; can't use some code byte-compiled under emacs 24

Eli Zaretskii
In reply to this post by Noam Postavsky-2
> From: Noam Postavsky <[hidden email]>
> Date: Mon, 06 Nov 2017 07:44:47 -0500
> Cc: Philipp Stephani <[hidden email]>, [hidden email]
>
> We should perhaps put something about throwing error on '&option &rest'
> into NEWS though.

Please do, and thanks.



Reply | Threaded
Open this post in threaded view
|

bug#29165: 26.0.90; can't use some code byte-compiled under emacs 24

Philipp Stephani
In reply to this post by Drew Adams


Drew Adams <[hidden email]> schrieb am Mo., 6. Nov. 2017 um 15:41 Uhr:
> >> We should perhaps put something about throwing error on '&option &rest'
> >> into NEWS though.
> >
> > I don't understand.  In Common Lisp it is perfectly correct
> > to use both &optional and &rest.
>
> What's rejected is (&optional &rest other-vars), whereas (&optional
> var1 &rest other-vars) is okay. Does CL accept the first form (and if
> yes, what does it mean)? I couldn't tell from the page you linked to.

CL accepts a single variable after &rest. And there must be
a variable after &optional.  (&optional foo &rest bar) is OK.

(&optional &rest foo) is not OK.
(&optional foo &rest bar toto titi) is not OK.

That should match the current behavior in Emacs Lisp now. 
Reply | Threaded
Open this post in threaded view
|

bug#29165: 26.0.90; can't use some code byte-compiled under emacs 24

Ken Raeburn-5
In reply to this post by Drew Adams

On Nov 6, 2017, at 09:40, Drew Adams <[hidden email]> wrote:

>>>> We should perhaps put something about throwing error on '&option &rest'
>>>> into NEWS though.
>>>
>>> I don't understand.  In Common Lisp it is perfectly correct
>>> to use both &optional and &rest.
>>
>> What's rejected is (&optional &rest other-vars), whereas (&optional
>> var1 &rest other-vars) is okay. Does CL accept the first form (and if
>> yes, what does it mean)? I couldn't tell from the page you linked to.
>
> CL accepts a single variable after &rest. And there must be
> a variable after &optional.  (&optional foo &rest bar) is OK.
>
> (&optional &rest foo) is not OK.
> (&optional foo &rest bar toto titi) is not OK.

Is this CL in general or a particular CL implementation? The web page you sent the URL for earlier reads like a specification, and from its use of “*” looks to me like it allows the (admittedly useless) form of &optional with no variables.

Ken


Reply | Threaded
Open this post in threaded view
|

bug#29165: 26.0.90; can't use some code byte-compiled under emacs 24

Andreas Schwab-2
On Nov 06 2017, Ken Raeburn <[hidden email]> wrote:

> On Nov 6, 2017, at 09:40, Drew Adams <[hidden email]> wrote:
>
>>>>> We should perhaps put something about throwing error on '&option &rest'
>>>>> into NEWS though.
>>>>
>>>> I don't understand.  In Common Lisp it is perfectly correct
>>>> to use both &optional and &rest.
>>>
>>> What's rejected is (&optional &rest other-vars), whereas (&optional
>>> var1 &rest other-vars) is okay. Does CL accept the first form (and if
>>> yes, what does it mean)? I couldn't tell from the page you linked to.
>>
>> CL accepts a single variable after &rest. And there must be
>> a variable after &optional.  (&optional foo &rest bar) is OK.
>>
>> (&optional &rest foo) is not OK.
>> (&optional foo &rest bar toto titi) is not OK.
>
> Is this CL in general or a particular CL implementation? The web page you sent the URL for earlier reads like a specification, and from its use of “*” looks to me like it allows the (admittedly useless) form of &optional with no variables.

clisp accepts it.

Andreas.

--
Andreas Schwab, [hidden email]
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."



Reply | Threaded
Open this post in threaded view
|

bug#29165: 26.0.90; can't use some code byte-compiled under emacs 24

Ken Raeburn-5

On Nov 6, 2017, at 13:10, Andreas Schwab <[hidden email]> wrote:

On Nov 06 2017, Ken Raeburn <[hidden email]> wrote:

On Nov 6, 2017, at 09:40, Drew Adams <[hidden email]> wrote:

We should perhaps put something about throwing error on '&option &rest'
into NEWS though.

I don't understand.  In Common Lisp it is perfectly correct
to use both &optional and &rest.

What's rejected is (&optional &rest other-vars), whereas (&optional
var1 &rest other-vars) is okay. Does CL accept the first form (and if
yes, what does it mean)? I couldn't tell from the page you linked to.

CL accepts a single variable after &rest. And there must be
a variable after &optional.  (&optional foo &rest bar) is OK.

(&optional &rest foo) is not OK.
(&optional foo &rest bar toto titi) is not OK.

Is this CL in general or a particular CL implementation? The web page you sent the URL for earlier reads like a specification, and from its use of “*” looks to me like it allows the (admittedly useless) form of &optional with no variables.

clisp accepts it.

It appears that the emacs-26 version of defun* is happy with it (the original Lisp code I posted, using &optional &key) as well, as long as I provide the source, or a byte-compiled file from Emacs 25 or 26; it’s the .elc file generated by the older Emacs that’s causing me a problem. The (new?) checks are incompatible with the the old compiled file, even though the Lisp code itself *appears* to be acceptable still.
Reply | Threaded
Open this post in threaded view
|

bug#29165: 26.0.90; can't use some code byte-compiled under emacs 24

Noam Postavsky-2
On Mon, Nov 6, 2017 at 2:10 PM, Ken Raeburn <[hidden email]> wrote:

> It appears that the emacs-26 version of defun* is happy with it (the
> original Lisp code I posted, using &optional &key) as well, as long as I
> provide the source, or a byte-compiled file from Emacs 25 or 26; it’s the
> .elc file generated by the older Emacs that’s causing me a problem. The
> (new?) checks are incompatible with the the old compiled file, even though
> the Lisp code itself *appears* to be acceptable still.

It looks like the cl-defun in newer Emacs throws away the &optional
for you in this case.



Reply | Threaded
Open this post in threaded view
|

bug#29165: 26.0.90; can't use some code byte-compiled under emacs 24

Drew Adams
In reply to this post by Ken Raeburn-5
> > CL accepts a single variable after &rest. And there must be
> > a variable after &optional.  (&optional foo &rest bar) is OK.
> >
> > (&optional &rest foo) is not OK.
> > (&optional foo &rest bar toto titi) is not OK.
>
> Is this CL in general or a particular CL implementation? The web page you
> sent the URL for earlier reads like a specification, and from its use of
> “*” looks to me like it allows the (admittedly useless) form of &optional
> with no variables.

What I wrote was from memory.  What counts is the definition
of CL.  If it doesn't forbid such constructions then an
implementation or emulation) of CL is free to support them.

If it doesn't define them then an implementation is free
to define them any way it wants.



Reply | Threaded
Open this post in threaded view
|

bug#29165: 26.0.90; can't use some code byte-compiled under emacs 24

Noam Postavsky-2
In reply to this post by Noam Postavsky-2
On Mon, Nov 6, 2017 at 2:16 PM, Noam Postavsky
<[hidden email]> wrote:
> On Mon, Nov 6, 2017 at 2:10 PM, Ken Raeburn <[hidden email]> wrote:
>
>> It appears that the emacs-26 version of defun* is happy with it (the
>> original Lisp code I posted, using &optional &key) as well, as long as I
>> provide the source, or a byte-compiled file from Emacs 25 or 26
>
> It looks like the cl-defun in newer Emacs throws away the &optional
> for you in this case.

I think we should make cl-defun reject this kind of code, to be
consistent with plain defun. See attached.

v1-0001-Mention-new-strictness-for-optional-rest-in-argli.patch (6K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

bug#29165: 26.0.90; can't use some code byte-compiled under emacs 24

Ken Raeburn-5

On Nov 13, 2017, at 13:06, Noam Postavsky <[hidden email]> wrote:

> On Mon, Nov 6, 2017 at 2:16 PM, Noam Postavsky
> <[hidden email]> wrote:
>> On Mon, Nov 6, 2017 at 2:10 PM, Ken Raeburn <[hidden email]> wrote:
>>
>>> It appears that the emacs-26 version of defun* is happy with it (the
>>> original Lisp code I posted, using &optional &key) as well, as long as I
>>> provide the source, or a byte-compiled file from Emacs 25 or 26
>>
>> It looks like the cl-defun in newer Emacs throws away the &optional
>> for you in this case.
>
> I think we should make cl-defun reject this kind of code, to be
> consistent with plain defun. See attached.

I’m of two minds about it… it’s a useless but harmless degenerate case, and based on the link Drew posted and the test Andreas did, making it an error would be a gratuitous incompatibility with CL or at least one implementation.  And maybe it’s not entirely useless if it simplifies someone’s macro so that they can treat zero-or-more optional variables with a single, simple common code path.  But even if we do make it an error, isn’t there usually a stage where it’s just a warning?

(And if we’re going to make that sort of thing an error, we should probably check whether empty &key or &aux variable lists are similarly rejected.  I haven’t looked.)

Also, what the CL macros do going forward is arguably a separate question from whether byte-code processing should reject the byte code generated for the same construct by older releases.  That’s what we’ve got now — Emacs 26.0.90 accepts the forms for CL macros but rejects them in byte code, and Emacs 24 CL macros produced such forms in byte code when given such forms as input.  So currently-acceptable code, or at least code still treated as acceptable under Emacs 25, when compiled by an older release, is no longer accepted. If the source isn’t going to be rejected (e.g., if it’s quietly accepted or only produces a warning), then the byte-code for it probably ought not to be rejected.

Ken


Reply | Threaded
Open this post in threaded view
|

bug#29165: 26.0.90; can't use some code byte-compiled under emacs 24

Noam Postavsky-2
On Mon, Nov 13, 2017 at 2:42 PM, Ken Raeburn <[hidden email]> wrote:

> But even if we do make it an error, isn’t there usually a stage where it’s just a warning?

Maybe. There hasn't been this time (for plain defun, I mean).

> (And if we’re going to make that sort of thing an error, we should probably check whether empty &key or &aux variable lists are similarly rejected.  I haven’t looked.)

I believe empty &key would be tested in my patch, though not &aux.

> If the source isn’t going to be rejected (e.g., if it’s quietly accepted or only produces a warning), then the byte-code for it probably ought not to be rejected.

Yes, that's why my patch rejects the source as well.