bug#31118: 27.0.50; Can't load/compile websocket in 32bit master

classic Classic list List threaded Threaded
21 messages Options
12
Reply | Threaded
Open this post in threaded view
|

bug#31118: 27.0.50; Can't load/compile websocket in 32bit master

Stefan Monnier
Package: Emacs
Version: 27.0.50


The websocket package available from GNU ELPA can't be used in a 32bit
build of master any more because of the #xffffffff constant in its code:

    (defun websocket-to-bytes (val nbytes)
      "Encode the integer VAL in NBYTES of data.
    NBYTES much be a power of 2, up to 8.
   
    This supports encoding values up to 536870911 bytes (2^29 - 1),
    approximately 537M long."
      (when (and (< nbytes 8)
                 (> val (expt 2 (* 8 nbytes))))
        ;; not a user-facing error, this must be caused from an error in
        ;; this library
        (error "websocket-to-bytes: Value %d could not be expressed in %d bytes"
               val nbytes))
      (if (= nbytes 8)
          (progn
            (let ((hi-32bits (lsh val -32))
                  ;; Test for systems that don't have > 32 bits, and
                  ;; for those systems just return the value.
                  (low-32bits (if (= 0 (expt 2 32))
                                  val
                                (logand #xffffffff val))))
              (when (or (> hi-32bits 0) (> (lsh low-32bits -29) 0))
                (signal 'websocket-frame-too-large val))
              (bindat-pack `((:val vec 2 u32))
                           `((:val . [,hi-32bits ,low-32bits])))))
        (bindat-pack
         `((:val ,(cond ((= nbytes 1) 'u8)
                        ((= nbytes 2) 'u16)
                        ((= nbytes 4) 'u32)
                        ;; Library error, not system error
                        (t (error "websocket-to-bytes: Unknown NBYTES: %S" nbytes)))))
         `((:val . ,val)))))

The code used to compile&run correctly before both on 32bit and 64bit
systems (but admittedly, it failed to run correctly on 64bit systems
after having been compiled on a 32bit system).


        Stefan



Reply | Threaded
Open this post in threaded view
|

bug#31118: 27.0.50; Can't load/compile websocket in 32bit master

Paul Eggert
I fixed that by installing the attached into ELPA.

I can't build ELPA from scratch on master now, for other reasons. The build
fails as follows:

In toplevel form:
packages/counsel-ebdb/counsel-ebdb.el:29:1:Error: Cannot open load file: No such
file or directory, ebdb
make: *** [GNUmakefile:171: packages/counsel-ebdb/counsel-ebdb.elc] Error 1

Is anybody building ELPA regularly? It looks like not.

0001-Port-websocket-to-bleeding-edge-32-bit-Emacs.patch (1K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

bug#31118: 27.0.50; Can't load/compile websocket in 32bit master

Stefan Monnier
> I fixed that by installing the attached into ELPA.
>
> I can't build ELPA from scratch on master now, for other reasons. The build
> fails as follows:
>
> In toplevel form:
> packages/counsel-ebdb/counsel-ebdb.el:29:1:Error: Cannot open load file: No
> such file or directory, ebdb
> make: *** [GNUmakefile:171: packages/counsel-ebdb/counsel-ebdb.elc] Error 1
>
> Is anybody building ELPA regularly?

Yes, I do.  Not sure why you can't find ebdb since it's in elpa.git.
Have you added the .../elpa/packages to your package-directory-list?

But I can tell you that the build fails a bit later because

    grep '#xffffffff' elpa/**/*.el

still finds other cases.


        Stefan



Reply | Threaded
Open this post in threaded view
|

bug#31118: 27.0.50; Can't load/compile websocket in 32bit master

Paul Eggert
Stefan Monnier wrote:

>> I fixed that by installing the attached into ELPA.
>>
>> I can't build ELPA from scratch on master now, for other reasons. The build
>> fails as follows:
>>
>> In toplevel form:
>> packages/counsel-ebdb/counsel-ebdb.el:29:1:Error: Cannot open load file: No
>> such file or directory, ebdb
>> make: *** [GNUmakefile:171: packages/counsel-ebdb/counsel-ebdb.elc] Error 1
>>
>> Is anybody building ELPA regularly?
>
> Yes, I do.  Not sure why you can't find ebdb since it's in elpa.git.
> Have you added the .../elpa/packages to your package-directory-list?

No, I tried to follow the instructions in README
<http://git.savannah.gnu.org/cgit/emacs/elpa.git/tree/README>. They say that to
install all the packages "in place" (which I assume is the simplest way to build
ELPA) one should run "make externals" first. This fails from a fresh elpa
checkout as follows:

$ make externals
emacs --batch -l admin/archive-contents.el \
     -f archive-add/remove/update-externals
No "emacs" subdir: will skip :core packages
Cloning branch auctex:
fatal: Not a valid object name: 'origin/externals/auctex'.
...
Cloning branch ebdb:
fatal: Not a valid object name: 'origin/externals/ebdb'.
...

and no doubt this explains the problem. I charged ahead and typed plain 'make'
which failed as I mentioned earlier. What am I doing wrong? Should the README be
updated? (As you can probably tell I've never used ELPA, and I must say that the
startup overhead is offputting.)


> But I can tell you that the build fails a bit later because
>
>      grep '#xffffffff' elpa/**/*.el
>
> still finds other cases.
That command doesn't work for me:

$ grep '#xffffffff' elpa/**/*.el
grep: elpa/**/*.el: No such file or directory

And this one doesn't find anything for me, in the elpa directory:

grep -r '#xffffffff' .

Are we talking about the same repository? I'm using the Savannah ELPA repository
under the emacs tree, as in the abovementioned URL.



Reply | Threaded
Open this post in threaded view
|

bug#31118: 27.0.50; Can't load/compile websocket in 32bit master

Andreas Schwab
On Apr 11 2018, Paul Eggert <[hidden email]> wrote:

> $ make externals
> emacs --batch -l admin/archive-contents.el \
>     -f archive-add/remove/update-externals
> No "emacs" subdir: will skip :core packages
> Cloning branch auctex:
> fatal: Not a valid object name: 'origin/externals/auctex'.
> ...
> Cloning branch ebdb:
> fatal: Not a valid object name: 'origin/externals/ebdb'.
> ...

How did you clone the repository?

Andreas.

--
Andreas Schwab, SUSE Labs, [hidden email]
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."



Reply | Threaded
Open this post in threaded view
|

bug#31118: 27.0.50; Can't load/compile websocket in 32bit master

Paul Eggert
Andreas Schwab wrote:
> How did you clone the repository?

I followed my usual practice:

git clone [hidden email]:/srv/git/emacs/elpa.git elpa-savannah
git clone elpa-savannah elpa

I do this so that I have a relatively-pristine repository (elpa-savannah) and a
dirtier working one (elpa). Hmm, I now see that this practice does not work with
ELPA for some reason. Too bad. I guess I'll not bother with the
relatively-pristine repository for ELPA. Thanks for the help.



Reply | Threaded
Open this post in threaded view
|

bug#31118: 27.0.50; Can't load/compile websocket in 32bit master

Stefan Monnier
In reply to this post by Paul Eggert
> $ make externals
> emacs --batch -l admin/archive-contents.el \
>     -f archive-add/remove/update-externals
> No "emacs" subdir: will skip :core packages
> Cloning branch auctex:
> fatal: Not a valid object name: 'origin/externals/auctex'.
> ...
> Cloning branch ebdb:
> fatal: Not a valid object name: 'origin/externals/ebdb'.
> ...

Indeed, "make externals" assumes that the other branches are in your
repository, in the remote called "origin".  For some reason your
local repository doesn't seem to be setup this way.

> That command doesn't work for me:
>
> $ grep '#xffffffff' elpa/**/*.el
> grep: elpa/**/*.el: No such file or directory

The "**/" glob is a zsh extension (supported also by Eshell, AFAIK),
whose meaning is clear enough.

> And this one doesn't find anything for me, in the elpa directory:
> grep -r '#xffffffff' .

This is because of the above error (the other matches are in exwm,
i.e. in the origin/externals/exwm branch).


        Stefan



Reply | Threaded
Open this post in threaded view
|

bug#31118: 27.0.50; Can't load/compile websocket in 32bit master

Basil L. Contovounesios
Stefan Monnier <[hidden email]> writes:

>> That command doesn't work for me:
>>
>> $ grep '#xffffffff' elpa/**/*.el
>> grep: elpa/**/*.el: No such file or directory
>
> The "**/" glob is a zsh extension (supported also by Eshell, AFAIK),
> whose meaning is clear enough.

FWIW, Bash 4.0 can also interpret "**" recursively via the globstar
shell option; see https://stackoverflow.com/a/28199633/3084001.

--
Basil



Reply | Threaded
Open this post in threaded view
|

bug#31118: 27.0.50; Can't load/compile websocket in 32bit master

Paul Eggert
In reply to this post by Stefan Monnier
On 04/11/2018 05:19 AM, Stefan Monnier wrote:
> "make externals" assumes that the other branches are in your
> repository, in the remote called "origin".  For some reason your
> local repository doesn't seem to be setup this way.

Yes, Andreas put me on the right track. I installed the attached to help
others who might have a similar problem.

However, even when I clone directly from Savannah so that 'make
externals' works, the resulting 'make' still fails from a fresh
checkout. Here's the output of redoing the 'make'. (I can resend the log
of the full 'make -j5' if you like, but it's 11,000 lines long....) Are
the people doing periodic builds starting from a clean sheet, or is
there some other mysterious thing that I am doing "wrong"?

$ make
Byte compiling packages/hyperbole/hinit.el
Unable to activate package ‘helm-ebdb’.
Required package ‘helm-1.0’ is unavailable
Unable to activate package ‘ebdb-i18n-chn’.
Required package ‘pyim-1.6.0’ is unavailable
Loading /home/eggert/src/gnu/elpa/packages/hyperbole/hui-window.el
(source)...

In toplevel form:
packages/hyperbole/hinit.el:26:1:Error: "hui-window.el": ‘hmouse-alist’
must be defined before loading this.
make: *** [GNUmakefile:171: packages/hyperbole/hinit.elc] Error 1

Compilation exited abnormally with code 2 at Wed Apr 11 08:56:13


0001-README-Don-t-clone-a-clone.patch (792 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

bug#31118: 27.0.50; Can't load/compile websocket in 32bit master

Eli Zaretskii
In reply to this post by Stefan Monnier
> From: Stefan Monnier <[hidden email]>
> Date: Wed, 11 Apr 2018 08:19:36 -0400
> Cc: [hidden email]
>
> > $ make externals
> > emacs --batch -l admin/archive-contents.el \
> >     -f archive-add/remove/update-externals
> > No "emacs" subdir: will skip :core packages
> > Cloning branch auctex:

Btw, this part of archive-contents.el:

    (setq archive--use-worktree
          (list
           (ignore-errors
             (zerop (call-process "git" nil nil nil "worktree" "--help"))))))

is sub-optimal: on MS-Windows, Git help commands invoke the system Web
browser to show the HTML version of the docs, so the browser suddenly
pops up when this code runs.

I actually don't understand why --help is the best idea here; is there
something wrong with running "worktree list" instead?  I tested that
with Git v2.3.5, which doesn't support worktrees, and it exited with
non-zero code.



Reply | Threaded
Open this post in threaded view
|

bug#31118: 27.0.50; Can't load/compile websocket in 32bit master

Stefan Monnier
In reply to this post by Paul Eggert
> if you like, but it's 11,000 lines long....) Are the people doing periodic
> builds starting from a clean sheet, or is there some other mysterious thing
> that I am doing "wrong"?

I don't re-compile everything all the time, no.  It would be good to
have such a thing in a CI somewhere, ideally with some way to
automatically send the build failure notices to the right package maintainer.


        Stefan



Reply | Threaded
Open this post in threaded view
|

bug#31118: 27.0.50; Can't load/compile websocket in 32bit master

Stefan Monnier
In reply to this post by Eli Zaretskii
> I actually don't understand why --help is the best idea here; is there
> something wrong with running "worktree list" instead?

I use "--help" because I tried it and it worked for me.
Feel free to replace it with "list".


        Stefan



Reply | Threaded
Open this post in threaded view
|

bug#31118: 27.0.50; Can't load/compile websocket in 32bit master

Glenn Morris-3
In reply to this post by Stefan Monnier
Stefan Monnier wrote:

> I don't re-compile everything all the time, no.  It would be good to
> have such a thing in a CI somewhere, ideally with some way to
> automatically send the build failure notices to the right package maintainer.

Such a setup was posted once:
http://lists.gnu.org/archive/html/emacs-devel/2013-07/msg00305.html



Reply | Threaded
Open this post in threaded view
|

bug#31118: 27.0.50; Can't load/compile websocket in 32bit master

Eli Zaretskii
In reply to this post by Stefan Monnier
> From: Stefan Monnier <[hidden email]>
> Cc: [hidden email], [hidden email]
> Date: Wed, 11 Apr 2018 14:38:45 -0400
>
> > I actually don't understand why --help is the best idea here; is there
> > something wrong with running "worktree list" instead?
>
> I use "--help" because I tried it and it worked for me.
> Feel free to replace it with "list".

Done.



Reply | Threaded
Open this post in threaded view
|

bug#31118: 27.0.50; Can't load/compile websocket in 32bit master

Paul Eggert
In reply to this post by Stefan Monnier
Although I still can't build ELPA (see Bug#31118#31), I took a shot at
making ELPA build for 32-bit Emacs on master by installing the attached
patches into the xelb and exwm packages, respectively.

As I understand it, some parts of ELPA still assume 64-bit Emacs, but
that's lower priority as it's not a regression. Also, one still cannot
reliably build ELPA on 64-bit Emacs and run it on 32-bit Emacs (or vice
versa), but people don't do that much and I suspect it wasn't really
working either.

0001-Port-to-32-bit-Emacs-master.patch (8K) Download Attachment
0001-Port-to-32-bit-Emacs-on-master-branch.patch (1K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

bug#31118: 27.0.50; Can't load/compile websocket in 32bit master

Stefan Monnier
> Although I still can't build ELPA (see Bug#31118#31), I took a shot at
> making ELPA build for 32-bit Emacs on master by installing the attached
> patches into the xelb and exwm packages, respectively.

I think a major problem remains: these problems in GNU ELPA packages
probably also affect other packages that we can't patch so easily.

So I think we should reconsider whether reading something like
#xffffffff in a 32bit build should signal an error.

More specifically, I think we should first make it emit a warning, while
keeping the old broken behavior.  Then in a later release we can make it
signal an error.


        Stefan



Reply | Threaded
Open this post in threaded view
|

bug#31118: 27.0.50; Can't load/compile websocket in 32bit master

Paul Eggert
On 04/15/2018 06:26 PM, Stefan Monnier wrote:
> I think we should first make it emit a warning, while
> keeping the old broken behavior.

Do you mean 'read' should call 'display-warning' for now? I suppose we could
make read-integer-overflow-as-float be a three-state variable: either (1) signal
an error, or (2) call display-warning and yield a float, or (3) silently yield a
float, and have (2) be the default for now. However, I worry that (2) might lead
to further problems, e.g., display-warning is Lisp code that might in turn call
'read' and loop recursively.

The code we're talking about had nonportable code like (eq desktop #xffffffff)
that doesn't work on platforms with 30-bit fixnums anyway. If the goal is to
make Elisp code safer on 32-bit platforms, --with-wide-int is a simpler and more
reliable way to get there (of course bignums would be better but that's a much
bigger project). Most Emacs development occurs on 64-bit platforms now, and it's
becoming more and more of a pain to insist that programmers must hack on code to
make it portable to platforms with 30-bit fixnums.

Admittedly --with-wide-hit is a ~30% CPU performance hit on my circa-2010 AMD
desktop. If defaulting to --with-wide-int is too drastic, I hope that the
already-existing read-integer-overflow-as-float flag is enough backstop for
people who want to run nonportable code on platforms with 30-bit fixnums.





Reply | Threaded
Open this post in threaded view
|

bug#31118: 27.0.50; Can't load/compile websocket in 32bit master

Stefan Monnier
> Do you mean 'read' should call 'display-warning' for now? I suppose we could
> make read-integer-overflow-as-float be a three-state variable: either (1)
> signal an error, or (2) call display-warning and yield a float, or (3)
> silently yield a float, and have (2) be the default for now. However,
> I worry that (2) might lead to further problems, e.g., display-warning is
> Lisp code that might in turn call 'read' and loop recursively.

We already solved this problem for old-style backquotes: don't emit the
warning directly, instead set a variable and let the caller check the
value of the variable to emit a warning if needed.

> The code we're talking about had nonportable code like (eq desktop
> #xffffffff) that doesn't work on platforms with 30-bit fixnums anyway.

I don't disagree with your change.  But the problems encountered with
code in GNU ELPA makes me think that we want to do it in two steps to
reduce the backward compatibility pain.

> Admittedly --with-wide-hit is a ~30% CPU performance hit on my circa-2010
> AMD desktop.

4 out of 5 of my machines predate your AMD desktop, and I use a 32bit
build on them.

> If defaulting to --with-wide-int is too drastic, I hope that the
> already-existing read-integer-overflow-as-float flag is enough
> backstop for people who want to run nonportable code on platforms with
> 30-bit fixnums.

If we want to signal an error by default, then I think it's important
for this error to give clear steps to set read-integer-overflow-as-float
(and in a message that's not too prone to disappear before the user
gets to see and act on it).


        Stefan



Reply | Threaded
Open this post in threaded view
|

bug#31118: 27.0.50; Can't load/compile websocket in 32bit master

Paul Eggert
On 04/17/2018 05:18 AM, Stefan Monnier wrote:
> If we want to signal an error by default, then I think it's important
> for this error to give clear steps to set read-integer-overflow-as-float

Something like the attached patch, perhaps?


0001-Tell-user-about-read-integer-overflow-as-float.patch (1K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

bug#31118: 27.0.50; Can't load/compile websocket in 32bit master

Stefan Monnier
> Something like the attached patch, perhaps?

That might be good enough.


        Stefan


> From eb2fa1a00c5c85aced40900b59183dc03d822958 Mon Sep 17 00:00:00 2001
> From: Paul Eggert <[hidden email]>
> Date: Tue, 17 Apr 2018 16:52:04 -0700
> Subject: [PATCH] Tell user about read-integer-overflow-as-float
>
> * src/lread.c (string_to_number): Suggest
> read-integer-overflow-as-float in signal message.
> Suggested by Stefan Monnier (Bug#31118#58).
> ---
>  src/lread.c | 7 ++++++-
>  1 file changed, 6 insertions(+), 1 deletion(-)
>
> diff --git a/src/lread.c b/src/lread.c
> index 6eda740540..16e34aff0c 100644
> --- a/src/lread.c
> +++ b/src/lread.c
> @@ -3797,7 +3797,12 @@ string_to_number (char const *string, int base, int flags)
>   value = n;
>  
>        if (! (state & DOT_CHAR) && ! (flags & S2N_OVERFLOW_TO_FLOAT))
> - xsignal1 (Qoverflow_error, build_string (string));
> + {
> +  AUTO_STRING (fmt, ("%s is out of fixnum range; "
> +     "maybe set `read-integer-overflow-as-float'?"));
> +  xsignal1 (Qoverflow_error,
> +    CALLN (Fformat_message, fmt, build_string (string)));
> + }
>      }
>  
>    /* Either the number uses float syntax, or it does not fit into a fixnum.



12