bug#46256: [feature/native-comp] AOT eln files ignored if run from build tree

classic Classic list List threaded Threaded
68 messages Options
1234
Reply | Threaded
Open this post in threaded view
|

bug#46256: [feature/native-comp] AOT eln files ignored if run from build tree

Andy Moreton-3
Hi,

I have built emacs native-comp branch for 64bit Mingw64 with
NATIVE_FULL_AOT=1 (out of tree, so build dir != source dir).

I notice that if I run the built emacs from the build dir then the
prebuilt .eln files are ignored, and async compilation of the .eln file
happens again to add them to the user eln-cache dir.

The prebuilt .eln files are not found in the user eln-cache (expected)
or the installed emacs directory (also expected), but it looks like it
does not also check the build dir (relative to the running emacs rather
than relative to the install prefix).

Running from the build dir without installing is common for developers
building from source, so it would be useful to keep this working with
native AOT builds.

     AndyM




Reply | Threaded
Open this post in threaded view
|

bug#46256: [feature/native-comp] AOT eln files ignored if run from build tree

Emacs - Bugs mailing list
Andy Moreton <[hidden email]> writes:

> Hi,
>
> I have built emacs native-comp branch for 64bit Mingw64 with
> NATIVE_FULL_AOT=1 (out of tree, so build dir != source dir).
>
> I notice that if I run the built emacs from the build dir then the
> prebuilt .eln files are ignored, and async compilation of the .eln file
> happens again to add them to the user eln-cache dir.
>
> The prebuilt .eln files are not found in the user eln-cache (expected)
> or the installed emacs directory (also expected), but it looks like it
> does not also check the build dir (relative to the running emacs rather
> than relative to the install prefix).
>
> Running from the build dir without installing is common for developers
> building from source, so it would be useful to keep this working with
> native AOT builds.
>
>     AndyM

Hi Andy,

could you share the values of PATH_DUMPLOADSEARCH and
PATH_REL_LOADSEARCH from your epaths.h ?

Thanks

  Andrea



Reply | Threaded
Open this post in threaded view
|

bug#46256: [feature/native-comp] AOT eln files ignored if run from build tree

Andy Moreton-3
In reply to this post by Andy Moreton-3
On Wed 03 Feb 2021, akrl--- via "Bug reports for GNU Emacs, the Swiss army knife of text editors" wrote:

> Andy Moreton <[hidden email]> writes:
>
>> Hi,
>>
>> I have built emacs native-comp branch for 64bit Mingw64 with
>> NATIVE_FULL_AOT=1 (out of tree, so build dir != source dir).
>>
>> I notice that if I run the built emacs from the build dir then the
>> prebuilt .eln files are ignored, and async compilation of the .eln file
>> happens again to add them to the user eln-cache dir.
>>
>> The prebuilt .eln files are not found in the user eln-cache (expected)
>> or the installed emacs directory (also expected), but it looks like it
>> does not also check the build dir (relative to the running emacs rather
>> than relative to the install prefix).
>>
>> Running from the build dir without installing is common for developers
>> building from source, so it would be useful to keep this working with
>> native AOT builds.
>>
>>     AndyM
>
> Hi Andy,
>
> could you share the values of PATH_DUMPLOADSEARCH and
> PATH_REL_LOADSEARCH from your epaths.h ?
>
> Thanks
>
>   Andrea

Native branch checkout is in: "c:/emacs/git/emacs/native/"

"c:/emacs/git/emacs/native/build/mingw64-x86_64-O2/src/epaths.h" contains:

#define PATH_DUMPLOADSEARCH "C:/emacs/git/emacs/native/lisp"
#define PATH_REL_LOADSEARCH "28.0.50/lisp"


HTH,

    AndyM




Reply | Threaded
Open this post in threaded view
|

bug#46256: [feature/native-comp] AOT eln files ignored if run from build tree

Emacs - Bugs mailing list
Andy Moreton <[hidden email]> writes:

> On Wed 03 Feb 2021, akrl--- via "Bug reports for GNU Emacs, the Swiss army knife of text editors" wrote:
>
>> Andy Moreton <[hidden email]> writes:
>>
>>> Hi,
>>>
>>> I have built emacs native-comp branch for 64bit Mingw64 with
>>> NATIVE_FULL_AOT=1 (out of tree, so build dir != source dir).
>>>
>>> I notice that if I run the built emacs from the build dir then the
>>> prebuilt .eln files are ignored, and async compilation of the .eln file
>>> happens again to add them to the user eln-cache dir.
>>>
>>> The prebuilt .eln files are not found in the user eln-cache (expected)
>>> or the installed emacs directory (also expected), but it looks like it
>>> does not also check the build dir (relative to the running emacs rather
>>> than relative to the install prefix).
>>>
>>> Running from the build dir without installing is common for developers
>>> building from source, so it would be useful to keep this working with
>>> native AOT builds.
>>>
>>>     AndyM
>>
>> Hi Andy,
>>
>> could you share the values of PATH_DUMPLOADSEARCH and
>> PATH_REL_LOADSEARCH from your epaths.h ?
>>
>> Thanks
>>
>>   Andrea
>
> Native branch checkout is in: "c:/emacs/git/emacs/native/"
>
> "c:/emacs/git/emacs/native/build/mingw64-x86_64-O2/src/epaths.h" contains:
>
> #define PATH_DUMPLOADSEARCH "C:/emacs/git/emacs/native/lisp"
> #define PATH_REL_LOADSEARCH "28.0.50/lisp"
>
>
> HTH,
>
>     AndyM
Hi Andy could you give it a go to the following blind patch?

Thanks

  Andrea


diff --git a/src/comp.c b/src/comp.c
index 289d89d37d..980462b520 100644
--- a/src/comp.c
+++ b/src/comp.c
@@ -433,6 +433,12 @@ #define TEXT_DATA_RELOC_EPHEMERAL_SYM "text_data_reloc_eph"
 #define TEXT_OPTIM_QLY_SYM "text_optim_qly"
 #define TEXT_FDOC_SYM "text_data_fdoc"
 
+#ifdef WINDOWSNT
+#define DIR_SLASH "\\"
+#else
+#define DIR_SLASH "/"
+#endif
+
 #define STR_VALUE(s) #s
 #define STR(s) STR_VALUE (s)
 
@@ -4032,9 +4038,11 @@ DEFUN ("comp-el-to-eln-filename", Fcomp_el_to_eln_filename,
     {
       Lisp_Object sys_re =
  concat2 (build_string ("\\`[[:ascii:]]+"),
- Fregexp_quote (build_string ("/" PATH_REL_LOADSEARCH "/")));
+ Fregexp_quote (build_string (DIR_SLASH PATH_REL_LOADSEARCH
+      DIR_SLASH)));
       loadsearch_re_list =
- list2 (sys_re, Fregexp_quote (build_string (PATH_DUMPLOADSEARCH "/")));
+ list2 (sys_re, Fregexp_quote (build_string (PATH_DUMPLOADSEARCH
+    DIR_SLASH)));
     }
 
   Lisp_Object lds_re_tail = loadsearch_re_list;
Reply | Threaded
Open this post in threaded view
|

bug#46256: [feature/native-comp] AOT eln files ignored if run from build tree

Emacs - Bugs mailing list
In reply to this post by Andy Moreton-3
Andy Moreton <[hidden email]> writes:

> On Thu 04 Feb 2021, Andy Moreton wrote:
>
>> On Wed 03 Feb 2021, akrl--- via "Bug reports for GNU Emacs, the Swiss army knife of text editors" wrote:
>>>
>>> Hi Andy,
>>>
>>> could you share the values of PATH_DUMPLOADSEARCH and
>>> PATH_REL_LOADSEARCH from your epaths.h ?
>>>
>>> Thanks
>>>
>>>   Andrea
>>
>> Native branch checkout is in: "c:/emacs/git/emacs/native/"
>>
>> "c:/emacs/git/emacs/native/build/mingw64-x86_64-O2/src/epaths.h" contains:
>>
>> #define PATH_DUMPLOADSEARCH "C:/emacs/git/emacs/native/lisp"
>> #define PATH_REL_LOADSEARCH "28.0.50/lisp"
>
> I've bootstrapped again after the recent hash shortening to ensure my build
> is up to date, from commit 1f626e9662d8120acd5a937f847123cc2b8c6e31. The
> paths above are unchanged.
>
> Running this from the build dir, I see messages like:
>
> error in process sentinel: Native elisp load failed: "file does not exists",
>  "c:/home/ajm/.emacs.d/eln-cache/28.0.50-e2ae3598/hl-line-e67628ec-664ef650.eln"
>
> This suggests that the AOT .eln files are not being found. It should find:

AFAIK you should not get any error but the eln should be recompiled
automatically.  I guess having updated the hash algorithm the startup is
failing and you need to clean-up completely the build directory
(especially native-lisp/), please retry after a git clean -xfd.

Thanks

  Andrea

> c:/emacs/git/emacs/native/build/mingw64-x86_64-O2/native-lisp/28.0.50-e2ae3598/hl-line-8fa29c14-664ef650.eln
>
> The middle hash (e67628ec vs. 8fa29c14) is not the same - any idea why ?
>
>    AndyM
>
>
>
>
>

--
[hidden email]



Reply | Threaded
Open this post in threaded view
|

bug#46256: [feature/native-comp] AOT eln files ignored if run from build tree

Eli Zaretskii
In reply to this post by Emacs - Bugs mailing list
> Cc: [hidden email]
> Date: Fri, 05 Feb 2021 14:39:58 +0000
> From:  Andrea Corallo via "Bug reports for GNU Emacs,
>  the Swiss army knife of text editors" <[hidden email]>
>
> > Native branch checkout is in: "c:/emacs/git/emacs/native/"
> >
> > "c:/emacs/git/emacs/native/build/mingw64-x86_64-O2/src/epaths.h" contains:
> >
> > #define PATH_DUMPLOADSEARCH "C:/emacs/git/emacs/native/lisp"
> > #define PATH_REL_LOADSEARCH "28.0.50/lisp"
> >
> >
> > HTH,
> >
> >     AndyM
>
> Hi Andy could you give it a go to the following blind patch?

You assume that Windows programs don't understand "/" as a directory
separator?  They do.



Reply | Threaded
Open this post in threaded view
|

bug#46256: [feature/native-comp] AOT eln files ignored if run from build tree

Andy Moreton-3
In reply to this post by Andy Moreton-3
On Fri 05 Feb 2021, Andrea Corallo via "Bug reports for GNU Emacs, the Swiss army knife of text editors" wrote:

> Andy Moreton <[hidden email]> writes:
>
>> On Thu 04 Feb 2021, Andy Moreton wrote:
>>
>>> On Wed 03 Feb 2021, akrl--- via "Bug reports for GNU Emacs, the Swiss army knife of text editors" wrote:
>>>>
>>>> Hi Andy,
>>>>
>>>> could you share the values of PATH_DUMPLOADSEARCH and
>>>> PATH_REL_LOADSEARCH from your epaths.h ?
>>>>
>>>> Thanks
>>>>
>>>>   Andrea
>>>
>>> Native branch checkout is in: "c:/emacs/git/emacs/native/"
>>>
>>> "c:/emacs/git/emacs/native/build/mingw64-x86_64-O2/src/epaths.h" contains:
>>>
>>> #define PATH_DUMPLOADSEARCH "C:/emacs/git/emacs/native/lisp"
>>> #define PATH_REL_LOADSEARCH "28.0.50/lisp"
>>
>> I've bootstrapped again after the recent hash shortening to ensure my build
>> is up to date, from commit 1f626e9662d8120acd5a937f847123cc2b8c6e31. The
>> paths above are unchanged.
>>
>> Running this from the build dir, I see messages like:
>>
>> error in process sentinel: Native elisp load failed: "file does not exists",
>>  "c:/home/ajm/.emacs.d/eln-cache/28.0.50-e2ae3598/hl-line-e67628ec-664ef650.eln"
>>
>> This suggests that the AOT .eln files are not being found. It should find:
>
> AFAIK you should not get any error but the eln should be recompiled
> automatically.  I guess having updated the hash algorithm the startup is
> failing and you need to clean-up completely the build directory
> (especially native-lisp/), please retry after a git clean -xfd.

All of these experiments are done from a clean tree after "git clean
-xdf", and after removing ~/.emacs.d/eln-cache/*.

>> c:/emacs/git/emacs/native/build/mingw64-x86_64-O2/native-lisp/28.0.50-e2ae3598/hl-line-8fa29c14-664ef650.eln
>>
>> The middle hash (e67628ec vs. 8fa29c14) is not the same - any idea why ?

Also a following obsrvation from this:

a) Initially, the AOT bootstrap creates:
   <build-dir>/native-lisp/28.0.50-e2ae3598/hl-line-8fa29c14-664ef650.eln

b) Running <build-dir>/src/emacs complains about:
   <user-emacs-dir>/eln-cache/28.0.50-e2ae3598/hl-line-e67628ec-664ef650.eln

c) The running emacs then builds:
   ~/.emacs.d/eln-cache/28.0.50-e2ae3598/hl-line-8fa29c14-664ef650.eln

The files (a) and (c) have the same filename, but different sizes and
content.

    AndyM





Reply | Threaded
Open this post in threaded view
|

bug#46256: [feature/native-comp] AOT eln files ignored if run from build tree

Emacs - Bugs mailing list
In reply to this post by Andy Moreton-3
Andy Moreton <[hidden email]> writes:

> On Thu 04 Feb 2021, Andy Moreton wrote:
>
>> On Thu 04 Feb 2021, Andy Moreton wrote:
>>
>>> On Wed 03 Feb 2021, akrl--- via "Bug reports for GNU Emacs, the Swiss army knife of text editors" wrote:
>>>>
>>>> Hi Andy,
>>>>
>>>> could you share the values of PATH_DUMPLOADSEARCH and
>>>> PATH_REL_LOADSEARCH from your epaths.h ?
>>>>
>>>> Thanks
>>>>
>>>>   Andrea
>>>
>>> Native branch checkout is in: "c:/emacs/git/emacs/native/"
>>>
>>> "c:/emacs/git/emacs/native/build/mingw64-x86_64-O2/src/epaths.h" contains:
>>>
>>> #define PATH_DUMPLOADSEARCH "C:/emacs/git/emacs/native/lisp"
>>> #define PATH_REL_LOADSEARCH "28.0.50/lisp"
>>
>> I've bootstrapped again after the recent hash shortening to ensure my build
>> is up to date, from commit 1f626e9662d8120acd5a937f847123cc2b8c6e31. The
>> paths above are unchanged.
>>
>> Running this from the build dir, I see messages like:
>>
>> error in process sentinel: Native elisp load failed: "file does not exists",
>>  "c:/home/ajm/.emacs.d/eln-cache/28.0.50-e2ae3598/hl-line-e67628ec-664ef650.eln"
>>
>> This suggests that the AOT .eln files are not being found. It should find:
>>
>> c:/emacs/git/emacs/native/build/mingw64-x86_64-O2/native-lisp/28.0.50-e2ae3598/hl-line-8fa29c14-664ef650.eln
>>
>> The middle hash (e67628ec vs. 8fa29c14) is not the same - any idea why ?
>
> After looking at what `comp-el-to-eln-filename' does, I observe that:
>
> (substring (md5 "c:/emacs/git/emacs/native/lisp/hl-line.el") 0 8)
> "e67628ec"
>
> (substring (md5 "//hl-line.el") 0 8)
> "8fa29c14"
>
> That matches the two middle hashes seen above.
>
> It looks like `comp-el-to-eln-filename` fails to match the filename
> prefix against PATH_DUMPLOADSEARCH. It is using case-sensitive matching,
> but on Windows filesystems are case-insensitive.

Hi Andy,

The Windows filesystem is case-insensitive but the case is preserved
correct?  If so it should work no?

Last queston: do reverse slashes '\' appear somewhere in those
filenames?  This was issue I tried to fix with the blind patch I've
sent.

Thanks

  Andrea



Reply | Threaded
Open this post in threaded view
|

bug#46256: [feature/native-comp] AOT eln files ignored if run from build tree

Eli Zaretskii
> Cc: [hidden email]
> Date: Thu, 18 Feb 2021 21:00:29 +0000
> From:  Andrea Corallo via "Bug reports for GNU Emacs,
>  the Swiss army knife of text editors" <[hidden email]>
>
> >> Last queston: do reverse slashes '\' appear somewhere in those
> >> filenames?  This was issue I tried to fix with the blind patch I've
> >> sent.
> >
> > As Eli pointed out, that is not the problem: forward slashes are ok.
>
> I understand they are handled, but here as we do a substitution we must
> substitute what's coming in.
>
> As you have the possibility to debug this piece of code on Windows
> please have a look at this (or try my blind patch if you haven't).

If the problem is with hashing file names, you will have to
canonicalize them first, including resolving the letter-case issue,
the forward/back-slashes issue, and also the issue with those pesky
numerical tails Windows sometimes produces.  We have a function
Fw32_long_file_name for that purpose, I think you should use it (if
you need it for C strings, we could add a wrapper around
w32_get_long_filename to do that instead).  This assumes that you are
talking about existing files; if that assumption is not true, we will
need a slightly different strategy.




Reply | Threaded
Open this post in threaded view
|

bug#46256: [feature/native-comp] AOT eln files ignored if run from build tree

Andy Moreton-3
In reply to this post by Andy Moreton-3
On Fri 19 Feb 2021, Eli Zaretskii wrote:

>> Cc: [hidden email]
>> Date: Thu, 18 Feb 2021 21:00:29 +0000
>> From:  Andrea Corallo via "Bug reports for GNU Emacs,
>>  the Swiss army knife of text editors" <[hidden email]>
>>
>> >> Last queston: do reverse slashes '\' appear somewhere in those
>> >> filenames?  This was issue I tried to fix with the blind patch I've
>> >> sent.
>> >
>> > As Eli pointed out, that is not the problem: forward slashes are ok.
>>
>> I understand they are handled, but here as we do a substitution we must
>> substitute what's coming in.
>>
>> As you have the possibility to debug this piece of code on Windows
>> please have a look at this (or try my blind patch if you haven't).
>
> If the problem is with hashing file names, you will have to
> canonicalize them first, including resolving the letter-case issue,
> the forward/back-slashes issue, and also the issue with those pesky
> numerical tails Windows sometimes produces.  We have a function
> Fw32_long_file_name for that purpose, I think you should use it (if
> you need it for C strings, we could add a wrapper around
> w32_get_long_filename to do that instead).  This assumes that you are
> talking about existing files; if that assumption is not true, we will
> need a slightly different strategy.

The problem is with the file names used to generate the hashes, where
comparison of file names.

As an experiment, I changed epaths.h from:
#define PATH_DUMPLOADSEARCH "C:/emacs/git/emacs/native/lisp"

to:
#define PATH_DUMPLOADSEARCH "c:/emacs/git/emacs/native/lisp"

and then ran make (to build without regenerating the header).
The resulting emacs did not complain about mismatched filenames.

Thus the fix outlined by Eli above looks like it will solve the problem.

    AndyM




Reply | Threaded
Open this post in threaded view
|

bug#46256: [feature/native-comp] AOT eln files ignored if run from build tree

Eli Zaretskii
> From: Andy Moreton <[hidden email]>
> Date: Fri, 19 Feb 2021 14:49:25 +0000
>
> As an experiment, I changed epaths.h from:
> #define PATH_DUMPLOADSEARCH "C:/emacs/git/emacs/native/lisp"
>
> to:
> #define PATH_DUMPLOADSEARCH "c:/emacs/git/emacs/native/lisp"
>
> and then ran make (to build without regenerating the header).
> The resulting emacs did not complain about mismatched filenames.
>
> Thus the fix outlined by Eli above looks like it will solve the problem.

Btw, there's a similar in principle, but different in details, problem
with macOS: it stores file names in decomposed form, i.e., for
example, รค will be stored as two codepoints: a, followed by U+00A8
DIAERESIS.  So any hashing that relies on comparing file names as
strings will need to normalize the file names on macOS filesystems
(HFS) as well.



Reply | Threaded
Open this post in threaded view
|

bug#46256: [feature/native-comp] AOT eln files ignored if run from build tree

Emacs - Bugs mailing list
In reply to this post by Andy Moreton-3
Andy Moreton <[hidden email]> writes:

> As an experiment, I changed epaths.h from:
> #define PATH_DUMPLOADSEARCH "C:/emacs/git/emacs/native/lisp"
>
> to:
> #define PATH_DUMPLOADSEARCH "c:/emacs/git/emacs/native/lisp"
>
> and then ran make (to build without regenerating the header).
> The resulting emacs did not complain about mismatched filenames.
>
> Thus the fix outlined by Eli above looks like it will solve the problem.

Sounds great!

I'll write the fix in the following days (in case somebody wants to take
over the task just mention it here, indeed this is very welcome).

Thanks

  Andrea



Reply | Threaded
Open this post in threaded view
|

bug#46256: [feature/native-comp] AOT eln files ignored if run from build tree

Emacs - Bugs mailing list
In reply to this post by Andy Moreton-3
Andy Moreton <[hidden email]> writes:

[...]

> The problem is with the file names used to generate the hashes, where
> comparison of file names.
>
> As an experiment, I changed epaths.h from:
> #define PATH_DUMPLOADSEARCH "C:/emacs/git/emacs/native/lisp"
>
> to:
> #define PATH_DUMPLOADSEARCH "c:/emacs/git/emacs/native/lisp"
>
> and then ran make (to build without regenerating the header).
> The resulting emacs did not complain about mismatched filenames.
>
> Thus the fix outlined by Eli above looks like it will solve the problem.
>
>     AndyM
Hi Andy,

could you give it a try to the attached patch?  It follows Eli's
suggestion of using 'Fw32_long_file_name'.

Thanks

  Andrea

From 312deba5302a8136fa104b054af54572cc64ea5e Mon Sep 17 00:00:00 2001
From: Andrea Corallo <[hidden email]>
Date: Fri, 26 Feb 2021 21:27:02 +0100
Subject: [PATCH] * Canonicalize filenames on Windows before hashing
 (bug#46256)

        * src/comp.c (Fcomp_el_to_eln_filename): On Windowns
        canonicalize filenames before hashing.
---
 src/comp.c | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)

diff --git a/src/comp.c b/src/comp.c
index a8b8ef95fa..1a89e4e62a 100644
--- a/src/comp.c
+++ b/src/comp.c
@@ -3983,6 +3983,10 @@ DEFUN ("comp-el-to-eln-filename", Fcomp_el_to_eln_filename,
   if (NILP (Ffile_exists_p (filename)))
     xsignal1 (Qfile_missing, filename);
 
+#ifdef WINDOWSNT
+  filename = Fw32_long_file_name (filename);
+#endif
+
   Lisp_Object content_hash = comp_hash_source_file (filename);
 
   if (suffix_p (filename, ".gz"))
@@ -4014,8 +4018,11 @@ DEFUN ("comp-el-to-eln-filename", Fcomp_el_to_eln_filename,
       Lisp_Object sys_re =
  concat2 (build_string ("\\`[[:ascii:]]+"),
  Fregexp_quote (build_string ("/" PATH_REL_LOADSEARCH "/")));
-      loadsearch_re_list =
- list2 (sys_re, Fregexp_quote (build_string (PATH_DUMPLOADSEARCH "/")));
+      Lisp_Object dump_load_search = build_string (PATH_DUMPLOADSEARCH "/");
+#ifdef WINDOWSNT
+      dump_load_search = Fw32_long_file_name (dump_load_search);
+#endif
+      loadsearch_re_list = list2 (sys_re, Fregexp_quote (dump_load_search));
     }
 
   Lisp_Object lds_re_tail = loadsearch_re_list;
--
2.20.1

Reply | Threaded
Open this post in threaded view
|

bug#46256: [feature/native-comp] AOT eln files ignored if run from build tree

Eli Zaretskii
> Cc: [hidden email]
> Date: Fri, 26 Feb 2021 20:34:10 +0000
> From:  Andrea Corallo via "Bug reports for GNU Emacs,
>  the Swiss army knife of text editors" <[hidden email]>
>
> --- a/src/comp.c
> +++ b/src/comp.c
> @@ -3983,6 +3983,10 @@ DEFUN ("comp-el-to-eln-filename", Fcomp_el_to_eln_filename,
>    if (NILP (Ffile_exists_p (filename)))
>      xsignal1 (Qfile_missing, filename);
>  
> +#ifdef WINDOWSNT
> +  filename = Fw32_long_file_name (filename);
> +#endif

Is "filename" here a name of an existing file?  If not,
Fw32_long_file_name will return nil.



Reply | Threaded
Open this post in threaded view
|

bug#46256: [feature/native-comp] AOT eln files ignored if run from build tree

Emacs - Bugs mailing list
Eli Zaretskii <[hidden email]> writes:

>> Cc: [hidden email]
>> Date: Fri, 26 Feb 2021 20:34:10 +0000
>> From:  Andrea Corallo via "Bug reports for GNU Emacs,
>>  the Swiss army knife of text editors" <[hidden email]>
>>
>> --- a/src/comp.c
>> +++ b/src/comp.c
>> @@ -3983,6 +3983,10 @@ DEFUN ("comp-el-to-eln-filename", Fcomp_el_to_eln_filename,
>>    if (NILP (Ffile_exists_p (filename)))
>>      xsignal1 (Qfile_missing, filename);
>>  
>> +#ifdef WINDOWSNT
>> +  filename = Fw32_long_file_name (filename);
>> +#endif
>
> Is "filename" here a name of an existing file?  If not,
> Fw32_long_file_name will return nil.

It should always be as we explicitly check for that.

Quick question: I assumed Fw32_long_file_name works for directories as
well, is this correct?

Thanks

  Andrea



Reply | Threaded
Open this post in threaded view
|

bug#46256: [feature/native-comp] AOT eln files ignored if run from build tree

Eli Zaretskii
> From: Andrea Corallo <[hidden email]>
> Cc: [hidden email], [hidden email]
> Date: Fri, 26 Feb 2021 20:48:52 +0000
>
> Quick question: I assumed Fw32_long_file_name works for directories as
> well, is this correct?

Yes, it does.



Reply | Threaded
Open this post in threaded view
|

bug#46256: [feature/native-comp] AOT eln files ignored if run from build tree

Eli Zaretskii
> From: Andrea Corallo <[hidden email]>
> Cc: [hidden email], [hidden email]
> Date: Sat, 27 Feb 2021 06:58:45 +0000
>
> Nice, thinking about I've got a last question: normalizing "c:/foo/" the
> trainling '/' is kept or removed?  If the case is the second the patch
> needs an adjustment.

A single trailing slash, if any, is kept.  Multiple trailing slashes
are collapsed into a single one.



Reply | Threaded
Open this post in threaded view
|

bug#46256: [feature/native-comp] AOT eln files ignored if run from build tree

Andy Moreton-3
In reply to this post by Andy Moreton-3
On Fri 26 Feb 2021, Andrea Corallo via "Bug reports for GNU Emacs, the Swiss army knife of text editors" wrote:

> Andy Moreton <[hidden email]> writes:
>
> [...]
>
>> The problem is with the file names used to generate the hashes, where
>> comparison of file names.
>>
>> As an experiment, I changed epaths.h from:
>> #define PATH_DUMPLOADSEARCH "C:/emacs/git/emacs/native/lisp"
>>
>> to:
>> #define PATH_DUMPLOADSEARCH "c:/emacs/git/emacs/native/lisp"
>>
>> and then ran make (to build without regenerating the header).
>> The resulting emacs did not complain about mismatched filenames.
>>
>> Thus the fix outlined by Eli above looks like it will solve the problem.
>>
>>     AndyM
>
> Hi Andy,
>
> could you give it a try to the attached patch?  It follows Eli's
> suggestion of using 'Fw32_long_file_name'.

The patch looks good - please apply it.

I tried building with the patch applied to a clean tree, and the
resulting emacs runs without the filename mismatch messages, and did not
recompile the AOT files into the per-user eln-cache.

There were also a couple of errors in the build:

Backtrace:
00007ff78467a2a2
00007ff78453be26
00007ff7845a98ac
...[snipped]...
00007ff784626548
Eager macro-expansion failure: (file-error "Renaming" "Permission
denied"
"c:/emacs/git/emacs/native/build/mingw64-x86_64-O2-native/native-lisp/28.0.50-e09cfb99/cc-bytecomp-4817e810-d16f606e.eln"
"c:/emacs/git/emacs/native/build/mingw64-x86_64-O2-native/native-lisp/28.0.50-e09cfb99/cc-bytecomp-4817e810-d16f606e.elnGMMUdn.eln.tmp")
C:/emacs/git/emacs/native/src/alloc.c:3160: Emacs fatal error: assertion failed: cu->handle
make[2]: *** [Makefile:319: progmodes/antlr-mode.elc] Error 3

The backtrace addresses did not give anything useful from addr2line.

There are still some elisp files that did not get native compiled when
the build was done with "make -j8 NATIVE_FULL_AOT=1", and so get async
compiled when running the built emacs:

  ansi-color auth_source byte-opt bytecomp cconv cl-extra cl-lib cl-macs
  cl-seq comint comp comp-cstr cus-edit cus-start desktop
  display-fill-column-indicator easy-mmode easymenu edmacro eieio
  eieio-core frameset gv help-mode hl-line image-file info json kmacro
  map minibuf-eldef package paren password-cache pcase pp ring rx seq
  subr-x time-date warnings wid-edit

That may be a result of the error during the build.

    AndyM




Reply | Threaded
Open this post in threaded view
|

bug#46256: [feature/native-comp] AOT eln files ignored if run from build tree

Emacs - Bugs mailing list
Andy Moreton <[hidden email]> writes:

> On Fri 26 Feb 2021, Andrea Corallo via "Bug reports for GNU Emacs, the Swiss army knife of text editors" wrote:
>
>> Andy Moreton <[hidden email]> writes:
>>
>> [...]
>>
>>> The problem is with the file names used to generate the hashes, where
>>> comparison of file names.
>>>
>>> As an experiment, I changed epaths.h from:
>>> #define PATH_DUMPLOADSEARCH "C:/emacs/git/emacs/native/lisp"
>>>
>>> to:
>>> #define PATH_DUMPLOADSEARCH "c:/emacs/git/emacs/native/lisp"
>>>
>>> and then ran make (to build without regenerating the header).
>>> The resulting emacs did not complain about mismatched filenames.
>>>
>>> Thus the fix outlined by Eli above looks like it will solve the problem.
>>>
>>>     AndyM
>>
>> Hi Andy,
>>
>> could you give it a try to the attached patch?  It follows Eli's
>> suggestion of using 'Fw32_long_file_name'.
>
> The patch looks good - please apply it.

Thanks for verifying it, installed as 312deba530.

> I tried building with the patch applied to a clean tree, and the
> resulting emacs runs without the filename mismatch messages, and did not
> recompile the AOT files into the per-user eln-cache.
>
> There were also a couple of errors in the build:
>
> Backtrace:
> 00007ff78467a2a2
> 00007ff78453be26
> 00007ff7845a98ac
> ...[snipped]...
> 00007ff784626548
> Eager macro-expansion failure: (file-error "Renaming" "Permission
> denied"
> "c:/emacs/git/emacs/native/build/mingw64-x86_64-O2-native/native-lisp/28.0.50-e09cfb99/cc-bytecomp-4817e810-d16f606e.eln"
> "c:/emacs/git/emacs/native/build/mingw64-x86_64-O2-native/native-lisp/28.0.50-e09cfb99/cc-bytecomp-4817e810-d16f606e.elnGMMUdn.eln.tmp")
> C:/emacs/git/emacs/native/src/alloc.c:3160: Emacs fatal error: assertion failed: cu->handle
> make[2]: *** [Makefile:319: progmodes/antlr-mode.elc] Error 3
>
> The backtrace addresses did not give anything useful from addr2line.
>
> There are still some elisp files that did not get native compiled when
> the build was done with "make -j8 NATIVE_FULL_AOT=1", and so get async
> compiled when running the built emacs:
>
>   ansi-color auth_source byte-opt bytecomp cconv cl-extra cl-lib cl-macs
>   cl-seq comint comp comp-cstr cus-edit cus-start desktop
>   display-fill-column-indicator easy-mmode easymenu edmacro eieio
>   eieio-core frameset gv help-mode hl-line image-file info json kmacro
>   map minibuf-eldef package paren password-cache pcase pp ring rx seq
>   subr-x time-date warnings wid-edit
>
> That may be a result of the error during the build.

Mmmmh, that's strange some of these are even compiled as COMPILE_FIRST
therfore are certainly native compiled.

One thing you could do (before one of these is recompiled) is to use
`comp-el-to-eln-filename' to check what the native compiler is expecting
as eln filename and if this is present in any of the folders in your
`comp-eln-load-path'.

Thanks

  Andrea



Reply | Threaded
Open this post in threaded view
|

bug#46256: [feature/native-comp] AOT eln files ignored if run from build tree

Emacs - Bugs mailing list
Andrea Corallo via "Bug reports for GNU Emacs, the Swiss army knife of
text editors" <[hidden email]> writes:

> Andy Moreton <[hidden email]> writes:
>
>> On Fri 26 Feb 2021, Andrea Corallo via "Bug reports for GNU Emacs, the Swiss army knife of text editors" wrote:
>>
>>> Andy Moreton <[hidden email]> writes:
>>>
>>> [...]
>>>
>>>> The problem is with the file names used to generate the hashes, where
>>>> comparison of file names.
>>>>
>>>> As an experiment, I changed epaths.h from:
>>>> #define PATH_DUMPLOADSEARCH "C:/emacs/git/emacs/native/lisp"
>>>>
>>>> to:
>>>> #define PATH_DUMPLOADSEARCH "c:/emacs/git/emacs/native/lisp"
>>>>
>>>> and then ran make (to build without regenerating the header).
>>>> The resulting emacs did not complain about mismatched filenames.
>>>>
>>>> Thus the fix outlined by Eli above looks like it will solve the problem.
>>>>
>>>>     AndyM
>>>
>>> Hi Andy,
>>>
>>> could you give it a try to the attached patch?  It follows Eli's
>>> suggestion of using 'Fw32_long_file_name'.
>>
>> The patch looks good - please apply it.
>
> Thanks for verifying it, installed as 312deba530.
>
>> I tried building with the patch applied to a clean tree, and the
>> resulting emacs runs without the filename mismatch messages, and did not
>> recompile the AOT files into the per-user eln-cache.
>>
>> There were also a couple of errors in the build:
>>
>> Backtrace:
>> 00007ff78467a2a2
>> 00007ff78453be26
>> 00007ff7845a98ac
>> ...[snipped]...
>> 00007ff784626548
>> Eager macro-expansion failure: (file-error "Renaming" "Permission
>> denied"
>> "c:/emacs/git/emacs/native/build/mingw64-x86_64-O2-native/native-lisp/28.0.50-e09cfb99/cc-bytecomp-4817e810-d16f606e.eln"
>> "c:/emacs/git/emacs/native/build/mingw64-x86_64-O2-native/native-lisp/28.0.50-e09cfb99/cc-bytecomp-4817e810-d16f606e.elnGMMUdn.eln.tmp")
>> C:/emacs/git/emacs/native/src/alloc.c:3160: Emacs fatal error: assertion failed: cu->handle
>> make[2]: *** [Makefile:319: progmodes/antlr-mode.elc] Error 3
>>
>> The backtrace addresses did not give anything useful from addr2line.
>>
>> There are still some elisp files that did not get native compiled when
>> the build was done with "make -j8 NATIVE_FULL_AOT=1", and so get async
>> compiled when running the built emacs:
>>
>>   ansi-color auth_source byte-opt bytecomp cconv cl-extra cl-lib cl-macs
>>   cl-seq comint comp comp-cstr cus-edit cus-start desktop
>>   display-fill-column-indicator easy-mmode easymenu edmacro eieio
>>   eieio-core frameset gv help-mode hl-line image-file info json kmacro
>>   map minibuf-eldef package paren password-cache pcase pp ring rx seq
>>   subr-x time-date warnings wid-edit
>>
>> That may be a result of the error during the build.
>
> Mmmmh, that's strange some of these are even compiled as COMPILE_FIRST
> therfore are certainly native compiled.
>
> One thing you could do (before one of these is recompiled) is to use
> `comp-el-to-eln-filename' to check what the native compiler is expecting
> as eln filename and if this is present in any of the folders in your
> `comp-eln-load-path'.

Apologies, to be more precise: if the file is compiled during the build
(as should be in this case) it should be in the "native-lisp/" dir in
the build tree.

  Andrea



1234