bug#36490: 26.1; directory-files-recursively breaks when it encounters a directory named "~"

classic Classic list List threaded Threaded
23 messages Options
12
Reply | Threaded
Open this post in threaded view
|

bug#36490: 26.1; directory-files-recursively breaks when it encounters a directory named "~"

Erik Hahn

Call (directory-files-recursively DIR ".*") where DIR contains a
subdirectory named "~" but is not the parent of your home directory. The
function will recurse into your home directory. This is probably because
(expand-file-name "~" ...) returns the home directory.


In GNU Emacs 26.1 (build 1, x86_64-pc-linux-gnu, X toolkit)
 of 2018-11-03 built on p5
Windowing system distributor 'The X.Org Foundation', version 11.0.12003000
System Description: Gentoo Base System release 2.6

Recent messages:
Configured using:
 'configure --prefix=/usr --build=x86_64-pc-linux-gnu
 --host=x86_64-pc-linux-gnu --mandir=/usr/share/man
 --infodir=/usr/share/info --datadir=/usr/share --sysconfdir=/etc
 --localstatedir=/var/lib --disable-silent-rules
 --docdir=/usr/share/doc/emacs-26.1-r3
 --htmldir=/usr/share/doc/emacs-26.1-r3/html --libdir=/usr/lib64
 --program-suffix=-emacs-26 --infodir=/usr/share/info/emacs-26
 --localstatedir=/var
 --enable-locallisppath=/etc/emacs:/usr/share/emacs/site-lisp
 --without-compress-install --without-hesiod --without-pop
 --with-file-notification=inotify --enable-acl --with-dbus
 --without-modules --without-gameuser --with-gpm --without-kerberos
 --without-kerberos5 --without-lcms2 --with-xml2 --without-mailutils
 --without-selinux --with-gnutls --with-libsystemd --with-threads
 --without-wide-int --with-zlib --with-sound=alsa --with-x --without-ns
 --without-gconf --without-gsettings --without-toolkit-scroll-bars
 --with-gif --with-jpeg --with-png --with-rsvg --with-tiff --with-xpm
 --without-imagemagick --without-xft --without-cairo --without-libotf
 --without-m17n-flt --with-x-toolkit=lucid --with-xaw3d 'CFLAGS=-O2
 -pipe -march=native' CPPFLAGS= 'LDFLAGS=-Wl,-O1 -Wl,--as-needed''

Configured features:
XAW3D XPM JPEG TIFF GIF PNG RSVG SOUND GPM DBUS NOTIFY ACL GNUTLS
LIBXML2 ZLIB LUCID X11 THREADS LIBSYSTEMD

Important settings:
  value of $LC_MONETARY: en_US.UTF-8
  value of $LC_NUMERIC: en_US.UTF-8
  value of $LC_TIME: en_GB.UTF-8
  value of $LANG: en_GB.utf8
  locale-coding-system: utf-8-unix

Major mode: Lisp Interaction



Reply | Threaded
Open this post in threaded view
|

bug#36490: 26.1; directory-files-recursively breaks when it encounters a directory named "~"

Lars Ingebrigtsen
Erik Hahn <[hidden email]> writes:

> Call (directory-files-recursively DIR ".*") where DIR contains a
> subdirectory named "~" but is not the parent of your home directory. The
> function will recurse into your home directory. This is probably because
> (expand-file-name "~" ...) returns the home directory.

Oops:

(expand-file-name "~" "/tmp/")
=> "/home/larsi"

Yup; that function should be rewritten to have no instances of
`expand-file-name', which is too DWIM for a function like this.  I'll do
a rewrite...

--
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no



Reply | Threaded
Open this post in threaded view
|

bug#36490: 26.1; directory-files-recursively breaks when it encounters a directory named "~"

Lars Ingebrigtsen
In reply to this post by Erik Hahn
Actually, the doc string of expand-file-name is either wrong, or the
implementation is.

It says:

---
An initial ‘~/’ expands to your home directory.
An initial ‘~USER/’ expands to USER’s home directory.
---

Assuming the "An initial" refers to the first parameter, then

(expand-file-name "~/" "/tmp/")
=> "/home/larsi/"

works as advertised, but

(expand-file-name "~" "/tmp/")
=> "/home/larsi"

is a different thing: "~" is a perfectly valid file name, so having this
function map that to something else is just...  wrong.

(expand-file-name "~larsi" "/tmp/")
=> "/home/larsi"

is the same: Also wrong and undocumented.

The doc string continues with further confusion:

---
See also the function ‘substitute-in-file-name’.
---

See it for...  what?  For further expansions this function is going to
do?  Fortunately not:

(expand-file-name "$HOME" "/tmp/")
=> "/tmp/$HOME"

So that's probably just meant as "that's also a function that does file
name stuff, but it has nothing to do with this ~ thing we just
discussed"?

So what to do here?  I think the current, undocumented

(expand-file-name "~" "/tmp/")
=> "/home/larsi"

must surely be an error, and that should be fixed instead of the
callers?  Opinions?

--
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no




Reply | Threaded
Open this post in threaded view
|

bug#36490: 26.1; directory-files-recursively breaks when it encounters a directory named "~"

Erik Hahn
On 08/07/19 23:08, Lars Ingebrigtsen wrote:
> The doc string continues with further confusion:
>
> ---
> See also the function ‘substitute-in-file-name’.
> ---
>
> See it for...  what?  For further expansions this function is going to
> do?

It is related in that it also chokes on files named "~":

(substitute-in-file-name "/tmp/~") => "~"

Its doc string is also wrong. It says

 > If ‘/~’ appears, all of FILENAME through that ‘/’ is discarded.

but

(substitute-in-file-name "/tmp/~foo") => "/tmp/~foo"



Reply | Threaded
Open this post in threaded view
|

bug#36490: 26.1; directory-files-recursively breaks when it encounters a directory named "~"

Lars Ingebrigtsen
Erik Hahn <[hidden email]> writes:

> It is related in that it also chokes on files named "~":
>
> (substitute-in-file-name "/tmp/~") => "~"
>
> Its doc string is also wrong. It says
>
>> If ‘/~’ appears, all of FILENAME through that ‘/’ is discarded.
>
> but
>
> (substitute-in-file-name "/tmp/~foo") => "/tmp/~foo"

Yup, it's only if the user exists:

(substitute-in-file-name "/tmp/~larsi")
=> "~larsi"

Same as with expand-file-name:

(expand-file-name "~larsi" "/tmp/")
=> "/home/larsi"

(expand-file-name "~foo" "/tmp/")
=> "/tmp/~foo"

--
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no



Reply | Threaded
Open this post in threaded view
|

bug#36490: 26.1; directory-files-recursively breaks when it encounters a directory named "~"

Eli Zaretskii
In reply to this post by Lars Ingebrigtsen
> From: Lars Ingebrigtsen <[hidden email]>
> Date: Mon, 08 Jul 2019 23:08:46 +0200
> Cc: [hidden email]
>
> Actually, the doc string of expand-file-name is either wrong, or the
> implementation is.
>
> It says:
>
> ---
> An initial ‘~/’ expands to your home directory.
> An initial ‘~USER/’ expands to USER’s home directory.
> ---
>
> Assuming the "An initial" refers to the first parameter, then
>
> (expand-file-name "~/" "/tmp/")
> => "/home/larsi/"
>
> works as advertised, but
>
> (expand-file-name "~" "/tmp/")
> => "/home/larsi"
>
> is a different thing: "~" is a perfectly valid file name, so having this
> function map that to something else is just...  wrong.

If you want "~" to be interpreted literally, you need to protect it
with "/:".

> (expand-file-name "~larsi" "/tmp/")
> => "/home/larsi"
>
> is the same: Also wrong and undocumented.

Why would we want to document that?

> The doc string continues with further confusion:
>
> ---
> See also the function ‘substitute-in-file-name’.
> ---
>
> See it for...  what?

For expanding environment variables, and for the special effect of
"//" etc.

> So what to do here?  I think the current, undocumented
>
> (expand-file-name "~" "/tmp/")
> => "/home/larsi"
>
> must surely be an error, and that should be fixed instead of the
> callers?  Opinions?

It is not an error.



Reply | Threaded
Open this post in threaded view
|

bug#36490: 26.1; directory-files-recursively breaks when it encounters a directory named "~"

Eli Zaretskii
In reply to this post by Erik Hahn
> From: Erik Hahn <[hidden email]>
> Date: Mon, 8 Jul 2019 23:30:38 +0200
> Cc: [hidden email]
>
>  > If ‘/~’ appears, all of FILENAME through that ‘/’ is discarded.
>
> but
>
> (substitute-in-file-name "/tmp/~foo") => "/tmp/~foo"

Do you have a user named "foo" on that system?  If not, try a real
user name instead of "foo".



Reply | Threaded
Open this post in threaded view
|

bug#36490: 26.1; directory-files-recursively breaks when it encounters a directory named "~"

Lars Ingebrigtsen
In reply to this post by Eli Zaretskii
Eli Zaretskii <[hidden email]> writes:

>> An initial ‘~/’ expands to your home directory.
>> An initial ‘~USER/’ expands to USER’s home directory.
>> ---
>>
>> Assuming the "An initial" refers to the first parameter, then

[...]

> If you want "~" to be interpreted literally, you need to protect it
> with "/:".

Sorry; I don't quite follow you here.  The doc string says that "~/" is
interpreted specially.  There's no "/" in "~".  :-)

That

(expand-file-name "~/" "/tmp/")
=> "/home/larsi/"

is fine, because "~/" isn't a valid file name.  But "~" is, and that's
the problem.

--
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no



Reply | Threaded
Open this post in threaded view
|

bug#36490: 26.1; directory-files-recursively breaks when it encounters a directory named "~"

Eli Zaretskii
> From: Lars Ingebrigtsen <[hidden email]>
> Cc: [hidden email],  [hidden email]
> Date: Tue, 09 Jul 2019 17:50:49 +0200
>
> > If you want "~" to be interpreted literally, you need to protect it
> > with "/:".
>
> Sorry; I don't quite follow you here.  The doc string says that "~/" is
> interpreted specially.  There's no "/" in "~".  :-)

But it does NOT say that "~" will NOT be interpreted specially.

> That
>
> (expand-file-name "~/" "/tmp/")
> => "/home/larsi/"
>
> is fine, because "~/" isn't a valid file name.  But "~" is, and that's
> the problem.

No, I think the problem is that the caller should have protected the
literal "~", as I said.



Reply | Threaded
Open this post in threaded view
|

bug#36490: 26.1; directory-files-recursively breaks when it encounters a directory named "~"

Andreas Schwab-2
In reply to this post by Lars Ingebrigtsen
On Jul 09 2019, Lars Ingebrigtsen <[hidden email]> wrote:

> is fine, because "~/" isn't a valid file name.

"~/" is a valid file name, that happens to be a directory.

Andreas.

--
Andreas Schwab, [hidden email]
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."



Reply | Threaded
Open this post in threaded view
|

bug#36490: 26.1; directory-files-recursively breaks when it encounters a directory named "~"

Lars Ingebrigtsen
In reply to this post by Eli Zaretskii
Eli Zaretskii <[hidden email]> writes:

>> > If you want "~" to be interpreted literally, you need to protect it
>> > with "/:".
>>
>> Sorry; I don't quite follow you here.  The doc string says that "~/" is
>> interpreted specially.  There's no "/" in "~".  :-)
>
> But it does NOT say that "~" will NOT be interpreted specially.

I think it's a good idea to specify what strings will be interpreted
specially -- otherwise it's a bit difficult to use that function.

>> is fine, because "~/" isn't a valid file name.  But "~" is, and that's
>> the problem.
>
> No, I think the problem is that the caller should have protected the
> literal "~", as I said.

Any function that iterates over a list of directory files (as returned
by directory-files) would then have to quote whatever it passes to
expand-file-name -- or avoid that function completely.  That seems like
an odd design.

--
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no



Reply | Threaded
Open this post in threaded view
|

bug#36490: 26.1; directory-files-recursively breaks when it encounters a directory named "~"

Eli Zaretskii
> From: Lars Ingebrigtsen <[hidden email]>
> Cc: [hidden email],  [hidden email]
> Date: Tue, 09 Jul 2019 18:27:28 +0200
>
> > No, I think the problem is that the caller should have protected the
> > literal "~", as I said.
>
> Any function that iterates over a list of directory files (as returned
> by directory-files) would then have to quote whatever it passes to
> expand-file-name -- or avoid that function completely.  That seems like
> an odd design.

What happens if a function that iterates over a directory finds a
subdirectory whose name is literally "~"?



Reply | Threaded
Open this post in threaded view
|

bug#36490: 26.1; directory-files-recursively breaks when it encounters a directory named "~"

Lars Ingebrigtsen
Eli Zaretskii <[hidden email]> writes:

>> Any function that iterates over a list of directory files (as returned
>> by directory-files) would then have to quote whatever it passes to
>> expand-file-name -- or avoid that function completely.  That seems like
>> an odd design.
>
> What happens if a function that iterates over a directory finds a
> subdirectory whose name is literally "~"?

I think I must be missing something, because I don't think anything
special should happen?

(find-file "/tmp/~/foo")

works fine, for instance.  And

(expand-file-name "foo" "/tmp/~")
=> "/tmp/~/foo"

--
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no



Reply | Threaded
Open this post in threaded view
|

bug#36490: 26.1; directory-files-recursively breaks when it encounters a directory named "~"

Eli Zaretskii
In reply to this post by Eli Zaretskii
> Date: Tue, 09 Jul 2019 19:44:31 +0300
> From: Eli Zaretskii <[hidden email]>
> Cc: [hidden email], [hidden email]
>
> What happens if a function that iterates over a directory finds a
> subdirectory whose name is literally "~"?

Sorry, that's not what I meant to say.  I meant to say how would a
Lisp program know whether (expand-file-name "~/") means the home
directory or a directory whose name is literally "~"?

Btw, stuff like (expand-file-name "foo/~/") already does what you
want, so the problem is only with the leading '~', and can be avoided
if we avoid that situation.  IOW, why should this example:

  (expand-file-name "~" "/tmp/")
     => "/home/larsi"

determine how directory-files-recursively behaves?



Reply | Threaded
Open this post in threaded view
|

bug#36490: 26.1; directory-files-recursively breaks when it encounters a directory named "~"

Lars Ingebrigtsen
Eli Zaretskii <[hidden email]> writes:

> Sorry, that's not what I meant to say.  I meant to say how would a
> Lisp program know whether (expand-file-name "~/") means the home
> directory or a directory whose name is literally "~"?

Well, we have documented that in expand-file-name "~/" means the home
directory, and I have no problems with that.

"~/" isn't something you'll ever get from functions like
directory-files, so it's not something you'd feed to expand-file-name in
these situations...

> Btw, stuff like (expand-file-name "foo/~/") already does what you
> want, so the problem is only with the leading '~', and can be avoided
> if we avoid that situation.  IOW, why should this example:
>
>   (expand-file-name "~" "/tmp/")
>      => "/home/larsi"
>
> determine how directory-files-recursively behaves?

expand-file-name's use case is to (basically) concatenate a directory
name and a file name, but it's used instead of concat because nobody
wants to care about whether the directory name has a trailing slash or
not.

(concat "/tmp/" "foo")
=> "/tmp/foo" ; Good

(concat "/tmp" "foo")
=> "/tmpfoo" ; Bad.

(expand-file-name "foo" "/tmp")
=> "/tmp/foo" ; Yay

That's basically the use case for expand-file-name, and using it has
avoided a lot of basic concatenation problems over the years (because
Emacs allows sloppy handling of directory file names in most
situations).

--
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no



Reply | Threaded
Open this post in threaded view
|

bug#36490: 26.1; directory-files-recursively breaks when it encounters a directory named "~"

Eli Zaretskii
> From: Lars Ingebrigtsen <[hidden email]>
> Cc: [hidden email],  [hidden email]
> Date: Tue, 09 Jul 2019 19:00:13 +0200
>
> Eli Zaretskii <[hidden email]> writes:
>
> > Sorry, that's not what I meant to say.  I meant to say how would a
> > Lisp program know whether (expand-file-name "~/") means the home
> > directory or a directory whose name is literally "~"?
>
> Well, we have documented that in expand-file-name "~/" means the home
> directory, and I have no problems with that.

Documenting a problem doesn't necessarily solve it.  E.g., it is also
documented that you must quote file names with special characters, but
you still raised the objection that the "~" use case makes that "odd".

I'm saying that the "~/" use case is "odd" as well, and for the same
reasons.

> "~/" isn't something you'll ever get from functions like
> directory-files

That's sheer luck, because:

  (file-name-as-directory "~")
    => "~/"

So just running "~" through an innocent API gives you a "magic"
directory name (if you consider "~" not "magic" by itself).  How is
this different from the "odd" use case where one must quote "~" to
avoid its interpretation as the home directory?  Who can guarantee
that some day directory-files-recursively will not want to do
something like the above?  If it does, we will be right back at the
same problem.

I say we should fix this problem in a way that isn't fragile, and
doesn't crucially depend on what the current code does or avoids
doing.

> > Btw, stuff like (expand-file-name "foo/~/") already does what you
> > want, so the problem is only with the leading '~', and can be avoided
> > if we avoid that situation.  IOW, why should this example:
> >
> >   (expand-file-name "~" "/tmp/")
> >      => "/home/larsi"
> >
> > determine how directory-files-recursively behaves?
>
> expand-file-name's use case is to (basically) concatenate a directory
> name and a file name, but it's used instead of concat because nobody
> wants to care about whether the directory name has a trailing slash or
> not.

Ah, but when the file name begins with a "~", the "concatenation" does
more than what meets the eye.

> That's basically the use case for expand-file-name, and using it has
> avoided a lot of basic concatenation problems over the years (because
> Emacs allows sloppy handling of directory file names in most
> situations).

I think this is a simplification.  It ignores the fact that
expand-file-name interprets ~/, it ignores the fact that it does
arbitrary stuff for "remote" file names, it ignores the fact that on
Windows it prepends the drive letter if there isn't one already, etc.
IOW, expand-file-name is concatenation-like, but it has a few tricks
up its sleeve, and in this case the trick works against us.  We need
to disable that trick to support files and directories whose names
begin with a literal "~".  I see no way around that.



Reply | Threaded
Open this post in threaded view
|

bug#36490: 26.1; directory-files-recursively breaks when it encounters a directory named "~"

Michael Albinus
Eli Zaretskii <[hidden email]> writes:

>> expand-file-name's use case is to (basically) concatenate a directory
>> name and a file name, but it's used instead of concat because nobody
>> wants to care about whether the directory name has a trailing slash or
>> not.
>
> Ah, but when the file name begins with a "~", the "concatenation" does
> more than what meets the eye.
>
>> That's basically the use case for expand-file-name, and using it has
>> avoided a lot of basic concatenation problems over the years (because
>> Emacs allows sloppy handling of directory file names in most
>> situations).
>
> I think this is a simplification.  It ignores the fact that
> expand-file-name interprets ~/, it ignores the fact that it does
> arbitrary stuff for "remote" file names, it ignores the fact that on
> Windows it prepends the drive letter if there isn't one already, etc.
> IOW, expand-file-name is concatenation-like, but it has a few tricks
> up its sleeve, and in this case the trick works against us.  We need
> to disable that trick to support files and directories whose names
> begin with a literal "~".  I see no way around that.

For the records, I second Eli. See also the discussion in bug#16984. And
yes, I believe it makes sense to quote file names (suppress special
meaning of "~") in the loop of directory-files-recursively.

Best regards, Michael.



Reply | Threaded
Open this post in threaded view
|

bug#36490: 26.1; directory-files-recursively breaks when it encounters a directory named "~"

Basil L. Contovounesios
In reply to this post by Eli Zaretskii
Eli Zaretskii <[hidden email]> writes:

>> From: Lars Ingebrigtsen <[hidden email]>
>> Cc: [hidden email],  [hidden email]
>> Date: Tue, 09 Jul 2019 17:50:49 +0200
>>
>> > If you want "~" to be interpreted literally, you need to protect it
>> > with "/:".
>>
>> Sorry; I don't quite follow you here.  The doc string says that "~/" is
>> interpreted specially.  There's no "/" in "~".  :-)
>
> But it does NOT say that "~" will NOT be interpreted specially.

Indeed it explicitly says that "~" will be interpreted specially:

  Second arg DEFAULT-DIRECTORY is directory to start with if NAME is
  relative (does not start with slash or tilde); both the directory name
  and a directory's file name are accepted.

So AIUI (expand-file-name "~") should be equivalent to
(directory-file-name (expand-file-name "~/")).

--
Basil



Reply | Threaded
Open this post in threaded view
|

bug#36490: 26.1; directory-files-recursively breaks when it encounters a directory named "~"

Eli Zaretskii
> From: "Basil L. Contovounesios" <[hidden email]>
> Cc: Lars Ingebrigtsen <[hidden email]>,  [hidden email],  [hidden email]
> Date: Tue, 09 Jul 2019 19:58:18 +0100
>
> So AIUI (expand-file-name "~") should be equivalent to
> (directory-file-name (expand-file-name "~/")).

And that's exactly what it does.



Reply | Threaded
Open this post in threaded view
|

bug#36490: 26.1; directory-files-recursively breaks when it encounters a directory named "~"

Lars Ingebrigtsen
In reply to this post by Eli Zaretskii
Eli Zaretskii <[hidden email]> writes:

>> "~/" isn't something you'll ever get from functions like
>> directory-files
>
> That's sheer luck, because:
>
>   (file-name-as-directory "~")
>     => "~/"
>
> So just running "~" through an innocent API gives you a "magic"
> directory name (if you consider "~" not "magic" by itself).  How is
> this different from the "odd" use case where one must quote "~" to
> avoid its interpretation as the home directory?  Who can guarantee
> that some day directory-files-recursively will not want to do
> something like the above?  If it does, we will be right back at the
> same problem.

Well...  That kinda sounds odd to me.

"~/" is not, and never will be, a valid file name in any OS that Emacs
is going to support from now on.  So having that have a special meaning
in `expand-file-name' is not surprising.  Having "~" do something
special is surprising.

But changing that is probably not going to happen, so how about just
clarifying the documentation in that function to say what "~" means
explicitly instead of the caller having to guess?

--
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no



12