bug#23033: 24.5; Lock file uses the same extension as the file it's locking

classic Classic list List threaded Threaded
17 messages Options
Reply | Threaded
Open this post in threaded view
|

bug#23033: 24.5; Lock file uses the same extension as the file it's locking

Michael Sloan
Lockfiles help the circumstance where multiple emacs processes are
editing the same file. The lockfile for 'File.hs' gets the name
'.#File.hs'.  This means that naive enumeration of all the files in the
directory with the extension '.hs' will also yield the lockfile.  Many
tools have behaviors that rely on enumerating all of the files which
have a particular extension, reasonably assuming that the user put them
there.

In particular, for me this caused the following issue:

It seems wrong up for emacs to be writing files that have the extension
'.cabal' that are not cabal files. Even if they are named pipes that start
with '.', this causes problems for tools that expect files to be what their
name says they are.

Contrast this with backup files, which append a tilda to the end of the
filepath.  This changes the extension, and so tools don't get confused
by the extra files.

Version info:

In GNU Emacs 24.5.1 (x86_64-unknown-linux-gnu, GTK+ Version 3.12.2)
 of 2015-09-10 on computer
Repository revision: 866501efe0fdc0c29448e0aaf8696eb0a3c8fcd6

-Michael
Reply | Threaded
Open this post in threaded view
|

bug#23033: 24.5; Lock file uses the same extension as the file it's locking

Glenn Morris-3
Michael Sloan wrote:

> Lockfiles help the circumstance where multiple emacs processes are
> editing the same file. The lockfile for 'File.hs' gets the name
> '.#File.hs'.  This means that naive enumeration of all the files in the
> directory with the extension '.hs' will also yield the lockfile.  Many
> tools have behaviors that rely on enumerating all of the files which
> have a particular extension, reasonably assuming that the user put them
> there.

It's very long-standing behaviour.
So that we can assess how big the issue is, can you give some examples
of the tools that have issues with this?
As you say, it seems naive for a tool to simply find all files with a
given extension, including dotfiles that are non-existent symlinks.

> In particular, for me this caused the following issue:
> https://github.com/commercialhaskell/stack/issues/1897

Which you fixed the same day in the tool in question, right?



Reply | Threaded
Open this post in threaded view
|

bug#23033: 24.5; Lock file uses the same extension as the file it's locking

Stefan Kangas
Glenn Morris <[hidden email]> writes:

> Michael Sloan wrote:
>
>> Lockfiles help the circumstance where multiple emacs processes are
>> editing the same file. The lockfile for 'File.hs' gets the name
>> '.#File.hs'.  This means that naive enumeration of all the files in the
>> directory with the extension '.hs' will also yield the lockfile.  Many
>> tools have behaviors that rely on enumerating all of the files which
>> have a particular extension, reasonably assuming that the user put them
>> there.
>
> It's very long-standing behaviour.
> So that we can assess how big the issue is, can you give some examples
> of the tools that have issues with this?
> As you say, it seems naive for a tool to simply find all files with a
> given extension, including dotfiles that are non-existent symlinks.
>
>> In particular, for me this caused the following issue:
>> https://github.com/commercialhaskell/stack/issues/1897
>
> Which you fixed the same day in the tool in question, right?

As much as I agree with the general sentiment above, one could also
consider users who are running e.g. 'find -iname "*.hs"' and are not
necessarily interested in seeing Emacs lock files.  IOW, if this is
not a big change, it could be worth doing it.  But it is mostly
cosmetic.

Best regards,
Stefan Kangas



Reply | Threaded
Open this post in threaded view
|

bug#23033: 24.5; Lock file uses the same extension as the file it's locking

Michael Sloan
Correct, it is easy to fix in the tens of thousands of places where it can cause misbehavior :)  But it is probably easier to fix it one place.

Here are some example issues I found searching for "emacs lockfile glob"  yields 22 results - https://github.com/search?q=emacs+lockfile+glob&type=Issues - here are some relevant ones:

https://github.com/joeyespo/pytest-watch/issues/68

Searching for emacs "lockfile" yields 733 results - https://github.com/search?q=emacs+"lockfile"&type=Issues - here are some relevant ones:
-Michael

On Thu, Nov 7, 2019 at 10:00 PM Stefan Kangas <[hidden email]> wrote:
Glenn Morris <[hidden email]> writes:

> Michael Sloan wrote:
>
>> Lockfiles help the circumstance where multiple emacs processes are
>> editing the same file. The lockfile for 'File.hs' gets the name
>> '.#File.hs'.  This means that naive enumeration of all the files in the
>> directory with the extension '.hs' will also yield the lockfile.  Many
>> tools have behaviors that rely on enumerating all of the files which
>> have a particular extension, reasonably assuming that the user put them
>> there.
>
> It's very long-standing behaviour.
> So that we can assess how big the issue is, can you give some examples
> of the tools that have issues with this?
> As you say, it seems naive for a tool to simply find all files with a
> given extension, including dotfiles that are non-existent symlinks.
>
>> In particular, for me this caused the following issue:
>> https://github.com/commercialhaskell/stack/issues/1897
>
> Which you fixed the same day in the tool in question, right?

As much as I agree with the general sentiment above, one could also
consider users who are running e.g. 'find -iname "*.hs"' and are not
necessarily interested in seeing Emacs lock files.  IOW, if this is
not a big change, it could be worth doing it.  But it is mostly
cosmetic.

Best regards,
Stefan Kangas
Reply | Threaded
Open this post in threaded view
|

bug#23033: 24.5; Lock file uses the same extension as the file it's locking

Eli Zaretskii
In reply to this post by Stefan Kangas
> From: Stefan Kangas <[hidden email]>
> Date: Fri, 08 Nov 2019 06:00:14 +0100
> Cc: [hidden email], Michael Sloan <[hidden email]>
>
> As much as I agree with the general sentiment above, one could also
> consider users who are running e.g. 'find -iname "*.hs"' and are not
> necessarily interested in seeing Emacs lock files.  IOW, if this is
> not a big change, it could be worth doing it.

What change did you have in mind?  AFAICT, no specific change was
proposed in that bug report.

The advantage of the current method is that Dired and 'ls' display the
lock file right near the file that is locked, at least on most systems
and with the default file sorting order.  If the change proposal will
order them far apart, it would be a disadvantage.



Reply | Threaded
Open this post in threaded view
|

bug#23033: 24.5; Lock file uses the same extension as the file it's locking

Eli Zaretskii
In reply to this post by Michael Sloan
> From: Michael Sloan <[hidden email]>
> Date: Thu, 7 Nov 2019 23:17:16 -0700
> Cc: [hidden email]
>
> Correct, it is easy to fix in the tens of thousands of places where it can cause misbehavior :)  But it is probably
> easier to fix it one place.

Easier for whom?

These tools all have bugs: they choke on symlinks that point to
non-existent targets.  There could be symlinks like that which have
nothing to do with Emacs's lock files.  So from my POV we did those
tools a favor by exposing their bugs ;-)  I see no reason to sweep
the bugs of those packages under the carpet so as to make it easier
for their developers to keep those bugs.



Reply | Threaded
Open this post in threaded view
|

bug#23033: 24.5; Lock file uses the same extension as the file it's locking

Stefan Kangas
Eli Zaretskii <[hidden email]> writes:

>> From: Michael Sloan <[hidden email]>
>> Date: Thu, 7 Nov 2019 23:17:16 -0700
>> Cc: [hidden email]
>>
>> Correct, it is easy to fix in the tens of thousands of places where it can cause misbehavior :)  But it is probably
>> easier to fix it one place.
>
> Easier for whom?

For humanity, taken as a whole, I think.  :-)

> These tools all have bugs: they choke on symlinks that point to
> non-existent targets.  There could be symlinks like that which have
> nothing to do with Emacs's lock files.  So from my POV we did those
> tools a favor by exposing their bugs ;-)  I see no reason to sweep
> the bugs of those packages under the carpet so as to make it easier
> for their developers to keep those bugs.

I agree with the general sentiment, but consider the amount of bugs
that Michael linked where the name "emacs" crops up.  Appearances
matters, to a certain extent.  To my mind it would be preferable, in
general, if we could avoid having a bunch of bug reports (because of
someone elses sloppiness in this case, yes) where it initially and
incorrectly may look like the culprit is "Emacs and its backwards
practices".

More importantly, I think, is what I mentioned before: this could
inconvenience users trying to run stuff using "find -iname '*.c'" and
the like, only to run into emacs lock files.  I do expect the tools
above to get their house in order and just Fix Their Bugs, but random
users doing one-offs is a different question.

That said, I don't feel super strongly about this.  I consider it a
minor cosmetic blemish.

Best regards,
Stefan Kangas



Reply | Threaded
Open this post in threaded view
|

bug#23033: 24.5; Lock file uses the same extension as the file it's locking

Stefan Kangas
In reply to this post by Eli Zaretskii
Eli Zaretskii <[hidden email]> writes:

>> From: Stefan Kangas <[hidden email]>
>> Date: Fri, 08 Nov 2019 06:00:14 +0100
>> Cc: [hidden email], Michael Sloan <[hidden email]>
>>
>> As much as I agree with the general sentiment above, one could also
>> consider users who are running e.g. 'find -iname "*.hs"' and are not
>> necessarily interested in seeing Emacs lock files.  IOW, if this is
>> not a big change, it could be worth doing it.
>
> What change did you have in mind?  AFAICT, no specific change was
> proposed in that bug report.
>
> The advantage of the current method is that Dired and 'ls' display the
> lock file right near the file that is locked, at least on most systems
> and with the default file sorting order.  If the change proposal will
> order them far apart, it would be a disadvantage.

My suggestion would be to add a one character suffix to the file name.
For example ".", so that:

lrwxrwxrwx  1 skangas skangas     30 2019-11-08 15:06 .#package.el -> skangas@joffe.31542:1572461517

Would instead look like:

lrwxrwxrwx  1 skangas skangas     30 2019-11-08 15:06 .#package.el. -> skangas@joffe.31542:1572461517

That, I think, looks visually not too busy.

(The most clear would be a suffix "-emacs-lock" or "-lock" but that is
probably too ugly and long.)

Best regards,
Stefan Kangas



Reply | Threaded
Open this post in threaded view
|

bug#23033: 24.5; Lock file uses the same extension as the file it's locking

Eli Zaretskii
In reply to this post by Stefan Kangas
> From: Stefan Kangas <[hidden email]>
> Cc: Michael Sloan <[hidden email]>,  [hidden email]
> Date: Fri, 08 Nov 2019 15:03:55 +0100
>
> I agree with the general sentiment, but consider the amount of bugs
> that Michael linked where the name "emacs" crops up.  Appearances
> matters, to a certain extent.  To my mind it would be preferable, in
> general, if we could avoid having a bunch of bug reports (because of
> someone elses sloppiness in this case, yes) where it initially and
> incorrectly may look like the culprit is "Emacs and its backwards
> practices".

Once again, this discussion is quite academic unless someone proposes
a specific change to consider.

Thanks.



Reply | Threaded
Open this post in threaded view
|

bug#23033: 24.5; Lock file uses the same extension as the file it's locking

Eli Zaretskii
In reply to this post by Stefan Kangas
> From: Stefan Kangas <[hidden email]>
> Cc: [hidden email],  [hidden email],  [hidden email]
> Date: Fri, 08 Nov 2019 15:10:50 +0100
>
> My suggestion would be to add a one character suffix to the file name.
> For example "."

This isn't portable enough, it won't work on MS-Windows, where an
empty extension is "normalized" by the file-name APIs in a way that
removes the last dot, so you get the file name without the last dot.



Reply | Threaded
Open this post in threaded view
|

bug#23033: 24.5; Lock file uses the same extension as the file it's locking

Michael Sloan
In reply to this post by Stefan Kangas
On Fri, Nov 8, 2019 at 6:10 AM Stefan Kangas <[hidden email]> wrote:
(The most clear would be a suffix "-emacs-lock" or "-lock" but that is
probably too ugly and long.)

Appending `.lock` seems fine to me (note the period rather than hyphen).  I don't find it ugly, instead explanatory!  If users do see these file names, then it is clearer that they are lock files.  There is precedent for use of this file extension[0], and there's also lots of precedent for stacking file extensions.

I propose keeping the traditional `.#` prefix but adding a `.lock` suffix.

Reply | Threaded
Open this post in threaded view
|

bug#23033: 24.5; Lock file uses the same extension as the file it's locking

Eli Zaretskii
> From: Michael Sloan <[hidden email]>
> Date: Fri, 8 Nov 2019 22:17:29 -0800
> Cc: Eli Zaretskii <[hidden email]>, Glenn Morris <[hidden email]>, [hidden email]
>
> Appending `.lock` seems fine to me (note the period rather than hyphen).  I don't find it ugly, instead
> explanatory!  If users do see these file names, then it is clearer that they are lock files.  There is precedent for
> use of this file extension[0], and there's also lots of precedent for stacking file extensions.
>
> I propose keeping the traditional `.#` prefix but adding a `.lock` suffix.

I think we should only consider adding punctuation characters, because
that would ensure these lock files are displayed right next to the
files they lock, like today.  Moving the lock files away of the files
they lock in the directory listing would be a disadvantage, IMO.



Reply | Threaded
Open this post in threaded view
|

bug#23033: 24.5; Lock file uses the same extension as the file it's locking

Stefan Kangas
Eli Zaretskii <[hidden email]> writes:

> > I propose keeping the traditional `.#` prefix but adding a `.lock` suffix.

OK, if that works on Windows it's better than what I proposed.

> I think we should only consider adding punctuation characters, because
> that would ensure these lock files are displayed right next to the
> files they lock, like today.  Moving the lock files away of the files
> they lock in the directory listing would be a disadvantage, IMO.

On this MacOS machine, I see the following:

$ ls -al
total 8
drwxr-xr-x  11 skangas  staff   352 Nov  9 10:05 .
lrwxr-xr-x   1 skangas  staff    33 Nov  9 10:05 .#foo ->
[hidden email].795
drwxr-xr-x  50 skangas  staff  1600 Nov  9 10:03 ..
-rw-r--r--   1 skangas  staff     0 Nov  9 10:03 a
-rw-r--r--   1 skangas  staff     0 Nov  9 10:03 e
-rw-r--r--   1 skangas  staff     3 Nov  9 10:05 foo
-rw-r--r--   1 skangas  staff     0 Nov  9 10:03 i
-rw-r--r--   1 skangas  staff     0 Nov  9 10:03 z

In other words, the lock file is not next to the file it locks.  Are
you seeing something else?

How would the ordering differ with a suffix like ".lock" compared to
"#" or some other punctuation character?  I would have thought that it
would be very similar.  Maybe I'm missing something.

Best regards,
Stefan Kangas



Reply | Threaded
Open this post in threaded view
|

bug#23033: 24.5; Lock file uses the same extension as the file it's locking

Eli Zaretskii
> From: Stefan Kangas <[hidden email]>
> Date: Sat, 9 Nov 2019 10:12:11 +0100
> Cc: Michael Sloan <[hidden email]>, Glenn Morris <[hidden email]>, [hidden email]
>
> > I think we should only consider adding punctuation characters, because
> > that would ensure these lock files are displayed right next to the
> > files they lock, like today.  Moving the lock files away of the files
> > they lock in the directory listing would be a disadvantage, IMO.
>
> On this MacOS machine, I see the following:
>
> $ ls -al
> total 8
> drwxr-xr-x  11 skangas  staff   352 Nov  9 10:05 .
> lrwxr-xr-x   1 skangas  staff    33 Nov  9 10:05 .#foo ->
> [hidden email].795
> drwxr-xr-x  50 skangas  staff  1600 Nov  9 10:03 ..
> -rw-r--r--   1 skangas  staff     0 Nov  9 10:03 a
> -rw-r--r--   1 skangas  staff     0 Nov  9 10:03 e
> -rw-r--r--   1 skangas  staff     3 Nov  9 10:05 foo
> -rw-r--r--   1 skangas  staff     0 Nov  9 10:03 i
> -rw-r--r--   1 skangas  staff     0 Nov  9 10:03 z
>
> In other words, the lock file is not next to the file it locks.  Are
> you seeing something else?

Yes.

Is the above Gnu 'ls'?  And what is your locale?

> How would the ordering differ with a suffix like ".lock" compared to
> "#" or some other punctuation character?  I would have thought that it
> would be very similar.  Maybe I'm missing something.

The default file sort order in UTF-8 locales ignores punctuation
characters.



Reply | Threaded
Open this post in threaded view
|

bug#23033: 24.5; Lock file uses the same extension as the file it's locking

Michael Sloan
Since sort order is lexicographic, lock files with ".lock" appended will still typically be next to the files when listed.  The only exception I can think of is if there are other related files with the same base name, perhaps with a different stacked extension or suffix (like ~)

It is not the case that the current lock file names are always listed next to the files they are clocking:

mgsloan@treetop:~/test$ ls -la                                                                                                      
total 8                                                                                                                              
drwxrwxr-x  2 mgsloan mgsloan 4096 Nov  9 16:36  .                                                                                  
drwxr-xr-x 57 mgsloan mgsloan 4096 Nov  9 16:35  ..
-rw-rw-r--  1 mgsloan mgsloan    0 Nov  9 16:36  .#test.md
-rw-rw-r--  1 mgsloan mgsloan    0 Nov  9 16:36 '#.test.md'
-rw-rw-r--  1 mgsloan mgsloan    0 Nov  9 16:35  test.md

On Sat, Nov 9, 2019 at 2:24 AM Eli Zaretskii <[hidden email]> wrote:
> From: Stefan Kangas <[hidden email]>
> Date: Sat, 9 Nov 2019 10:12:11 +0100
> Cc: Michael Sloan <[hidden email]>, Glenn Morris <[hidden email]>, [hidden email]
>
> > I think we should only consider adding punctuation characters, because
> > that would ensure these lock files are displayed right next to the
> > files they lock, like today.  Moving the lock files away of the files
> > they lock in the directory listing would be a disadvantage, IMO.
>
> On this MacOS machine, I see the following:
>
> $ ls -al
> total 8
> drwxr-xr-x  11 skangas  staff   352 Nov  9 10:05 .
> lrwxr-xr-x   1 skangas  staff    33 Nov  9 10:05 .#foo ->
> [hidden email].795
> drwxr-xr-x  50 skangas  staff  1600 Nov  9 10:03 ..
> -rw-r--r--   1 skangas  staff     0 Nov  9 10:03 a
> -rw-r--r--   1 skangas  staff     0 Nov  9 10:03 e
> -rw-r--r--   1 skangas  staff     3 Nov  9 10:05 foo
> -rw-r--r--   1 skangas  staff     0 Nov  9 10:03 i
> -rw-r--r--   1 skangas  staff     0 Nov  9 10:03 z
>
> In other words, the lock file is not next to the file it locks.  Are
> you seeing something else?

Yes.

Is the above Gnu 'ls'?  And what is your locale?

> How would the ordering differ with a suffix like ".lock" compared to
> "#" or some other punctuation character?  I would have thought that it
> would be very similar.  Maybe I'm missing something.

The default file sort order in UTF-8 locales ignores punctuation
characters.
Reply | Threaded
Open this post in threaded view
|

bug#23033: 24.5; Lock file uses the same extension as the file it's locking

Eli Zaretskii
> From: Michael Sloan <[hidden email]>
> Date: Sat, 9 Nov 2019 16:37:45 -0700
> Cc: Stefan Kangas <[hidden email]>, Glenn Morris <[hidden email]>, [hidden email]
>
> Since sort order is lexicographic, lock files with ".lock" appended will still typically be next to the files when
> listed.

The sort order is not exactly lexicographic, because it ignores some
characters when comparing strings.

> It is not the case that the current lock file names are always listed next to the files they are clocking:
>
> mgsloan@treetop:~/test$ ls -la                                                                                                      
> total 8                                                                                                                              
> drwxrwxr-x  2 mgsloan mgsloan 4096 Nov  9 16:36  .                                                                                  
> drwxr-xr-x 57 mgsloan mgsloan 4096 Nov  9 16:35  ..
> -rw-rw-r--  1 mgsloan mgsloan    0 Nov  9 16:36  .#test.md
> -rw-rw-r--  1 mgsloan mgsloan    0 Nov  9 16:36 '#.test.md'
> -rw-rw-r--  1 mgsloan mgsloan    0 Nov  9 16:35  test.md

I agree that it is easy to cook an example where this doesn't happen,
but usually it does.



Reply | Threaded
Open this post in threaded view
|

bug#23033: 24.5; Lock file uses the same extension as the file it's locking

Michael Sloan
On Thu, Nov 14, 2019 at 2:18 AM Eli Zaretskii <[hidden email]> wrote:
> It is not the case that the current lock file names are always listed next to the files they are clocking:
>
> mgsloan@treetop:~/test$ ls -la                                                                                                       
> total 8                                                                                                                             
> drwxrwxr-x  2 mgsloan mgsloan 4096 Nov  9 16:36  .                                                                                   
> drwxr-xr-x 57 mgsloan mgsloan 4096 Nov  9 16:35  ..
> -rw-rw-r--  1 mgsloan mgsloan    0 Nov  9 16:36  .#test.md
> -rw-rw-r--  1 mgsloan mgsloan    0 Nov  9 16:36 '#.test.md'
> -rw-rw-r--  1 mgsloan mgsloan    0 Nov  9 16:35  test.md

I agree that it is easy to cook an example where this doesn't happen,
but usually it does.

Indeed!  And appending ".lock" to a file name is also typically adjacent in listings, and it is easy to cook an example where it is not.