Change in vc-git ls-files-unknown makes emacs hang

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

Change in vc-git ls-files-unknown makes emacs hang

Filipp Gunbin
Hi, this commit makes Emacs hang and become unusable, this happens when
loading the desktop.  Quitting doesn't unfreeze, so I can't get the
backtrace.  I'm on macOS, tty emacs.

commit 1a2445c45f901d9d939f2992dd8e616e5d7cb9ca
Author: Andrii Kolomoiets <[hidden email]>
Date:   Tue Dec 17 01:49:34 2019 +0200
  Make ls-files-unknown only return proper files, not directories

I can't tell the reason of the hang, but reverting it helps.

Andrii, could you please look into it, or revert?

Thanks.
Filipp

Reply | Threaded
Open this post in threaded view
|

Re: Change in vc-git ls-files-unknown makes emacs hang

Dmitry Gutov
Hi Filipp,

On 19.12.2019 16:16, Filipp Gunbin wrote:
> Hi, this commit makes Emacs hang and become unusable, this happens when
> loading the desktop.  Quitting doesn't unfreeze, so I can't get the
> backtrace.  I'm on macOS, tty emacs.

I'm the one who reviewed that patch.

Could you figure out under what circumstances that code
(vc-git-dir-status-goto-stage which triggers the freeze) gets called
while loading the desktop? In particular, for which directory (but which
list of files is passed to vc-git-dir-status-files might also be
important). And after you figure that out, does VC-Dir work okay in it?

Since backtrace is hard to get, how about some print-debugging with
logging to a file? write-region should help.

UUIC, the only case where this change should lead to a performance
problem is a Git repository with lots of unregistered files. Ones not
hidden by gitignore.

Reply | Threaded
Open this post in threaded view
|

Re: Change in vc-git ls-files-unknown makes emacs hang

Filipp Gunbin
Hi Dmitry,

Yes, I caught it, and this:

> UUIC, the only case where this change should lead to a performance
> problem is a Git repository with lots of unregistered files. Ones not
> hidden by gitignore.

is exactly my use case.  I have git repo right in my home dir, mainly
for dot-files.  Most other stuff there is in .gitignore, but there're
some not-ignored directories with large amount of files, like .ivy2 with
cache of jars from ivy repositories.

And vc-dir there (or in a subdirectory) leads to a freeze.

HTH, Filipp

Reply | Threaded
Open this post in threaded view
|

Re: Change in vc-git ls-files-unknown makes emacs hang

Stefan Monnier
In reply to this post by Filipp Gunbin
> Hi, this commit makes Emacs hang and become unusable, this happens when
> loading the desktop.  Quitting doesn't unfreeze, so I can't get the
> backtrace.  I'm on macOS, tty emacs.

kill -USR2 <emacspid>


        Stefan


Reply | Threaded
Open this post in threaded view
|

Re: Change in vc-git ls-files-unknown makes emacs hang

Dmitry Gutov
In reply to this post by Filipp Gunbin
On 19.12.2019 22:44, Filipp Gunbin wrote:

> Yes, I caught it, and this:
>
>> UUIC, the only case where this change should lead to a performance
>> problem is a Git repository with lots of unregistered files. Ones not
>> hidden by gitignore.
>
> is exactly my use case.  I have git repo right in my home dir, mainly
> for dot-files.  Most other stuff there is in .gitignore, but there're
> some not-ignored directories with large amount of files,

Then, first of all, you know the workaround: add some gitignore entries.

> like .ivy2 with
> cache of jars from ivy repositories.

I wonder: how many files?

And it would be helpful, of course, to figure out which code calls it
during desktop-read. It's probably not VC-Dir, or you would have have
noticed it much earlier. Maybe some third-party code?

> And vc-dir there (or in a subdirectory) leads to a freeze.

I'm hesitant to revert it because VC-Dir really doesn't deal with the
previous format very well (meaning, directories with statuses returned
only sometimes).

And before the latest change you couldn't easily register new files from
the VC-Dir  buffer if the directory they are in is also new.

So, opinions welcome.

Reply | Threaded
Open this post in threaded view
|

Re: Change in vc-git ls-files-unknown makes emacs hang

Eli Zaretskii
> From: Dmitry Gutov <[hidden email]>
> Date: Fri, 20 Dec 2019 00:42:24 +0200
>
> So, opinions welcome.

My HO is that we need to understand better which code causes the
freeze, before we can discuss solutions.

Filipp, please produce a backtrace during the freeze, either by using
SIGUSR2, as Stefan suggested, or by attaching a debugger, and post the
backtrace.

Reply | Threaded
Open this post in threaded view
|

Re: Change in vc-git ls-files-unknown makes emacs hang

Filipp Gunbin
On 20/12/2019 09:33 +0200, Eli Zaretskii wrote:

>> From: Dmitry Gutov <[hidden email]>
>> Date: Fri, 20 Dec 2019 00:42:24 +0200
>>
>> So, opinions welcome.
>
> My HO is that we need to understand better which code causes the
> freeze, before we can discuss solutions.
>
> Filipp, please produce a backtrace during the freeze, either by using
> SIGUSR2, as Stefan suggested, or by attaching a debugger, and post the
> backtrace.
Hi, please find the trace in the attach (it's large).

Stefan, thanks for the advice.


trace--vc-git-after-dir-status-stage.txt (14K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Change in vc-git ls-files-unknown makes emacs hang

Dmitry Gutov
On 20.12.2019 12:52, Filipp Gunbin wrote:

>> Filipp, please produce a backtrace during the freeze, either by using
>> SIGUSR2, as Stefan suggested, or by attaching a debugger, and post the
>> backtrace.
>
> Hi, please find the trace in the attach (it's large).

Thank you. It doesn't exactly answer my question about desktop-read (did
you really have an old VC-Dir buffer in that saved session?), but it
shows one bottleneck. One that is even prominently decorated with an old
comment saying "this is VERY inefficient". :-)

Here's a patch you can try. It should help to an extent. And if it's the
main bottleneck, we can optimize further, e.g. by eliminating the
expand-file-name calls.

But if in the end you should see one very long VC-Dir buffer, which can
be a usability problem on its own.

vc-dir-update-faster.diff (1K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Change in vc-git ls-files-unknown makes emacs hang

Filipp Gunbin
Dmitry, thanks, it helps - no freezes and no visible delays now, at
least not yet.

On 20/12/2019 16:13 +0200, Dmitry Gutov wrote:

> Thank you. It doesn't exactly answer my question about desktop-read (did
> you really have an old VC-Dir buffer in that saved session?), but it
> shows one bottleneck. One that is even prominently decorated with an old
> comment saying "this is VERY inefficient". :-)

When I was trying to generate trace, I started & killed (-s USR2) emacs
several times, and saw that desktop had several buffers which could
trigger the problem.

Filipp

Reply | Threaded
Open this post in threaded view
|

Re: Change in vc-git ls-files-unknown makes emacs hang

Dmitry Gutov
On 20.12.2019 17:36, Filipp Gunbin wrote:
> Dmitry, thanks, it helps - no freezes and no visible delays now, at
> least not yet.

Very good.

Does the resulting VC-Dir buffer look usable?

> When I was trying to generate trace, I started & killed (-s USR2) emacs
> several times, and saw that desktop had several buffers which could
> trigger the problem.

So there were several VC-Dir buffers like that in that session?

Reply | Threaded
Open this post in threaded view
|

Re: Change in vc-git ls-files-unknown makes emacs hang

Filipp Gunbin
On 20/12/2019 17:40 +0200, Dmitry Gutov wrote:

> On 20.12.2019 17:36, Filipp Gunbin wrote:
>> Dmitry, thanks, it helps - no freezes and no visible delays now, at
>> least not yet.
>
> Very good.
>
> Does the resulting VC-Dir buffer look usable?

Yes, currently no problems.  I'm in emacs with the patch now, will post
if anything appears in the neartime.

>> When I was trying to generate trace, I started & killed (-s USR2) emacs
>> several times, and saw that desktop had several buffers which could
>> trigger the problem.
>
> So there were several VC-Dir buffers like that in that session?

My feeling was that any dired buffer for a directory underneath home dir
(where .git is) also invoked freeze.

Reply | Threaded
Open this post in threaded view
|

Re: Change in vc-git ls-files-unknown makes emacs hang

Dmitry Gutov
On 20.12.2019 19:14, Filipp Gunbin wrote:

>> Does the resulting VC-Dir buffer look usable?
>
> Yes, currently no problems.  I'm in emacs with the patch now, will post
> if anything appears in the neartime.

Thank you. I've pushed that change as 2c8f153 now.

>> So there were several VC-Dir buffers like that in that session?
>
> My feeling was that any dired buffer for a directory underneath home dir
> (where .git is) also invoked freeze.

If you are using diff-hl-dired-mode, maybe. But it shouldn't suffer from
the same problem.