Introducing thread-safe Tramp

classic Classic list List threaded Threaded
124 messages Options
1234 ... 7
Reply | Threaded
Open this post in threaded view
|

Introducing thread-safe Tramp

Michael Albinus
Hi,

I have created a new branch feature/tramp-thread-safe. The major change
is making Tramp thread-safe, that means, several basic file operations
can run concurrently now.

Being a library, Tramp does not create new threads on its own. This must
be performed by the callers.

I have changed the `find-file' family of commands to raise their
underlying file operations asynchronously if indicated. Every visiting
of a file happens in its own thread then. If wildcards are used, one
thread per involved file is created.

In order to enable this, the respective command must be called with a
prefix argument, like "C-u C-x C-f ...".  Emacs will remain responsive,
you can continue with whatever operation. This will be best seen with
remote files, because they need more time to load into a buffer. Emacs
is not as responsive yet as I hoped to achieve; further optimization is
needed.

As second change, I have modified `vc-refresh-state' to run always in
its own thread. This is because I have observed, that for git
repositories, loading a remote file is often much faster than the
following update of its state in the modeline. For the time being this
behaviour is hard-coded, I haven't seen any drawback for this even for
local files. But we could trigger it also via a user option.

@Dmitry?

Please give it as much testing as possible. I cannot promise Emacs won't
crash ever with these changes, so it is not time for production
systems. But all bug reports and feedbacks will help to improve this new
feature.

And, if there are not too serious complaints, I will merge it into
master after a while :-) You are warned!

Best regards, Michael.

Reply | Threaded
Open this post in threaded view
|

Re: Introducing thread-safe Tramp

Eli Zaretskii
> From: Michael Albinus <[hidden email]>
> Date: Mon, 23 Jul 2018 17:58:39 +0200
> Cc: Dmitry Gutov <[hidden email]>
>
> I have created a new branch feature/tramp-thread-safe. The major change
> is making Tramp thread-safe, that means, several basic file operations
> can run concurrently now.

Thank you!

> I have changed the `find-file' family of commands to raise their
> underlying file operations asynchronously if indicated. Every visiting
> of a file happens in its own thread then. If wildcards are used, one
> thread per involved file is created.
>
> In order to enable this, the respective command must be called with a
> prefix argument, like "C-u C-x C-f ...".  Emacs will remain responsive,
> you can continue with whatever operation.

What happens if find-file or its subroutine ask a question that the
user must answer?

Reply | Threaded
Open this post in threaded view
|

Re: Introducing thread-safe Tramp

Michael Albinus
Eli Zaretskii <[hidden email]> writes:

>> In order to enable this, the respective command must be called with a
>> prefix argument, like "C-u C-x C-f ...".  Emacs will remain responsive,
>> you can continue with whatever operation.
>
> What happens if find-file or its subroutine ask a question that the
> user must answer?

It's presented in the minibuffer, and you can answer. But perhaps, it is
hidden by other messages. One of my open points.

"Release early. Release often." (ESR)

Best regards, Michael.

Reply | Threaded
Open this post in threaded view
|

Re: Introducing thread-safe Tramp

Eli Zaretskii
> From: Michael Albinus <[hidden email]>
> Cc: [hidden email],  [hidden email]
> Date: Mon, 23 Jul 2018 18:11:59 +0200
>
> > What happens if find-file or its subroutine ask a question that the
> > user must answer?
>
> It's presented in the minibuffer, and you can answer. But perhaps, it is
> hidden by other messages. One of my open points.

What bothers me more is that it sounds like you trigger redisplay from
a non-main thread (because 'message' triggers a special type of
redisplay).  Moreover, what happens if the main thread is in the
middle of prompting the user by a command that runs in the main
thread?

I suggest to test these situations and see what happens and whether
something (perhaps some infrastructure) is needed to DTRT in those
cases.

P.S. I'm glad to finally see a major Emacs package going threaded.

Reply | Threaded
Open this post in threaded view
|

Re: Introducing thread-safe Tramp

Michael Albinus
Eli Zaretskii <[hidden email]> writes:

> What bothers me more is that it sounds like you trigger redisplay from
> a non-main thread (because 'message' triggers a special type of
> redisplay).  Moreover, what happens if the main thread is in the
> middle of prompting the user by a command that runs in the main
> thread?

I believe that message and other minibuffer related functions need to
check whether they are in the main thread. If not, they need a changed
behaviour.

> I suggest to test these situations and see what happens and whether
> something (perhaps some infrastructure) is needed to DTRT in those
> cases.

Likely yes.

Best regards, Michael.

Reply | Threaded
Open this post in threaded view
|

Re: Introducing thread-safe Tramp

Ken Raeburn-2
In reply to this post by Michael Albinus
On Jul 23, 2018, at 11:58, Michael Albinus <[hidden email]> wrote:
>
> Hi,
>
> I have created a new branch feature/tramp-thread-safe. The major change
> is making Tramp thread-safe, that means, several basic file operations
> can run concurrently now.

This sounds great!
I haven’t had much time for Emacs work recently but I’ll try to try this out soon.

The changes you describe to the find-file interfaces relate to another idea I’ve been thinking of for a while, though I wanted to wait until I had time to look into it a bit more myself. Namely, make some of the C-level local file system interactions release the global lock, and use additional threads for invoking some of those operations.

So, for example, while loading a large file from a slow NFS server, we can also be processing subprocess output, running garbage collection, etc. If the NFS server hangs, ^G should interrupt the main UI thread that’s sitting waiting for a notification from the file I/O thread (which is hung until the server comes back or the access times out; however, other file I/O threads could be used to access other files or file systems), break it out of the wait, and let the user go do something else. In auto-save-mode or auto-revert-mode, read/write/stat calls shouldn’t cause delays in the UI just because a file server is slow.

There are plenty of details I haven’t worked out, like what to do about these background threads acting on a buffer the user is actively modifying at the same time. But one thing that had occurred to me was that as far as the user interaction is concerned, a multithreaded Tramp would be just the way to try out some of these sorts of changes.

Ken
Reply | Threaded
Open this post in threaded view
|

Re: Introducing thread-safe Tramp

Michael Albinus
Ken Raeburn <[hidden email]> writes:

Hi Ken,

> The changes you describe to the find-file interfaces relate to another
> idea I’ve been thinking of for a while, though I wanted to wait until
> I had time to look into it a bit more myself. Namely, make some of the
> C-level local file system interactions release the global lock, and
> use additional threads for invoking some of those operations.

Yes. But it might be better to use an own mutex for that.

In Tramp, I have introduced one mutex per connection. Whenever any Tramp
file name operation is called, it goes via `tramp-file-name-handler'. Here
I determine the responsible mutex (according to the connection
characteristics), and I lock this mutex. Once the operation has
finished, at the end of `tramp-file-name-handler', the mutex is
unlocked. (To be honest, I simply use `with-mutex'.)

By this, only one file operation per connection can run at a given
time. This is still better than w/o threads, because if you access
concurrently "/ssh:..." and "/sudo:...", the operations could run in
parallel. And, due to the threads created by find-file, Tramp operations
do not block global editing functions.

Of course, this is a coarse approach. On my todo list is a change, to
apply conection-oriented mutexes not on the file operations level, but
on process related level (sending commands remotely, and retrieve the
output). By this, more concurrency shall be possible, with (hopefully)
better performance. Similar to what you have in mind.

> So, for example, while loading a large file from a slow NFS server, we
> can also be processing subprocess output, running garbage collection,
> etc. If the NFS server hangs, ^G should interrupt the main UI thread
> that’s sitting waiting for a notification from the file I/O thread
> (which is hung until the server comes back or the access times out;
> however, other file I/O threads could be used to access other files or
> file systems), break it out of the wait, and let the user go do
> something else. In auto-save-mode or auto-revert-mode, read/write/stat
> calls shouldn’t cause delays in the UI just because a file server is
> slow.

Same would be true for Tramp then.

> There are plenty of details I haven’t worked out, like what to do
> about these background threads acting on a buffer the user is actively
> modifying at the same time. But one thing that had occurred to me was
> that as far as the user interaction is concerned, a multithreaded
> Tramp would be just the way to try out some of these sorts of changes.

Yep. Currently I'm sitting on the problem of handling the minibuffer
properly, when several find-file operations are in progress in
parallel. Something like confirmation of risky local variables in a
file, and alike.

Another problem will be cached values. Tramp caches a lot of file
information, in order to reduce roundtrips. Those cached values might
become invalid due to operations in another thread.

So there's still a lot to do. I hope to get much feedback from you and
other people.

> Ken

Best regards, Michael.

Reply | Threaded
Open this post in threaded view
|

Re: Introducing thread-safe Tramp

martin rudalics
 > Yep. Currently I'm sitting on the problem of handling the minibuffer
 > properly, when several find-file operations are in progress in
 > parallel. Something like confirmation of risky local variables in a
 > file, and alike.

I would try mapping each operation to a separate minibuffer-only
frame.

martin

Reply | Threaded
Open this post in threaded view
|

RE: Introducing thread-safe Tramp

Drew Adams
In reply to this post by Michael Albinus
This sounds like a good feature.  I haven't looked at
the code, so I don't really know how it's implemented.
I have one objection to what I think you're proposing
for the design, so far at least.

It's true that `find-file' etc. are commands.  They
are also functions that users sometimes call from Lisp.

My a priori objection is the use of a prefix arg to
indicate that you want a separate thread.  I would
prefer that users specify this intention in some other
way, and that we reserve the use of a prefix arg for
something else (future).

How else might a user specify use of a separate thread?
Possibilities include:

* Use a different function/command.  This is the
  usual way Emacs handles such things: new function.
  It's what we do for the choice of same/other window,
  for example - we have two commands, which can be
  bound to two different keys.

* Bind a defvar variable.  This is what we do for
  respect of `find-file-wildcards', for example.

I'd be OK with either of those approaches, and perhaps
with others.  I doubt that I can be in favor of the
prefix-arg approach.

If it were I, I would probably do something like this:

1. Add another optional arg (Boolean), to determine
   the behavior.  I haven't seen your code, but this
   is probably what you've done already.

2. In the body, use that optional arg to override a
   defvar variable, which otherwise determines the
   behavior (choice).

3. Define additional commands for convenience, which
   pass the optional arg (non-nil) to give the
   separate-thread behavior.

Any user who wants to combine such a separate command
with the corresponding usual command, and who wants
to use a prefix arg to make the choice, can easily do
so.  But Emacs itself will have reserved the ability
to use a prefix arg for something else, and it will
have provided new commands that users can bind to
other keys.

Just one opinion, and open to change.  Thanks again
for this feature.  Let me know, if I misunderstand
something.

Reply | Threaded
Open this post in threaded view
|

Re: Introducing thread-safe Tramp

Michael Albinus
Drew Adams <[hidden email]> writes:

Hi Drew,

> My a priori objection is the use of a prefix arg to
> indicate that you want a separate thread.  I would
> prefer that users specify this intention in some other
> way, and that we reserve the use of a prefix arg for
> something else (future).

I don't understand your reservation. 40+ years of Emacs, and there was
no need yet for a prefix argument of find-file. I would be surprised if
it happens next time.

Running the commands asynchronously is very tight bound to the usual
invocation of the commands, so I don't see that something else, which
might asks for a prefix argument of the commands, will be better suited.

> How else might a user specify use of a separate thread?
> Possibilities include:
>
> * Use a different function/command.  This is the
>   usual way Emacs handles such things: new function.
>   It's what we do for the choice of same/other window,
>   for example - we have two commands, which can be
>   bound to two different keys.

It isn't about command names. It is about key bindings. We have all
these bindings 'C-x C-f', 'C-x C-r', 'C-x C-v', 'C-x 4 f', 'C-x 5 f'.
If we don't allow a prefix argument, we need complete new bindings. My
memory muscle tells me I'm too old to learn them.

> * Bind a defvar variable.  This is what we do for
>   respect of `find-file-wildcards', for example.

That's a good idea to visit files asynchronously by default. Maybe we
shall add such user option.

But I don't believe people will always visit files asynchronously. They
will do it for remote files, they will do it for huge files. But for my
daily work with local small files, this would be overkill.

> I'd be OK with either of those approaches, and perhaps
> with others.  I doubt that I can be in favor of the
> prefix-arg approach.
>
> If it were I, I would probably do something like this:
>
> 1. Add another optional arg (Boolean), to determine
>    the behavior.  I haven't seen your code, but this
>    is probably what you've done already.

All these functions have a new optional argument `threads'.

> 2. In the body, use that optional arg to override a
>    defvar variable, which otherwise determines the
>    behavior (choice).
>
> 3. Define additional commands for convenience, which
>    pass the optional arg (non-nil) to give the
>    separate-thread behavior.

These are not needed I believe for the reasons above.

> Any user who wants to combine such a separate command
> with the corresponding usual command, and who wants
> to use a prefix arg to make the choice, can easily do
> so.  But Emacs itself will have reserved the ability
> to use a prefix arg for something else, and it will
> have provided new commands that users can bind to
> other keys.

Again, this is based on the assumption we'll need prefix arguments for
something else. I doubt, "something else" will hit us, and if it
happens, that it is better suited to occupy the prefix arg.

> Just one opinion, and open to change.  Thanks again
> for this feature.  Let me know, if I misunderstand
> something.

Same for me. Just my opinion. Let's see what other people say, and what
the maintainers decide.

Best regards, Michael.

Reply | Threaded
Open this post in threaded view
|

Re: Introducing thread-safe Tramp

Michael Albinus
In reply to this post by martin rudalics
martin rudalics <[hidden email]> writes:

Hi Martin,

>> Yep. Currently I'm sitting on the problem of handling the minibuffer
>> properly, when several find-file operations are in progress in
>> parallel. Something like confirmation of risky local variables in a
>> file, and alike.
>
> I would try mapping each operation to a separate minibuffer-only
> frame.

Might work. But threads are not bound to frames, and I doubt that it
shall be the general solution. What about Emacs in a terminal window,
w/o frames?

I believe we'd rather go into locking the minibuffer / echo area while
reading. Somewhere in the read_minibuf function of minibuf.c. But I
couldn't nail it down until now.

> martin

Best regards, Michael.

Reply | Threaded
Open this post in threaded view
|

Re: Introducing thread-safe Tramp

Phil Sainty
In reply to this post by Michael Albinus
On 25/07/18 21:46, Michael Albinus wrote:
> Drew Adams <[hidden email]> writes:
>> My a priori objection is the use of a prefix arg to indicate that
>> you want a separate thread.  I would prefer that users specify this
>> intention in some other way, and that we reserve the use of a
>> prefix arg for something else (future).
>
> I don't understand your reservation. 40+ years of Emacs, and there
> was no need yet for a prefix argument of find-file. I would be
> surprised if it happens next time.

At first glance I would lean towards Drew's viewpoint on the basis
that it seems to me that there may be a benefit in having a *common*
interface for saying "do a thing asynchronously" -- for all manner of
things which support that (now or in the future).  find-file may never
have acquired a prefix arg, but other commands which may support
asynchronous execution in the future might already use prefix args,
in which case there would end up being inconsistencies in how the
user invoked async behaviour for any given command.

The analog that springs to my mind is the command `append-next-kill'
(bound to C-M-w), which can be invoked in order to "Cause following
command, if it kills, to add to previous kill."

Acting asynchronously might be such a generic requirement that maybe
it makes sense to implement some similar facility to say "Cause the
following command, if it can act asynchronously, to do so."

We might have a user option meaning "commands should act asynchronously
if possible" which, at least in the early days, might be set to nil by
default; but invoking the "run the next command asynchronously" command
could bind that variable as non-nil for the duration of the following
command.

Then if it was a convention that functions implementing optional async
behaviour did so conditional on this variable, we could gain the
desired outcome not only for `find-file' but for anything else in the
future which implemented such support.

Individual libraries might implement their own additional user options
to enable users to specify default async behaviour for particular sets
of commands implemented by the library in question (e.g. `find-file'
and friends), and/or users could use advice to bind the "use async
behaviour" variable for specific commands if they wanted it for some
things but not others.

Ultimately I can envisage that async-supporting commands which are
initially synchronous-by-default may in the future end up being
asynchronous-by-default once this functionality is better established,
and so tying prefix arguments to such functionality now may end up
seeming short-sighted in the future?


-Phil


Reply | Threaded
Open this post in threaded view
|

Re: Introducing thread-safe Tramp

Andreas Schwab
On Jul 25 2018, Phil Sainty <[hidden email]> wrote:

> The analog that springs to my mind is the command `append-next-kill'
> (bound to C-M-w), which can be invoked in order to "Cause following
> command, if it kills, to add to previous kill."

Or like C-x C-m c (universal-coding-system-argument).

Andreas.

--
Andreas Schwab, SUSE Labs, [hidden email]
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

Reply | Threaded
Open this post in threaded view
|

Re: Introducing thread-safe Tramp

Michael Albinus
In reply to this post by Phil Sainty
Phil Sainty <[hidden email]> writes:

Hi Phil,

> Ultimately I can envisage that async-supporting commands which are
> initially synchronous-by-default may in the future end up being
> asynchronous-by-default once this functionality is better established,
> and so tying prefix arguments to such functionality now may end up
> seeming short-sighted in the future?

You have a point. But I still believe that synchronous visiting of files
will still be used, especially for small local files. Se we could have a
user option which toggles synchronous-by-default or asynchronous-by-default,
and we should have an indication do-it-the-other-way. Then people could
decide their default, and they could overwrite this default for single files.

The variable I could add easily. But which general indication do we want
to use, if it isn't prefix arg?

> -Phil

Best regards, Michael.

Reply | Threaded
Open this post in threaded view
|

Re: Introducing thread-safe Tramp

Robert Pluim
Michael Albinus <[hidden email]> writes:

> Phil Sainty <[hidden email]> writes:
>
> Hi Phil,
>
>> Ultimately I can envisage that async-supporting commands which are
>> initially synchronous-by-default may in the future end up being
>> asynchronous-by-default once this functionality is better established,
>> and so tying prefix arguments to such functionality now may end up
>> seeming short-sighted in the future?
>
> You have a point. But I still believe that synchronous visiting of files
> will still be used, especially for small local files. Se we could have a
> user option which toggles synchronous-by-default or asynchronous-by-default,
> and we should have an indication do-it-the-other-way. Then people could
> decide their default, and they could overwrite this default for single files.
>
> The variable I could add easily. But which general indication do we want
> to use, if it isn't prefix arg?

Given that this is all highly experimental, I wouldn't mind a 'be 100%
asynchronous' variable in order to ensure maximal coverage (plus I
donʼt want the cognitive load of having to think "do I want this
operation to be asynchronous"). Once we have more experience with that
we can decide which if any operations should default to asynchronous.

Robert

Reply | Threaded
Open this post in threaded view
|

Re: Introducing thread-safe Tramp

Eli Zaretskii
In reply to this post by Michael Albinus
> From: Michael Albinus <[hidden email]>
> Date: Wed, 25 Jul 2018 11:51:49 +0200
> Cc: Ken Raeburn <[hidden email]>, [hidden email],
> Dmitry Gutov <[hidden email]>
>
> I believe we'd rather go into locking the minibuffer / echo area while
> reading.

What do you mean by "locking" in this context?  Ask a thread to take a
mutex before it can use the minibuffer?

Reply | Threaded
Open this post in threaded view
|

Re: Introducing thread-safe Tramp

Eli Zaretskii
> Date: Wed, 25 Jul 2018 17:44:58 +0300
> From: Eli Zaretskii <[hidden email]>
> Cc: [hidden email], [hidden email], [hidden email], [hidden email]
>
> > From: Michael Albinus <[hidden email]>
> > Date: Wed, 25 Jul 2018 11:51:49 +0200
> > Cc: Ken Raeburn <[hidden email]>, [hidden email],
> > Dmitry Gutov <[hidden email]>
> >
> > I believe we'd rather go into locking the minibuffer / echo area while
> > reading.
>
> What do you mean by "locking" in this context?  Ask a thread to take a
> mutex before it can use the minibuffer?

For the messages part, how about if we create a buffer named
*Messages-for-threadHHHHH*, where HHHHH is the thread ID, and put the
echo-area messages from that thread in that buffer?  We could later
invent a machinery to display that buffer automatically when some
event happens, like when the thread exits?

We could do something similar with echo-area buffers: create
additional buffers when threads are created.

WDYT?

Reply | Threaded
Open this post in threaded view
|

Re: Introducing thread-safe Tramp

Michael Albinus
Eli Zaretskii <[hidden email]> writes:

>> > I believe we'd rather go into locking the minibuffer / echo area while
>> > reading.
>>
>> What do you mean by "locking" in this context?  Ask a thread to take a
>> mutex before it can use the minibuffer?

Yes, I believe that's what we shall do when input is read from the
minibuffer. There shall be one common minibuffer mutex, and it is locked
when input has to be dealt from the minibuffer. Something like the
appended patch, but it doesn't work yet. Likely, due to the messages
arriving the echo area in parallel.

> For the messages part, how about if we create a buffer named
> *Messages-for-threadHHHHH*, where HHHHH is the thread ID, and put the
> echo-area messages from that thread in that buffer?  We could later
> invent a machinery to display that buffer automatically when some
> event happens, like when the thread exits?
>
> We could do something similar with echo-area buffers: create
> additional buffers when threads are created.
>
> WDYT?
Could work. In general, I don't believe it is a problem when messages
arrive the echo area from different threads in parallel. The problem is
when they disturb input reading. While input is taken, messages must be
blocked writing to the minibuffer being their echo area. With the
proposal you have done, or looking for the mutex, or something else.

Best regards, Michael.


attachment0 (1K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Introducing thread-safe Tramp

martin rudalics
In reply to this post by Eli Zaretskii
 > For the messages part, how about if we create a buffer named
 > *Messages-for-threadHHHHH*, where HHHHH is the thread ID, and put the
 > echo-area messages from that thread in that buffer?  We could later
 > invent a machinery to display that buffer automatically when some
 > event happens, like when the thread exits?
 >
 > We could do something similar with echo-area buffers: create
                                      ^^^^^^^^^^
Did you mean 'minibuffer' here?

martin

Reply | Threaded
Open this post in threaded view
|

Re: Introducing thread-safe Tramp

Michael Albinus
martin rudalics <[hidden email]> writes:

>> For the messages part, how about if we create a buffer named
>> *Messages-for-threadHHHHH*, where HHHHH is the thread ID, and put the
>> echo-area messages from that thread in that buffer?  We could later
>> invent a machinery to display that buffer automatically when some
>> event happens, like when the thread exits?
>>
>> We could do something similar with echo-area buffers: create
>                                      ^^^^^^^^^^
> Did you mean 'minibuffer' here?

Yes, that's the minibuffer. I should have said "echo area" only; I've
meant messages sent to the echo area and appearing in the minibuffer.

> martin

Best regards, Michael.

1234 ... 7