bug#28139: 25.2; TRAMP vs bash 4.4 / readline 7.0 "bracketed paste" mode

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

bug#28139: 25.2; TRAMP vs bash 4.4 / readline 7.0 "bracketed paste" mode

Daniel Pittman-3
After upgrading a remote system to a version of GNU/Linux that includes bash
4.4 (from Debian/testing), with the readline option `set
enable-bracketed-paste on` in /etc/inputrc, and no personal inputrc,
tramp breaks.

The exact version of `/bin/sh --version` is:
GNU bash, version 4.4.12(1)-release (x86_64-pc-linux-gnu)

Specifically, it breaks because bash emits "enable bracketed paste" and
"disable bracketed paste" CSI control sequences to the terminal around the
shell prompt.

That results in TRAMP trying to read `<ESC> [ 2 0 0 4 h "..."` as a Lisp
expression, when reading the remote `uname -sr` output, which doesn't work.

Passing the `--noediting` option to the remote shell command resolves the
problem, by disabling readline entirely, but it should also be possible to
augment the `tramp-open-connection-setup-interactive-shell` to issue,
unconditionally, the command:

    bind 'set enable-bracketed-paste off'

That will fail on some variants of /bin/sh (dash, posh, tested, but non-GNU
systems may also show this.)

Ignoring the failure should be harmless, however, both when this is not a
valid internal command, and when it is not known to the bash /
readline version.


In GNU Emacs 25.2.1 (x86_64-apple-darwin16.7.0, NS appkit-1504.83 Version 10.12.6 (Build 16G29))
 of 2017-08-08 built on slippycheeze-macbookpro.roam
Windowing system distributor 'Apple', version 10.3.1504
Configured using:
 'configure --disable-dependency-tracking --disable-silent-rules
 --enable-locallisppath=/Users/slippycheeze/homebrew/share/emacs/site-lisp
 --infodir=/Users/slippycheeze/homebrew/Cellar/emacs-edge/25.2/share/info/emacs
 --prefix=/Users/slippycheeze/homebrew/Cellar/emacs-edge/25.2 --with-modules
 --with-rsvg --with-xml2 --without-pop --disable-ns-self-contained --with-ns
 --without-x --without-dbus --with-gnutls'

Configured features:
JPEG RSVG NOTIFY ACL GNUTLS LIBXML2 ZLIB TOOLKIT_SCROLL_BARS NS MODULES

Important settings:
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Emacs-Lisp

Minor modes in effect:
  diff-hl-flydiff-mode: t
  diff-hl-mode: t
  diff-auto-refine-mode: t
  highlight-parentheses-mode: t
  flyspell-mode: t
  ws-butler-mode: t
  shell-dirtrack-mode: t
  global-undo-tree-mode: t
  undo-tree-mode: t
  global-emojify-mode: t
  emojify-mode: t
  async-bytecomp-package-mode: t
  override-global-mode: t
  midnight-mode: t
  global-auto-revert-mode: t
  display-battery-mode: t
  desktop-save-mode: t
  auto-insert-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  column-number-mode: t
  line-number-mode: t
  auto-fill-function: do-auto-fill
  transient-mark-mode: t
  abbrev-mode: t

Recent messages:
Checking 34 files in /Users/slippycheeze/homebrew/Cellar/emacs-edge/25.2/share/emacs/25.2/lisp/emulation...
Checking 167 files in /Users/slippycheeze/homebrew/Cellar/emacs-edge/25.2/share/emacs/25.2/lisp/emacs-lisp...
Checking 24 files in /Users/slippycheeze/homebrew/Cellar/emacs-edge/25.2/share/emacs/25.2/lisp/cedet...
Checking 57 files in /Users/slippycheeze/homebrew/Cellar/emacs-edge/25.2/share/emacs/25.2/lisp/calendar...
Checking 87 files in /Users/slippycheeze/homebrew/Cellar/emacs-edge/25.2/share/emacs/25.2/lisp/calc...
Checking 120 files in /Users/slippycheeze/homebrew/Cellar/emacs-edge/25.2/share/emacs/25.2/lisp/obsolete...
Checking for load-path shadows...done
user-error: End of history; no default available [2 times]

Features:
(shadow sort footnote mail-extr gnus-msg gnus-art mm-uu mml2015 mm-view
mml-smime smime dig mailcap gnus-sum gnus-group gnus-undo gnus-start
gnus-cloud nnimap nnmail mail-source tls gnutls utf7 netrc nnoo parse-time
gnus-spec gnus-int gnus-range gnus-win emacsbug message rfc822 mml mml-sec epg
mm-decode mm-bodies mm-encode mail-parse rfc2231 mailabbrev gmm-utils
mailheader sendmail rfc2047 rfc2045 ietf-drums pp tabify bs executable
tramp-cmds debug tramp-cache misearch multi-isearch find-func cursor-sensor
diff-mode highlight-parentheses flyspell ispell ws-butler jka-compr skeleton
generic-x cc-mode cc-fonts cc-guess cc-menus cc-cmds cc-styles cc-align align
tramp tramp-compat auth-source eieio eieio-core password-cache tramp-loaddefs
delight emojify apropos tar-mode json map subr-x ht dracula-theme edmacro
kmacro async-bytecomp advice async cus-edit memoize dash-functional f dash s
assoc use-package diminish bind-key easy-mmode finder-inf cl-macs cl info
package epg-config seq byte-opt gv bytecomp byte-compile cl-extra cconv
midnight icomplete grep compile comint ansi-color ring gnus gnus-ems nnheader
gnus-util mail-utils mm-util help-fns help-mode easymenu mail-prsvr wid-edit
autorevert filenotify battery desktop frameset autoinsert cus-start cus-load
cl-seq cl-loaddefs pcase cl-lib time-date mule-util tooltip eldoc electric
uniquify ediff-hook vc-hooks lisp-float-type mwheel ns-win ucs-normalize
term/common-win tool-bar dnd fontset image regexp-opt fringe tabulated-list
newcomment elisp-mode lisp-mode prog-mode register page menu-bar rfn-eshadow
timer select scroll-bar mouse jit-lock font-lock syntax facemenu font-core
frame cl-generic cham georgian utf-8-lang misc-lang vietnamese tibetan thai
tai-viet lao korean japanese eucjp-ms cp51932 hebrew greek romanian slovak
czech european ethiopic indian cyrillic chinese charscript case-table epa-hook
jka-cmpr-hook help simple abbrev minibuffer cl-preloaded nadvice loaddefs
button faces cus-face macroexp files text-properties overlay sha1 md5 base64
format env code-pages mule custom widget hashtable-print-readable backquote
kqueue cocoa ns multi-tty make-network-process emacs)

Memory information:
((conses 16 582897 386689)
 (symbols 48 46748 3)
 (miscs 40 565 2646)
 (strings 32 160816 184679)
 (string-bytes 1 4469214)
 (vectors 16 96463)
 (vector-slots 8 3893724 189356)
 (floats 8 11677 1729)
 (intervals 56 6386 3416)
 (buffers 976 39))

Reply | Threaded
Open this post in threaded view
|

bug#28139: 25.2; TRAMP vs bash 4.4 / readline 7.0 "bracketed paste" mode

Michael Albinus
Daniel Pittman <[hidden email]> writes:

Hi Daniel,

> After upgrading a remote system to a version of GNU/Linux that
> includes bash
> 4.4 (from Debian/testing), with the readline option `set
> enable-bracketed-paste on` in /etc/inputrc, and no personal inputrc,
> tramp breaks.

Many different shells are busy to break Tramp.

> Passing the `--noediting` option to the remote shell command resolves
> the problem, by disabling readline entirely, but it should also be
> possible to augment the
> `tramp-open-connection-setup-interactive-shell` to issue,
> unconditionally, the command:
>
>     bind 'set enable-bracketed-paste off'

Does it help to add the following line into your remote ~/.bashrc:

[ $TERM = "dumb" ] && bind 'set enable-bracketed-paste off'

Best regards, Michael.



Reply | Threaded
Open this post in threaded view
|

bug#28139: 25.2; TRAMP vs bash 4.4 / readline 7.0 "bracketed paste" mode

Daniel Pittman-3
On Sat, Aug 19, 2017 at 1:59 PM, Michael Albinus <[hidden email]> wrote:
> After upgrading a remote system to a version of GNU/Linux that
> includes bash
> 4.4 (from Debian/testing), with the readline option `set
> enable-bracketed-paste on` in /etc/inputrc, and no personal inputrc,
> tramp breaks.

Many different shells are busy to break Tramp.

Believe me, I know the pain.  This isn't my first time poking into the internals of how TRAMP hits remote shells.
 
> Passing the `--noediting` option to the remote shell command resolves
> the problem, by disabling readline entirely, but it should also be
> possible to augment the
> `tramp-open-connection-setup-interactive-shell` to issue,
> unconditionally, the command:
>
>     bind 'set enable-bracketed-paste off'

Does it help to add the following line into your remote ~/.bashrc:

[ $TERM = "dumb" ] && bind 'set enable-bracketed-paste off'

Alas, no, and the bash(1) man page "INVOCATION" section makes it clear why:

    Since a shell invoked as sh does not attempt to read and 
    execute commands from any other startup files, 
    the --rcfile option has no effect.

Interactive testing confirms this: if I invoke the shell the as TRAMP does, with:

env ENV='' HISTFILE='' HISTFILESIZE=0 HISTSIZE=0 PROMPT_COMMAND='' PS1=\#\$\  PS2='' PS3='' /bin/sh 

...then strace confirms that no init files are read, other than /etc/inputrc

So, it's not possible to work around this issue without injecting additional code into the shell setup process, using `/bin/sh --noediting`, or using an interactive shell and adding that to the .profile or equivalent.

note: --noediting has been present since bash 2.0 according to their changelog, so it may be reasonable to add by default to the non-interactive shell invocations.

Ultimately, though, perhaps adding the ability to run additional commands during the interactive shell setup process and/or consuming any CSI sequences prior to matching, would be a more future-proof solution to this.
Reply | Threaded
Open this post in threaded view
|

bug#28139: 25.2; TRAMP vs bash 4.4 / readline 7.0 "bracketed paste" mode

Michael Albinus
Daniel Pittman <[hidden email]> writes:

Hi Daniel,

> So, it's not possible to work around this issue without injecting
> additional code into the shell setup process, using `/bin/sh -
> -noediting`, or using an interactive shell and adding that to the .
> profile or equivalent.
>
> note: --noediting has been present since bash 2.0 according to their
> changelog, so it may be reasonable to add by default to the
> non-interactive shell invocations.
>
> Ultimately, though, perhaps adding the ability to run additional
> commands during the interactive shell setup process and/or consuming
> any CSI sequences prior to matching, would be a more future-proof
> solution to this.

Well, we have `tramp-sh-extra-args' for ages. But this doesn't work
usually, because I don't know a bullet-proof way to detect a bash shell
prior invocation, unless "*bash" is called.

Let me contemplate about. With Emacs 26 / Tramp 2.3, we have
connection-local variables. Maybe we could use it for your case.

Best regards, Michael.



Reply | Threaded
Open this post in threaded view
|

bug#28139: 25.2; TRAMP vs bash 4.4 / readline 7.0 "bracketed paste" mode

Daniel Pittman-3
On Mon, Aug 21, 2017 at 11:45 AM, Michael Albinus <[hidden email]> wrote:
Well, we have `tramp-sh-extra-args' for ages. But this doesn't work
usually, because I don't know a bullet-proof way to detect a bash shell
prior invocation, unless "*bash" is called.

*nod*  The best I can think of, since we already have the "outer" shell, would be to invoke `/bin/sh --version -c exit`, which should work on any basic posix shell, and will emit a bash version string for bash, then wait for the prompt, and scan the output for the appropriate version data.

Let me contemplate about. With Emacs 26 / Tramp 2.3, we have
connection-local variables. Maybe we could use it for your case.

FWIW, I already extend `tramp-actions-before-shell`, so something similarly low level would be fine by me, to handle the custom stuff.

However, `ansi-color-filter-region` and friends from the shipped ANSI SGR->face handling library can strip all control sequences out of a region, or string, and might be useful here; that would allow TRAMP to become insensitive to ANSI control codes injected anywhere into the output, which will also fix the next time a shell decides to add some additional terminal integration like bracketed paste mode.

If you want to apply this to streaming output, because that will rewrite the buffer fragment with a smaller number of characters in the presence of control codes, throw a marker at your endpoint, and retrieve the position after the stripping, to avoid getting confused about the new offset.
Reply | Threaded
Open this post in threaded view
|

bug#28139: 25.2; TRAMP vs bash 4.4 / readline 7.0 "bracketed paste" mode

Michael Albinus
Daniel Pittman <[hidden email]> writes:

> However, `ansi-color-filter-region` and friends from the shipped ANSI
> SGR->face handling library can strip all control sequences out of a
> region, or string, and might be useful here; that would allow TRAMP to
> become insensitive to ANSI control codes injected anywhere into the
> output, which will also fix the next time a shell decides to add some
> additional terminal integration like bracketed paste mode.

Tramp does already something like this on its own, using
`tramp-display-escape-sequence-regexp' and
`tramp-device-escape-sequence-regexp'. Let me check, whether I could
rewrite this to the more general `ansi-color-filter-region`.

(I wouldn't reject a patch from your side :-)

Best regards, Michael.



Reply | Threaded
Open this post in threaded view
|

bug#28139: 25.2; TRAMP vs bash 4.4 / readline 7.0 "bracketed paste" mode

Daniel Pittman-3
On Mon, Aug 21, 2017 at 12:17 PM, Michael Albinus <[hidden email]> wrote:
Tramp does already something like this on its own, using
`tramp-display-escape-sequence-regexp' and
`tramp-device-escape-sequence-regexp'. Let me check, whether I could
rewrite this to the more general `ansi-color-filter-region`.

So it does, in versions newer than what is shipped with Emacs.  Is the current release of TRAMP available through any ELPA style package repository, or do I need obtain and compile it myself to use it ahead of the built-in?  I'll certainly do that, but I'm lazy.
 
(I wouldn't reject a patch from your side :-)

If I confirm it still exists in the current version I'll try and cut a patch for this, but my contribution time is limited, I'm afraid.
Reply | Threaded
Open this post in threaded view
|

bug#28139: 25.2; TRAMP vs bash 4.4 / readline 7.0 "bracketed paste" mode

Michael Albinus
Daniel Pittman <[hidden email]> writes:

Hi Daniel,

> Is the current release of TRAMP available through any ELPA style
> package repository, or do I need obtain and compile it myself to use
> it ahead of the built-in?  I'll certainly do that, but I'm lazy.

A while ago, I've started to transform Tramp into an ELPA package. There
were some non-trivial problems, this task is stalled for the moment.

So you must use Tramp's git repository:

git clone git://git.savannah.gnu.org/tramp.git

>     (I wouldn't reject a patch from your side :-)
>
> If I confirm it still exists in the current version I'll try and cut a
> patch for this, but my contribution time is limited, I'm afraid.

Take your time, there's no rush. The next regular Tramp release is
scheduled for end of the year.

If it helps you, I could also add your login to the Tramp repo. You
would need a Savannah account, maybe you have already.

Best regards, Michael.



Reply | Threaded
Open this post in threaded view
|

bug#28139: 25.2; TRAMP vs bash 4.4 / readline 7.0 "bracketed paste" mode

Daniel Pittman-3
On Fri, Aug 25, 2017 at 3:31 AM, Michael Albinus <[hidden email]> wrote:
Daniel Pittman <[hidden email]> writes:
>     (I wouldn't reject a patch from your side :-)
>
> If I confirm it still exists in the current version I'll try and cut a
> patch for this, but my contribution time is limited, I'm afraid.

Take your time, there's no rush. The next regular Tramp release is
scheduled for end of the year.

Please find attached a patch that resolves the bracketed paste issues nicely for me.  It does so by setting two environment variables when initially establishing the remote shell: `EMACS=t`, which bash interprets to disable line editing, and `TERM=dumb`, which is a better match than the default interactive terminal type I get.

Google own the copyright on this change, such as it is, but they have papers on file with the FSF.  (As do I, for TRAMP, though I have not contributed since 2000 or so, haha.)



0001-Inform-shells-to-avoid-fancy-features-more-vigorousl.patch (1K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

bug#28139: 25.2; TRAMP vs bash 4.4 / readline 7.0 "bracketed paste" mode

Michael Albinus
Daniel Pittman <[hidden email]> writes:

Hi Daniel,

> Please find attached a patch that resolves the bracketed paste issues
> nicely for me.

Thanks!

> It does so by setting two environment variables when
> initially establishing the remote shell: `EMACS=t`, which bash
> interprets to disable line editing, and `TERM=dumb`, which is a better
> match than the default interactive terminal type I get.

Well, TERM=dumb setting is not needed. It is already set, inherited from
the outer environment. I've just cross-checked this.

The EMACS=t setting is much more problematic. It has been thrown away
from Emacs years ago, and it was expected that bash 4.4 honors the
replacement INSIDE_EMACS. Furthermore, it was reported that EMACS=t
settings have damaged other packages. See Emacs bug#20484 for a related
discussion.

It doesn't seem to be applicable for Tramp. So I'm kind of undecided how
to continue ...

Best regards, Michael.



Reply | Threaded
Open this post in threaded view
|

bug#28139: 25.2; TRAMP vs bash 4.4 / readline 7.0 "bracketed paste" mode

Daniel Pittman-3
Michael, you are completely right that my previous change was not necessary.  I blame my prehistoric knowledge of Emacs here, as I wasn't aware of the change to use INSIDE_EMACS.  On testing, it seems that EMACS=t just happened to work in bash (for legacy reasons, probably other dinosaurs like me), and INSIDE_EMACS also disables editing, and fixes my problem.

You are also quite correct that INSIDE_EMACS was being set by tramp.  The problem was that it was set at the wrong time: we need it set *before* we start the shell, to disable readline bracketed paste (and other advanced features) before we start, for example, interrogating the remote system for the running version with `uname` during `tramp-open-shell`.

I have attached a patch that moves two variables, TERM and INSIDE_EMACS, out of `tramp-remote-process-environment`, and instead, handles setting them prior to shell execution.

This has two advantages:

First, INSIDE_EMACS is visible to the shell we are about to probe for usability, which fixes my own problem with bracketed paste support in libreadline / bash breaking, and which makes it uniformly available on the remote host.

Second, changes to `tramp-terminal-type` (and the TRAMP version) are now handled correctly without having to recompute the value of `tramp-remote-process-environment`; previously this was not done, and a restart or user intervention would be required to correct them.

Hopefully this meets with approval, so I append both the patch, and what I think is the appropriate changelog entry for this, to be manually added:

2017-11-13  Daniel Pittman  <[hidden email]>

* tramp-sh.el: Set TERM and INSIDE_EMACS environment earlier.
(tramp-remote-process-environment): Remove TERM and INSIDE_EMACS.
(tramp-remote-process-environment): Document their special handling.
(tramp-open-shell): Set TERM and INSIDE_EMACS prior to starting the
initial remote shell, so that it is also aware of the environment
in which it is operating.


emacs-tramp-no-editing.patch (3K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

bug#28139: 25.2; TRAMP vs bash 4.4 / readline 7.0 "bracketed paste" mode

Michael Albinus
Version: 26.1

Daniel Pittman <[hidden email]> writes:

Hi Daniel,

> Hopefully this meets with approval, so I append both the patch, and
> what I think is the appropriate changelog entry for this, to be
> manually added:

Thanks, it looks good. I have applied it to the emacs-26 branch, with
some minor cosmetic changes.

Closing the bug.

Best regards, Michael.