Copy-paste of large files from Windows to Linux - base64 issue?

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Copy-paste of large files from Windows to Linux - base64 issue?

Guillaume Demeyère
Hi all,

I'm encountering a new issue on transfering files from my Windows 10 machine to a Linux server.

Here's what I do (in file tramp-debug-with-encoding.log) :

1. Launch emacs with :
C:\Users\gde3\Documents\emacs-26.2-x86_64\bin\runemacs -Q -l tramp -l tramp-sh
2. Execute the following:
(setq tramp-verbose 10)
 (copy-file "c:/Users/gde3/Documents/GDE3-Widgets/widget-template-vue/dist/1.bundle.js" "/plink:nxuser@vdemopro892dsy:~/Documents/")

The copy fails with the base64 decoding failing.

I thought this could come from the bzip2 encoding from Windows to Linux. However, I think I eliminated this hypothesis by 1) bzipping, transferring and un-bzipping manually this very file over MobaXterm succesfully and 2) by completely deactivating compression with (setq tramp-inline-compress-start-size nil) and redoing the test. The transfer fails with the same error (file tramp-debug-without-encoding.log).

Do you have any idea what I should do next to try and resolve this issue?

The file 1.bundle.js is a 20MB minimified JS file.

Regards,

Guillaume



tramp-debug-without-encoding.log (185K) Download Attachment
tramp-debug-with-encoding.log (240K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Copy-paste of large files from Windows to Linux - base64 issue?

Michael Albinus
Guillaume Demeyère <[hidden email]> writes:

> Hi all,

Hi Guillaume

> I'm encountering a new issue on transfering files from my Windows 10
> machine to a Linux server.
>
> Here's what I do (in file tramp-debug-with-encoding.log) :
>
> 1. Launch emacs with :
> C:\Users\gde3\Documents\emacs-26.2-x86_64\bin\runemacs -Q -l tramp -l
> tramp-sh
> 2. Execute the following:
> (setq tramp-verbose 10)
>  (copy-file
> "c:/Users/gde3/Documents/GDE3-Widgets/widget-template-vue/dist/1.bundle.js"
> "/plink:nxuser@vdemopro892dsy:~/Documents/")
>
> The copy fails with the base64 decoding failing.
>
> I thought this could come from the bzip2 encoding from Windows to
> Linux. However, I think I eliminated this hypothesis by 1) bzipping,
> transferring and un-bzipping manually this very file over MobaXterm
> succesfully and 2) by completely deactivating compression with (setq
> tramp-inline-compress-start-size nil) and redoing the test. The
> transfer fails with the same error (file
> tramp-debug-without-encoding.log).

According to the logs, it looks like the same problem with and without
compressing. So I use just one trace file.

> 08:22:30.194971 tramp-send-command (6) # ( echo xyzzy | base64 | base64 -d -i 2>/dev/null; echo tramp_exit_status $? )
> 08:22:30.248779 tramp-wait-for-regexp (6) #
> xyzzy
> tramp_exit_status 0
> ///2e27906ce8c828244dca14f15b2baca4#$

This shows, that base64 works in general.

> 08:22:30.352218 tramp-send-command (6) # base64 -d -i >/home/nxuser/Documents/1.bundle.js <<'58274261e8a0f9f17931998a89b9c1fb'
>
> [HUNDREDS OF LINES OF BASE64 DATA]
>
> 08:22:30.422526 tramp-get-connection-property (7) # process-name nil
> 08:22:30.422589 tramp-get-connection-property (7) # chunksize 0
> 08:22:30.422631 tramp-set-connection-property (7) # last-cmd-time (24092 6838 422621 0)
> 08:22:30.422677 tramp-send-string (10) # base64 -d -i >/home/nxuser/Documents/1.bundle.js <<'58274261e8a0f9f17931998a89b9c1fb'
>
> [HUNDREDS OF LINE OF BASE64 DATA]
>
> 08:22:30.498776 tramp-get-connection-property (7) # process-buffer nil
> 08:22:31.368697 tramp-sh-handle-write-region (3) # Decoding remote file ‘/plink:nxuser@vdemopro892dsy:/home/nxuser/Documents/1.bundle.js’ using ‘base64 -d -i >%s’...failed
> 08:22:31.371330 tramp-do-copy-or-rename-file (0) # Copying c:/Users/gde3/Documents/GDE3-Widgets/widget-template-vue/dist/1.bundle.js to /plink:nxuser@vdemopro892dsy:/home/nxuser/Documents/1.bundle.js...failed

I miss several trace lines after your second [HUNDREDS OF LINES OF BASE64
DATA]. Is there really nothing else between the BASE64 DATA and the last
three lines you have shown in the trace?

Maybe you could show *everything*, starting with the last 5 BASE64 DATA lines?

> Regards,
>
> Guillaume

Best regards, Michael.

Reply | Threaded
Open this post in threaded view
|

Re: Copy-paste of large files from Windows to Linux - base64 issue?

Guillaume Demeyère
Hi Michael,

I really think that I edited the file correctly, in order to make it less huge.

Here's an unedited version (5.3 MB) run this morning, so you can make sure I did.

Best regards,

Guillaume


Michael Albinus <[hidden email]> escreveu no dia segunda, 13/01/2020 à(s) 16:25:
Guillaume Demeyère <[hidden email]> writes:

> Hi all,

Hi Guillaume

> I'm encountering a new issue on transfering files from my Windows 10
> machine to a Linux server.
>
> Here's what I do (in file tramp-debug-with-encoding.log) :
>
> 1. Launch emacs with :
> C:\Users\gde3\Documents\emacs-26.2-x86_64\bin\runemacs -Q -l tramp -l
> tramp-sh
> 2. Execute the following:
> (setq tramp-verbose 10)
>  (copy-file
> "c:/Users/gde3/Documents/GDE3-Widgets/widget-template-vue/dist/1.bundle.js"
> "/plink:nxuser@vdemopro892dsy:~/Documents/")
>
> The copy fails with the base64 decoding failing.
>
> I thought this could come from the bzip2 encoding from Windows to
> Linux. However, I think I eliminated this hypothesis by 1) bzipping,
> transferring and un-bzipping manually this very file over MobaXterm
> succesfully and 2) by completely deactivating compression with (setq
> tramp-inline-compress-start-size nil) and redoing the test. The
> transfer fails with the same error (file
> tramp-debug-without-encoding.log).

According to the logs, it looks like the same problem with and without
compressing. So I use just one trace file.

> 08:22:30.194971 tramp-send-command (6) # ( echo xyzzy | base64 | base64 -d -i 2>/dev/null; echo tramp_exit_status $? )
> 08:22:30.248779 tramp-wait-for-regexp (6) #
> xyzzy
> tramp_exit_status 0
> ///2e27906ce8c828244dca14f15b2baca4#$

This shows, that base64 works in general.

> 08:22:30.352218 tramp-send-command (6) # base64 -d -i >/home/nxuser/Documents/1.bundle.js <<'58274261e8a0f9f17931998a89b9c1fb'
>
> [HUNDREDS OF LINES OF BASE64 DATA]
>
> 08:22:30.422526 tramp-get-connection-property (7) # process-name nil
> 08:22:30.422589 tramp-get-connection-property (7) # chunksize 0
> 08:22:30.422631 tramp-set-connection-property (7) # last-cmd-time (24092 6838 422621 0)
> 08:22:30.422677 tramp-send-string (10) # base64 -d -i >/home/nxuser/Documents/1.bundle.js <<'58274261e8a0f9f17931998a89b9c1fb'
>
> [HUNDREDS OF LINE OF BASE64 DATA]
>
> 08:22:30.498776 tramp-get-connection-property (7) # process-buffer nil
> 08:22:31.368697 tramp-sh-handle-write-region (3) # Decoding remote file ‘/plink:nxuser@vdemopro892dsy:/home/nxuser/Documents/1.bundle.js’ using ‘base64 -d -i >%s’...failed
> 08:22:31.371330 tramp-do-copy-or-rename-file (0) # Copying c:/Users/gde3/Documents/GDE3-Widgets/widget-template-vue/dist/1.bundle.js to /plink:nxuser@vdemopro892dsy:/home/nxuser/Documents/1.bundle.js...failed

I miss several trace lines after your second [HUNDREDS OF LINES OF BASE64
DATA]. Is there really nothing else between the BASE64 DATA and the last
three lines you have shown in the trace?

Maybe you could show *everything*, starting with the last 5 BASE64 DATA lines?

> Regards,
>
> Guillaume

Best regards, Michael.

tramp-debug-no-compression-2020-01-14.log (7M) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Copy-paste of large files from Windows to Linux - base64 issue?

Michael Albinus
Guillaume Demeyère <[hidden email]> writes:

> Hi Michael,

Hi Guillaume,

> I really think that I edited the file correctly, in order to make it
> less huge.

And I appreciate it. But often, every single trace line counts in order
to follow the work of Tramp when debugging.

> Here's an unedited version (5.3 MB) run this morning, so you can make
> sure I did.

Thanks. This is pretty good for analysis.

First, I've checked the encoded data. They have decoded fine into a
Javascript file, so it isn't a problem of base64 I'm confident.

Following the traces, Tramp has done its work until the end of the
tramp-send-string function. Likely, an error happened in the final
process-send-string call. There is a known Emacs problem, that sometimes
it fails to send a very large string at once. For this reason, we have
the tramp-chunksize variable. Please set it to a proper value (say, 100
or 50), and repeat your test.

If this doesn't help, we must know which error happens. The progress
reporter for copying a file in Tramp hides this, we only see "Decoding
remote file `/plink:nxuser@vdemopro892dsy:/home/nxuser/Documents/1.bundle.js'
using `base64 -d -i >%s'...failed". Newer Tramp versions tell us the
real error in such a case, when tramp-verbose is 10. Could you please
install the recent Tramp 2.4.3 from GNU ELPA, and try this? Send the
trace file (this case, w/o base64 data might be sufficient).

Finally, you could of course avoid the whole trouble by using pscp
instead of plink. But this wouldn't help to catch the problem :-)

> Best regards,
>
> Guillaume

Best regards, Michael.

Reply | Threaded
Open this post in threaded view
|

Re: Copy-paste of large files from Windows to Linux - base64 issue?

Guillaume Demeyère
Hi Michael,

Thank you for your explanations!

I tried setting tramp-chunksize to 100, and then to 50. I still got the same error, although I did see the file getting sent in numerous chunks in the logs.

So I installed tramp 2.4.3 from GNU ELPA and re-did the same test (without setting tramp-chunksize). The log gets huge and I am getting lost in it. Here it is.

Regards,

Guillaume


Michael Albinus <[hidden email]> escreveu no dia terça, 14/01/2020 à(s) 11:10:
Guillaume Demeyère <[hidden email]> writes:

> Hi Michael,

Hi Guillaume,

> I really think that I edited the file correctly, in order to make it
> less huge.

And I appreciate it. But often, every single trace line counts in order
to follow the work of Tramp when debugging.

> Here's an unedited version (5.3 MB) run this morning, so you can make
> sure I did.

Thanks. This is pretty good for analysis.

First, I've checked the encoded data. They have decoded fine into a
Javascript file, so it isn't a problem of base64 I'm confident.

Following the traces, Tramp has done its work until the end of the
tramp-send-string function. Likely, an error happened in the final
process-send-string call. There is a known Emacs problem, that sometimes
it fails to send a very large string at once. For this reason, we have
the tramp-chunksize variable. Please set it to a proper value (say, 100
or 50), and repeat your test.

If this doesn't help, we must know which error happens. The progress
reporter for copying a file in Tramp hides this, we only see "Decoding
remote file `/plink:nxuser@vdemopro892dsy:/home/nxuser/Documents/1.bundle.js'
using `base64 -d -i >%s'...failed". Newer Tramp versions tell us the
real error in such a case, when tramp-verbose is 10. Could you please
install the recent Tramp 2.4.3 from GNU ELPA, and try this? Send the
trace file (this case, w/o base64 data might be sufficient).

Finally, you could of course avoid the whole trouble by using pscp
instead of plink. But this wouldn't help to catch the problem :-)

> Best regards,
>
> Guillaume

Best regards, Michael.

tramp-2.4.3-debug-2020-01-14.log (18M) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Copy-paste of large files from Windows to Linux - base64 issue?

Michael Albinus
Guillaume Demeyère <[hidden email]> writes:

> Hi Michael,

Hi Guillaume,

> So I installed tramp 2.4.3 from GNU ELPA and re-did the same test
> (without setting tramp-chunksize). The log gets huge and I am getting
> lost in it. Here it is.

Well, the log tells us that my suspicion is right: a problem in
process-send-string:

> 15:29:32.075647 tramp-send-string (10) #
>   backtrace()
...
>   tramp-signal-hook-function(file-error ("Writing to process" "Invalid argument" #<process *tramp/plink nxuser@vdemopro892dsy*>))
>   process-send-string(#<process *tramp/plink nxuser@vdemopro892dsy*> "base64 -d -i >/home/nxuser/Documents/1.bundle.js <<'2a093387460512545bda27067c56b2df'

Don't worry about the Tramp functions in the backtrace; I have simply
hooked tramp-signal-hook-function into *any* error in order to get the
information. The error comes from process-send-string.

No idea what this "Writing to process" "Invalid argument"
means. Obviously, the argument it whines about is a process object,
which looks proper to me. Maybe the process has died, due to buffer
overflow, or whatever?

Could you, pls, rerun the test with Tramp 2.4.3, and tramp-chunksize set
to 1? It will be slow, but I want to see, whether it works.

In parallel, I'll check whether I could find something about the error
message. Neither process-send-string, nor Emacs on MS Windows, is
something I'm familiar with, unfortunately.

Btw, does this error happen always, or only for huge files? Does it also
happen when you run "emacs -Q"?

> Regards,
>
> Guillaume

Best regards, Michael.

Reply | Threaded
Open this post in threaded view
|

Re: Copy-paste of large files from Windows to Linux - base64 issue?

Guillaume Demeyère
Hi Michael,

I re-ran the test with Tramp 2.4.3 and tramp-chunksize set to 1. It completes with no error. I don't suppose the log (212MB, 18MB when compressed) would be helpful with no error, so I'm not adding them to this message (but tell me if you need them).

To make sure this was no coincidence, I re-ran the test several times. It worked each time. It also worked with tramp-chunksize set to 2, 5,6... It failed with tramp-chunksize set to 10.

Weirdly enough, I got frequent (but not systematic) fails on tramp-chunksize set to 1,2 and 5 but *only when tramp-verbose was left unchanged* (to 3).

All the tests I have been doing until now were made using "emacs -Q". The problem only happens with large files (although contrary to what I said, the test file 1.bundle.js weighs 2MB, not 20).

It is true that I tend to lose my Tramp connection quite often and unexplicably. I also get asked for my password from time to time, when establishing the connection, even though it should (and does, like, 70% of the time) get it right from  ~/.authinfo. During all the tests of this morning, it continually asked for the password (could it be that the file was locked ?).

I acknowledge that pscp works perfectly in this use case. However, in my real use case, I execute the copy-paste as root (from dired+), and to the best of my knowledge, pscp does not support multi-hops (I don't know why, but I wish it dit, it would be really useful).

Best regards,
Guillaume

Reply | Threaded
Open this post in threaded view
|

Re: Copy-paste of large files from Windows to Linux - base64 issue?

Michael Albinus
Guillaume Demeyère <[hidden email]> writes:

> Hi Michael,

Hi Guillaume,

sorry for the delay, I wanted to run my own tests. Since I do not use MS
Windows on a regular basis, I had to steal such a machine
temporarily. It runs MS Windows 7 (pls don't discuss with me, it isn't
my own machine). I've installed the recent Emacs 28.0.50 from
http://alpha.gnu.org/, which includes Tramp 2.4.4-pre.

It doesn't differ too much from the configuration you use, because all
these process-send-string thingies I haven't touched for years.

> I re-ran the test with Tramp 2.4.3 and tramp-chunksize set to 1. It
> completes with no error. I don't suppose the log (212MB, 18MB when
> compressed) would be helpful with no error, so I'm not adding them to
> this message (but tell me if you need them).
>
> To make sure this was no coincidence, I re-ran the test several times.
> It worked each time. It also worked with tramp-chunksize set to 2,
> 5,6... It failed with tramp-chunksize set to 10.

OK.

> Weirdly enough, I got frequent (but not systematic) fails on
> tramp-chunksize set to 1,2 and 5 but *only when tramp-verbose was left
> unchanged* (to 3).

Sounds like a timing issue.

> All the tests I have been doing until now were made using "emacs -Q".
> The problem only happens with large files (although contrary to what I
> said, the test file 1.bundle.js weighs 2MB, not 20).

Well, I have performed my test also with "emacs -Q", and I haven't set
any additional config. Copying a 2 MiB file (emacsclientw.exe) to a
remote machine works w/o problems.

Hmm, copying emacs.exe to a remote machine (115 MiB) fails, but this is
another file size. Copying with tramp-chunksize set to 10 works,
slowly. I haven't tested with larger tramp-chunksize.

> It is true that I tend to lose my Tramp connection quite often and
> unexplicably. I also get asked for my password from time to time, when
> establishing the connection, even though it should (and does, like,
> 70% of the time) get it right from  ~/.authinfo. During all the tests
> of this morning, it continually asked for the password (could it be
> that the file was locked ?).

Have you tried whether it works with another remote machine? The machine
you have used for your tests reports "Linux 3.10.0-1062.4.1.el7.x86_64",
which is quite old.

> I acknowledge that pscp works perfectly in this use case. However, in
> my real use case, I execute the copy-paste as root (from dired+), and
> to the best of my knowledge, pscp does not support multi-hops (I don't
> know why, but I wish it dit, it would be really useful).

Unfortunately, for technical reasons multi-hops cannot be implemented
for external methods like pscp. I wish it would be possible.

> Best regards,
> Guillaume

Best regards, Michael.