bug#43566: 27.1; url-http and excorporate does not authenticate

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

bug#43566: 27.1; url-http and excorporate does not authenticate

Lars Ingebrigtsen
Domingo Gómez Pérez <[hidden email]> writes:

> url-http-handle-authentication: Wrong authorization used for https://correo.unican.es/ews/exchange.asmx
> (Note, the user and password are right, since I can download the page
> using wget)

[...]

> Authorization: NTLM TlRMTVN[etc]

Is wget using NTLM authentication, though?  And how have you set up
url.el to use NTLM?

I'm not familiar with NTLM at all, so I hope there wasn't anything
secret in that base64'd auth string.

--
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no



Reply | Threaded
Open this post in threaded view
|

bug#43566: 27.1; url-http and excorporate does not authenticate

Domingo Gómez Pérez
En la fecha jue, 24 sep 2020, Lars Ingebrigtsen <[hidden email]>
escribió:

> Domingo Gómez Pérez <[hidden email]> writes:
>
>> url-http-handle-authentication: Wrong authorization used for https://correo.unican.es/ews/exchange.asmx
>> (Note, the user and password are right, since I can download the page
>> using wget)
>
> [...]
>
>> Authorization: NTLM TlRMTVN[etc]
>
> Is wget using NTLM authentication, though?  And how have you set up
> url.el to use NTLM?


Dear Lars Ingebrigtsen,

First of all, thanks for your time and help.

I would say so, because in the terminal it reads "Autenticación
seleccionada: NTLM" (sorry, it is in spanish but I think the meaning is clear).

>
> I'm not familiar with NTLM at all, so I hope there wasn't anything
> secret in that base64'd auth string.
>
Neither I do, so I might change my password.

Thanks for the advice.

Best regards, Domingo.



Reply | Threaded
Open this post in threaded view
|

bug#43566: 27.1; url-http and excorporate does not authenticate

Lars Ingebrigtsen
Domingo Gómez Pérez <[hidden email]> writes:

>> Is wget using NTLM authentication, though?  And how have you set up
>> url.el to use NTLM?
>
> Dear Lars Ingebrigtsen,
>
> First of all, thanks for your time and help.
>
> I would say so, because in the terminal it reads "Autenticación
> seleccionada: NTLM" (sorry, it is in spanish but I think the meaning is clear).

Thanks for checking.

I'm absolutely completely unfamiliar with NTLM (I assume that's a
Windows-only thing?), so somebody else will have to try debugging this.

--
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no



Reply | Threaded
Open this post in threaded view
|

bug#43566: 27.1; url-http and excorporate does not authenticate

Thomas Fitzsimmons-2
Hi,

Lars Ingebrigtsen <[hidden email]> writes:

> Domingo Gómez Pérez <[hidden email]> writes:
>
>>> Is wget using NTLM authentication, though?  And how have you set up
>>> url.el to use NTLM?
>>
>> Dear Lars Ingebrigtsen,
>>
>> First of all, thanks for your time and help.
>>
>> I would say so, because in the terminal it reads "Autenticación
>> seleccionada: NTLM" (sorry, it is in spanish but I think the meaning is clear).
>
> Thanks for checking.
>
> I'm absolutely completely unfamiliar with NTLM (I assume that's a
> Windows-only thing?), so somebody else will have to try debugging this.

It turns out that this is a regression in Emacs 27.1 introduced by the
fix for bug 27022:

   https://debbugs.gnu.org/27022

The NTLM protocol requires a three step handshake with the server, where
handshake information is passed from the client to the server via
Authorization headers.  I'm not sure if the 27022 scenario would result
in url-http-ntlm inflooping too; if it would, then the obvious
fix-of-the-fix would probably be wrong (i.e., to disable the check when
NTLM is present in the authorization string).

It also turns out that another user found the regression and filed a bug
report with a workaround a month ago:

   https://debbugs.gnu.org/44439

(From now on I'll search for "excorporate" using debbugs-gnu-search
instead of just scanning subject lines.)

I've merged bug#44439 and another report of the same problem, bug#44195,
into this one.

I no longer use the NTLM mode myself, but I did manage to independently
figure out the issue by implementing an NTLM server in elisp.  I'm going
to add it to the test suite, since otherwise it's not feasible for Emacs
maintainers to test ntlm.el properly (which is what happened in Emacs
27.1).  I can test inflooping against it too once I have the changes
ready.

Thomas



Reply | Threaded
Open this post in threaded view
|

bug#43566: 27.1; url-http and excorporate does not authenticate

Lars Ingebrigtsen
Thomas Fitzsimmons <[hidden email]> writes:

> I no longer use the NTLM mode myself, but I did manage to independently
> figure out the issue by implementing an NTLM server in elisp.  I'm going
> to add it to the test suite, since otherwise it's not feasible for Emacs
> maintainers to test ntlm.el properly (which is what happened in Emacs
> 27.1).  I can test inflooping against it too once I have the changes
> ready.

Great!

--
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no



Reply | Threaded
Open this post in threaded view
|

bug#43566: 27.1; url-http and excorporate does not authenticate

Lars Ingebrigtsen
Thomas Fitzsimmons <[hidden email]> writes:

> Is there a test suite entry for bug 27022 that I can run against my
> elisp NTLM server?

Not that I know of.

--
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no



Reply | Threaded
Open this post in threaded view
|

bug#43566: 27.1; url-http and excorporate does not authenticate

Thomas Fitzsimmons-2
Lars Ingebrigtsen <[hidden email]> writes:

> Thomas Fitzsimmons <[hidden email]> writes:
>
>> Is there a test suite entry for bug 27022 that I can run against my
>> elisp NTLM server?
>
> Not that I know of.

OK, I pushed the workaround to emacs-27.

I'll leave this bug report open until I've pushed the test suite item to
master.

Thomas



Reply | Threaded
Open this post in threaded view
|

bug#43566: 27.1; url-http and excorporate does not authenticate

Lars Ingebrigtsen
Thomas Fitzsimmons <[hidden email]> writes:

> I pushed to master the new tests, and some build logic to load
> dependencies from GNU ELPA.
>
> Lars, can you confirm this works for you:
>
>     make -C test ntlm-tests

Yup, works for me after:

> (Maybe revert my NTLM workaround in url-http.el, and confirm that the
> two new tests fail for you like in this original bug report?)
>
> By default this looks for dependencies in test/../../elpa, but you can
> use:
>
>     make -C test ntlm-tests GNU_ELPA_DIRECTORY=/path/to/elpa
>
> if you have it checked out somewhere else.  You'll probably need to
> "make externals" to pull my recent web-server.el change.

My ELPA is in that location, but I had to update it in this way before
the tests succeed -- so they fail unless you have ELPA installed.

So the tests should check for that first, otherwise most people are
going to see failing tests.

--
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no



Reply | Threaded
Open this post in threaded view
|

bug#43566: 27.1; url-http and excorporate does not authenticate

Thomas Fitzsimmons-2
Lars Ingebrigtsen <[hidden email]> writes:

> Thomas Fitzsimmons <[hidden email]> writes:
>
>> I pushed to master the new tests, and some build logic to load
>> dependencies from GNU ELPA.
>>
>> Lars, can you confirm this works for you:
>>
>>     make -C test ntlm-tests
>
> Yup, works for me after:

OK, thanks.

>> (Maybe revert my NTLM workaround in url-http.el, and confirm that the
>> two new tests fail for you like in this original bug report?)
>>
>> By default this looks for dependencies in test/../../elpa, but you can
>> use:
>>
>>     make -C test ntlm-tests GNU_ELPA_DIRECTORY=/path/to/elpa
>>
>> if you have it checked out somewhere else.  You'll probably need to
>> "make externals" to pull my recent web-server.el change.
>
> My ELPA is in that location, but I had to update it in this way before
> the tests succeed -- so they fail unless you have ELPA installed.

The tests will fail if web-server.el is present but out-of-date, which I
think is unavoidable.  But...

> So the tests should check for that first, otherwise most people are
> going to see failing tests.

... I did implement that; can you try moving your elpa checkout
somewhere else, or try:

    make -C test ntlm-tests GNU_ELPA_DIRECTORY=/nonexistent

and confirm you see the tests being skipped?

Thanks,
Thomas



Reply | Threaded
Open this post in threaded view
|

bug#43566: 27.1; url-http and excorporate does not authenticate

Lars Ingebrigtsen
Thomas Fitzsimmons <[hidden email]> writes:

> The tests will fail if web-server.el is present but out-of-date, which I
> think is unavoidable.  But...

Would it be possible to check the web-server.el version to make this
more robust?

>> So the tests should check for that first, otherwise most people are
>> going to see failing tests.
>
> ... I did implement that; can you try moving your elpa checkout
> somewhere else, or try:
>
>     make -C test ntlm-tests GNU_ELPA_DIRECTORY=/nonexistent
>
> and confirm you see the tests being skipped?

Yup; they're skipped if I do that.

--
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no



Reply | Threaded
Open this post in threaded view
|

bug#43566: 27.1; url-http and excorporate does not authenticate

Thomas Fitzsimmons-2
Lars Ingebrigtsen <[hidden email]> writes:

> Thomas Fitzsimmons <[hidden email]> writes:
>
>> The tests will fail if web-server.el is present but out-of-date, which I
>> think is unavoidable.  But...
>
> Would it be possible to check the web-server.el version to make this
> more robust?

Michael also asked about this on emacs-devel.  I looked but I can't see
a simple way to do it.  package.el functionality isn't available in this
context.

Besides, I wouldn't really want to require a specific version range of
web-server.el.  Anyone with up-to-date elpa.git won't have the issue you
ran into.  I think elpa main tip and emacs master tip should always be
compatible for these tests from now on.

>>> So the tests should check for that first, otherwise most people are
>>> going to see failing tests.
>>
>> ... I did implement that; can you try moving your elpa checkout
>> somewhere else, or try:
>>
>>     make -C test ntlm-tests GNU_ELPA_DIRECTORY=/nonexistent
>>
>> and confirm you see the tests being skipped?
>
> Yup; they're skipped if I do that.

OK, thanks.

Thomas



Reply | Threaded
Open this post in threaded view
|

bug#43566: 27.1; url-http and excorporate does not authenticate

Thomas Fitzsimmons-2
Thomas Fitzsimmons <[hidden email]> writes:

> Lars Ingebrigtsen <[hidden email]> writes:
>
>> Thomas Fitzsimmons <[hidden email]> writes:
>>
>>> The tests will fail if web-server.el is present but out-of-date, which I
>>> think is unavoidable.  But...
>>
>> Would it be possible to check the web-server.el version to make this
>> more robust?
>
> Michael also asked about this on emacs-devel.  I looked but I can't see
> a simple way to do it.  package.el functionality isn't available in this
> context.

Michael wanted a general approach to requiring specific GNU ELPA package
versions; he suggested an implementation which I adapted and pushed to
master.  You can try it by alternating between revisions
a1c767fe56acfed6627682fadc337203f3b8678c and
4edcba28826869b07f622fd83f6aab96fd8e4d19 of web-server.el and confirming
that for the former commit the tests are skipped and for the latter,
they're run.

I also tested how url-retrieve-synchronously behaves with NTLM
authentication when the authinfo password is wrong.  It doesn't
infinite-loop -- it tries once with the authinfo credentials and if they
fail, it prompts the user for a username and password.  Therefore, the
patch that caused the regression isn't needed for NTLM and the
workaround patch I pushed is fine as-is.

I'm marking this bug done.

Thanks,
Thomas