bug#35443: 27.0.50; Gnus (nnimap) shows "ghost" messages in summary buffer

classic Classic list List threaded Threaded
30 messages Options
12
Reply | Threaded
Open this post in threaded view
|

bug#35443: 27.0.50; Gnus (nnimap) shows "ghost" messages in summary buffer

Ulrich Mueller-2
In GNU Emacs 27.0.50 (build 1, x86_64-pc-linux-gnu, X toolkit)
 of 2019-04-21 built on a1i15
Windowing system distributor 'The X.Org Foundation', version 11.0.11905000
System Description: Gentoo/Linux

When fetching e-mail from a dovecot-2.3.5.1 IMAP server, Gnus/nnimap
gets confused and displays ghost messages with address "nobody" and
subject "(none)" in the summary buffer, like in this example:

*Summary nnimap+dev.gentoo.org:INBOX*
----------------------------------------------------------------------
R. [   1: Ulrich Mueller         ] test 1
 . [   ?: nobody                 ] (none)
 . [   1: Ulrich Mueller         ] test 2
----------------------------------------------------------------------

My gnus-secondary-select-methods looks like this:

      '((nnimap "dev.gentoo.org"
                (nnimap-stream tls)
                (nnimap-user "ulm"))
        ;; ... further methods omitted ...
        )

For debugging, I have enabled logging with nnimap-record-commands, and
set debug-on-entry for function nnimap-transform-headers.

I then get the following "*imap log*" buffer:
----------------------------------------------------------------------
07:41:57 [dev.gentoo.org] (inhibited)
07:41:59 [dev.gentoo.org] 10105 CAPABILITY
07:41:59 [dev.gentoo.org] 10106 ENABLE QRESYNC
07:41:59 [dev.gentoo.org] 10107 LIST "" "*"
[... other servers ...]
07:42:00 [dev.gentoo.org] 10147 EXAMINE "INBOX" (QRESYNC (1364468255 63360))
[... other servers ...]
07:42:08 [dev.gentoo.org] 10192 SELECT "INBOX"
07:42:08 [dev.gentoo.org] 10193 SELECT "INBOX"
07:42:08 [dev.gentoo.org] 10194 UID FETCH 32409:32410 (UID RFC822.SIZE BODYSTRUCTURE BODY.PEEK[HEADER.FIELDS (Subject From Date Message-Id References In-Reply-To Xref X-Diary-Time-Zone X-Diary-Dow X-Diary-Year X-Diary-Month X-Diary-Dom X-Diary-Hour X-Diary-Minute To Newsgroups Cc)])
----------------------------------------------------------------------

Buffer " *nnimap dev.gentoo.org nil  *nntpd**" looks like this, upon
entering nnimap-transform-headers:
----------------------------------------------------------------------
* 583 FETCH (UID 32409 RFC822.SIZE 698 BODYSTRUCTURE ("text" "plain" ("charset" "us-ascii") NIL NIL "7bit" 8 1 NIL NIL NIL NIL) BODY[HEADER.FIELDS (SUBJECT FROM DATE MESSAGE-ID REFERENCES IN-REPLY-TO XREF X-DIARY-TIME-ZONE X-DIARY-DOW X-DIARY-YEAR X-DIARY-MONTH X-DIARY-DOM X-DIARY-HOUR X-DIARY-MINUTE TO NEWSGROUPS CC)] {165}
From: Ulrich Mueller <[hidden email]>
To: [hidden email]
Subject: test 1
Date: Sat, 27 Apr 2019 07:39:56 +0200
Message-ID: <[hidden email]>

)
* 584 FETCH (UID 32410 RFC822.SIZE 698 BODYSTRUCTURE ("text" "plain" ("charset" "us-ascii") NIL NIL "7bit" 8 1 NIL NIL NIL NIL) BODY[HEADER.FIELDS (SUBJECT FROM DATE MESSAGE-ID REFERENCES IN-REPLY-TO XREF X-DIARY-TIME-ZONE X-DIARY-DOW X-DIARY-YEAR X-DIARY-MONTH X-DIARY-DOM X-DIARY-HOUR X-DIARY-MINUTE TO NEWSGROUPS CC)] {165}
From: Ulrich Mueller <[hidden email]>
To: [hidden email]
Subject: test 2
Date: Sat, 27 Apr 2019 07:40:15 +0200
Message-ID: <[hidden email]>

)
* 583 FETCH (UID 32409 MODSEQ (63364) FLAGS ($HasNoAttachment))
* 584 FETCH (UID 32410 MODSEQ (63364) FLAGS ($HasNoAttachment))
10194 OK Fetch completed (0.003 + 0.000 + 0.002 secs).
----------------------------------------------------------------------



Reply | Threaded
Open this post in threaded view
|

bug#35443: 27.0.50; Gnus (nnimap) shows "ghost" messages in summary buffer

Eric Abrahamsen-2
Ulrich Mueller <[hidden email]> writes:

> In GNU Emacs 27.0.50 (build 1, x86_64-pc-linux-gnu, X toolkit)
>  of 2019-04-21 built on a1i15
> Windowing system distributor 'The X.Org Foundation', version 11.0.11905000
> System Description: Gentoo/Linux
>
> When fetching e-mail from a dovecot-2.3.5.1 IMAP server, Gnus/nnimap
> gets confused and displays ghost messages with address "nobody" and
> subject "(none)" in the summary buffer, like in this example:

Are you able to build an earlier version of Emacs (say, from a couple of
months ago) and test if you see the same thing? I made some fairly deep
changes to Gnus over the past month, and while I don't see any reason
why that would cause this behavior, it would be nice to rule it out.

Thanks,
Eric



Reply | Threaded
Open this post in threaded view
|

bug#35443: 27.0.50; Gnus (nnimap) shows "ghost" messages in summary buffer

Ulrich Mueller-2
>>>>> On Sat, 27 Apr 2019, Eric Abrahamsen wrote:

> Are you able to build an earlier version of Emacs (say, from a couple
> of months ago) and test if you see the same thing? I made some fairly
> deep changes to Gnus over the past month, and while I don't see any
> reason why that would cause this behavior, it would be nice to rule it
> out.

Yes, I had tried with Emacs 26.2 and it has the same problem.



Reply | Threaded
Open this post in threaded view
|

bug#35443: 27.0.50; Gnus (nnimap) shows "ghost" messages in summary buffer

Eric Abrahamsen-2
Ulrich Mueller <[hidden email]> writes:

>>>>>> On Sat, 27 Apr 2019, Eric Abrahamsen wrote:
>
>> Are you able to build an earlier version of Emacs (say, from a couple
>> of months ago) and test if you see the same thing? I made some fairly
>> deep changes to Gnus over the past month, and while I don't see any
>> reason why that would cause this behavior, it would be nice to rule it
>> out.
>
> Yes, I had tried with Emacs 26.2 and it has the same problem.

Huh. Can you show us the value of gnus-newsgroup-data and
gnus-newsgroup-headers in this group? Does the dummy article always show
up? Only in this group? Is this a newly-added IMAP server, and did it
display this problem right from the beginning?



Reply | Threaded
Open this post in threaded view
|

bug#35443: 27.0.50; Gnus (nnimap) shows "ghost" messages in summary buffer

Ulrich Mueller-2
>>>>> On Sat, 27 Apr 2019, Eric Abrahamsen wrote:

> Huh. Can you show us the value of gnus-newsgroup-data and
> gnus-newsgroup-headers in this group?

See below, for the current group, whose summary buffer shows your
article and a dummy article.

> Does the dummy article always show up?

It always shows up when new articles have been fetched. It doesn't
show up when entering a group that doesn't have new mail (i.e., if a
group previously had a dummy article, it will be gone when reentering
that group).

> Only in this group?

All groups for that particular IMAP server.

> Is this a newly-added IMAP server, and did it display this problem
> right from the beginning?

AFAICS, the problem appeared after dovecot on the server side was
upgraded from version 2.2.34 to 2.3.5.1. But I see it only with Gnus.
I don't have any issues with Mozilla Thunderbird (on GNU/Linux) nor with
K-9 Mail (on Android/Linux).

----------------------------------------------------------------------
gnus-newsgroup-data is a variable defined in ‘gnus-sum.el’.
Its value is shown below.

Documentation:
Not documented as a variable.

Value:
((32415 82 2
        [32415 "Re: bug#35443: 27.0.50; Gnus (nnimap) shows \"ghost\" messages in summary buffer" "Eric Abrahamsen <[hidden email]>" "Sat, 27 Apr 2019 13:47:18 -0700" "<[hidden email]>" "<[hidden email]> <[hidden email]> <[hidden email]>" 2609 16 nil
               ((Cc . "[hidden email]")
                (To . "Ulrich Mueller <[hidden email]>"))]
        0)
 (32415 32 116
        [32415 "(none)" "(nobody)" "" "fake+none+nnimap+dev.gentoo.org:INBOX+32415" nil -1 -1 nil nil]
        0))
Local in buffer *Summary nnimap+dev.gentoo.org:INBOX*; global value is the same.
----------------------------------------------------------------------
gnus-newsgroup-headers is a variable defined in ‘gnus-sum.el’.
Its value is
([32415 "Re: bug#35443: 27.0.50; Gnus (nnimap) shows \"ghost\" messages in summary buffer" "Eric Abrahamsen <[hidden email]>" "Sat, 27 Apr 2019 13:47:18 -0700" "<[hidden email]>" "<[hidden email]> <[hidden email]> <[hidden email]>" 2609 16 nil
        ((Cc . "[hidden email]")
         (To . "Ulrich Mueller <[hidden email]>"))]
 [32415 "(none)" "(nobody)" "" "fake+none+nnimap+dev.gentoo.org:INBOX+32415" nil -1 -1 nil nil])
Local in buffer *Summary nnimap+dev.gentoo.org:INBOX*; global value is nil

Documentation:
List of article headers in the current newsgroup.
----------------------------------------------------------------------



Reply | Threaded
Open this post in threaded view
|

bug#35443: 27.0.50; Gnus (nnimap) shows "ghost" messages in summary buffer

Eric Abrahamsen-2
Ulrich Mueller <[hidden email]> writes:

>>>>>> On Sat, 27 Apr 2019, Eric Abrahamsen wrote:
>
>> Huh. Can you show us the value of gnus-newsgroup-data and
>> gnus-newsgroup-headers in this group?
>
> See below, for the current group, whose summary buffer shows your
> article and a dummy article.
>
>> Does the dummy article always show up?
>
> It always shows up when new articles have been fetched. It doesn't
> show up when entering a group that doesn't have new mail (i.e., if a
> group previously had a dummy article, it will be gone when reentering
> that group).
>
>> Only in this group?
>
> All groups for that particular IMAP server.
>
>> Is this a newly-added IMAP server, and did it display this problem
>> right from the beginning?
>
> AFAICS, the problem appeared after dovecot on the server side was
> upgraded from version 2.2.34 to 2.3.5.1. But I see it only with Gnus.
> I don't have any issues with Mozilla Thunderbird (on GNU/Linux) nor with
> K-9 Mail (on Android/Linux).

Thanks, this should be enough to make some progress in tracking the
problem down. Give me a couple of days...



Reply | Threaded
Open this post in threaded view
|

bug#35443: 27.0.50; Gnus (nnimap) shows "ghost" messages in summary buffer

Eric Abrahamsen-2
In reply to this post by Ulrich Mueller-2
Ulrich Mueller <[hidden email]> writes:

> In GNU Emacs 27.0.50 (build 1, x86_64-pc-linux-gnu, X toolkit)
>  of 2019-04-21 built on a1i15
> Windowing system distributor 'The X.Org Foundation', version 11.0.11905000
> System Description: Gentoo/Linux
>
> When fetching e-mail from a dovecot-2.3.5.1 IMAP server, Gnus/nnimap
> gets confused and displays ghost messages with address "nobody" and
> subject "(none)" in the summary buffer, like in this example:
>
> *Summary nnimap+dev.gentoo.org:INBOX*
> ----------------------------------------------------------------------
> R. [   1: Ulrich Mueller         ] test 1
>  . [   ?: nobody                 ] (none)
>  . [   1: Ulrich Mueller         ] test 2
> ----------------------------------------------------------------------

[...]

> Buffer " *nnimap dev.gentoo.org nil *nntpd**" looks like this, upon
> entering nnimap-transform-headers:
>----------------------------------------------------------------------
> * 583 FETCH (UID 32409 RFC822.SIZE 698 BODYSTRUCTURE ("text" "plain" ("charset" "us-ascii") NIL NIL "7bit" 8 1 NIL NIL NIL NIL) BODY[HEADER.FIELDS (SUBJECT FROM DATE MESSAGE-ID REFERENCES IN-REPLY-TO XREF X-DIARY-TIME-ZONE X-DIARY-DOW X-DIARY-YEAR X-DIARY-MONTH X-DIARY-DOM X-DIARY-HOUR X-DIARY-MINUTE TO NEWSGROUPS CC)] {165}
> From: Ulrich Mueller <[hidden email]>
> To: [hidden email]
> Subject: test 1
> Date: Sat, 27 Apr 2019 07:39:56 +0200
> Message-ID: <[hidden email]>
>
> )
> * 584 FETCH (UID 32410 RFC822.SIZE 698 BODYSTRUCTURE ("text" "plain" ("charset" "us-ascii") NIL NIL "7bit" 8 1 NIL NIL NIL NIL) BODY[HEADER.FIELDS (SUBJECT FROM DATE MESSAGE-ID REFERENCES IN-REPLY-TO XREF X-DIARY-TIME-ZONE X-DIARY-DOW X-DIARY-YEAR X-DIARY-MONTH X-DIARY-DOM X-DIARY-HOUR X-DIARY-MINUTE TO NEWSGROUPS CC)] {165}
> From: Ulrich Mueller <[hidden email]>
> To: [hidden email]
> Subject: test 2
> Date: Sat, 27 Apr 2019 07:40:15 +0200
> Message-ID: <[hidden email]>
>
> )
> * 583 FETCH (UID 32409 MODSEQ (63364) FLAGS ($HasNoAttachment))
> * 584 FETCH (UID 32410 MODSEQ (63364) FLAGS ($HasNoAttachment))
> 10194 OK Fetch completed (0.003 + 0.000 + 0.002 secs).
> ----------------------------------------------------------------------

Okay, I've made a bit of progress on this.

Locally I'm using Dovecot 2.3.6, and it does not output those last two
lines before the OK.

`nnimap-transform-headers' is not expecting those two lines -- it
deletes all but the last, leaving a buffer that looks like:

211 32409 Article retrieved.
Chars: 698
Lines: 1
From: Ulrich Mueller <[hidden email]>
To: [hidden email]
Subject: test 1
Date: Sat, 27 Apr 2019 07:39:56 +0200
Message-ID: <[hidden email]>

.
211 32410 Article retrieved.
Chars: 698
Lines: 1
From: Ulrich Mueller <[hidden email]>
To: [hidden email]
Subject: test 2
Date: Sat, 27 Apr 2019 07:40:15 +0200
Message-ID: <[hidden email]>

.
211 32409 Article retrieved.
* 584 FETCH (UID 32410 MODSEQ (63364) FLAGS ($HasNoAttachment))
10194 OK Fetch completed (0.003 + 0.000 + 0.002 secs).
.

Which Gnus then parses as an _extra_ article 32409, but then there's no
header data for it, which is why you get all the "nobody" "none"
nonsense.

Essentially, we're not set up to parse this particular return value.

I guess what I'll do is ask on the dovecot mailing list under what
circumstances/versions we'd get a string like that, and try to help Gnus
parse it correctly.

Thanks for your patience,
Eric



Reply | Threaded
Open this post in threaded view
|

bug#35443: 27.0.50; Gnus (nnimap) shows "ghost" messages in summary buffer

Eric Abrahamsen-2
In reply to this post by Ulrich Mueller-2

On 04/28/19 05:53 AM, Ulrich Mueller wrote:
>>>>>> On Sat, 27 Apr 2019, Eric Abrahamsen wrote:
>
>> Huh. Can you show us the value of gnus-newsgroup-data and
>> gnus-newsgroup-headers in this group?
>
> See below, for the current group, whose summary buffer shows your
> article and a dummy article.

Can you also tell me your value for `nnmail-extra-headers'?

Thanks,
Eric



Reply | Threaded
Open this post in threaded view
|

bug#35443: 27.0.50; Gnus (nnimap) shows "ghost" messages in summary buffer

Ulrich Mueller-2
>>>>> On Tue, 07 May 2019, Eric Abrahamsen wrote:

> Can you also tell me your value for `nnmail-extra-headers'?

(X-Diary-Time-Zone X-Diary-Dow X-Diary-Year X-Diary-Month X-Diary-Dom X-Diary-Hour X-Diary-Minute To Newsgroups Cc)

Not sure where the X-Diary-* ones come from, since I don't have that
variable in my config files.



Reply | Threaded
Open this post in threaded view
|

bug#35443: 27.0.50; Gnus (nnimap) shows "ghost" messages in summary buffer

Ulrich Mueller-2
In reply to this post by Eric Abrahamsen-2
>>>>> On Tue, 07 May 2019, Eric Abrahamsen wrote:

> Locally I'm using Dovecot 2.3.6, and it does not output those last two
> lines before the OK.

Hm, on the server side here dovecot was upgraded to 2.3.6 today, but
the problem persists. Upon entering nnimap-transform-headers, the
" *nnimap dev.gentoo.org nil  *nntpd**" buffer shows the same pattern
as in my original report:

* 590 FETCH (UID 32471 RFC822.SIZE 694 BODYSTRUCTURE ("text" "plain" ("charset" "us-ascii") NIL NIL "7bit" 6 1 NIL NIL NIL NIL) BODY[HEADER.FIELDS (SUBJECT FROM DATE MESSAGE-ID REFERENCES IN-REPLY-TO XREF X-DIARY-TIME-ZONE X-DIARY-DOW X-DIARY-YEAR X-DIARY-MONTH X-DIARY-DOM X-DIARY-HOUR X-DIARY-MINUTE TO NEWSGROUPS CC)] {163}
From: Ulrich Mueller <[hidden email]>
To: [hidden email]
Subject: test
Date: Wed, 08 May 2019 13:59:32 +0200
Message-ID: <[hidden email]>

)
* 590 FETCH (UID 32471 MODSEQ (63547) FLAGS ($HasNoAttachment))
16742 OK Fetch completed (0.003 + 0.000 + 0.002 secs).



Reply | Threaded
Open this post in threaded view
|

bug#35443: 27.0.50; Gnus (nnimap) shows "ghost" messages in summary buffer

Eric Abrahamsen-2
In reply to this post by Eric Abrahamsen-2
Eric Abrahamsen <[hidden email]> writes:

--8<---------------cut here---------------start------------->8---

> Ulrich Mueller <[hidden email]> writes:
>
>>>>>>> On Sat, 27 Apr 2019, Eric Abrahamsen wrote:
>>
>>> Huh. Can you show us the value of gnus-newsgroup-data and
>>> gnus-newsgroup-headers in this group?
>>
>> See below, for the current group, whose summary buffer shows your
>> article and a dummy article.
>>
>>> Does the dummy article always show up?
>>
>> It always shows up when new articles have been fetched. It doesn't
>> show up when entering a group that doesn't have new mail (i.e., if a
>> group previously had a dummy article, it will be gone when reentering
>> that group).
>>
>>> Only in this group?
>>
>> All groups for that particular IMAP server.
>>
>>> Is this a newly-added IMAP server, and did it display this problem
>>> right from the beginning?
>>
>> AFAICS, the problem appeared after dovecot on the server side was
>> upgraded from version 2.2.34 to 2.3.5.1. But I see it only with Gnus.
>> I don't have any issues with Mozilla Thunderbird (on GNU/Linux) nor with
>> K-9 Mail (on Android/Linux).
>
> Thanks, this should be enough to make some progress in tracking the
> problem down. Give me a couple of days...

Okay, here's what I was told on irc:

--8<---------------cut here---------------start------------->8---
<tss> looks like it's the new feature that adds $HasAttachment or
      $HasNoAttachment flags to mails. they're actually supposed to be added
      only during mail delivery, and there's a setting needed to enable them:
      mail_attachment_detection_options = add-flags-on-save
<tss> but .. there's an "unintentional feature" :) that they also get added
      during some FETCH commands if they're not already there
<tss> but even without this, imap protocol allows sending FETCH FLAGS updates
      (or any updates really) as a response to any command. and with
      concurrent imap access dovecot does this.
<tss> for example if another client adds a \Seen flag, that same thing could
      happen
<jkt> either way, a client should really be able to process these
      "unexpeceted" FETCH updates, that's what the standard is about
--8<---------------cut here---------------end--------------->8---

Ultimately, the "real" fix would be to teach Gnus to handle all possible
responses from IMAP servers, probably using a proper parser.

But if you have access to the dovecot config in your case, you might try
removing the mail_attachment_detection_options setting above, if it's
set. If it's not set, then you might be getting the "unintentional
feature".

I suppose in the interim I could mess with `nnimap-transform-headers'
and try to add some bookkeeping so that multiple FETCH responses for the
same article UID get merged together. On the other hand, this feature
only seems to relate to has/hasnoattachment flags, which Gnus doesn't
handle anyway -- we could safely drop those lines altogether.

(Though it sure would be nice to handle hasattachment flags, that's
something that users have requested in the past...)

Well, that's where we're at.

Eric



Reply | Threaded
Open this post in threaded view
|

bug#35443: 27.0.50; Gnus (nnimap) shows "ghost" messages in summary buffer

Ulrich Mueller-2
>>>>> On Thu, 09 May 2019, Eric Abrahamsen wrote:

> Okay, here's what I was told on irc:

> <tss> looks like it's the new feature that adds $HasAttachment or
>       $HasNoAttachment flags to mails. they're actually supposed to be added
>       only during mail delivery, and there's a setting needed to enable them:
>       mail_attachment_detection_options = add-flags-on-save
> <tss> but .. there's an "unintentional feature" :) that they also get added
>       during some FETCH commands if they're not already there
> <tss> but even without this, imap protocol allows sending FETCH FLAGS updates
>       (or any updates really) as a response to any command. and with
>       concurrent imap access dovecot does this.
> <tss> for example if another client adds a \Seen flag, that same thing could
>       happen
> <jkt> either way, a client should really be able to process these
>       "unexpeceted" FETCH updates, that's what the standard is about
> --8<---------------cut here---------------end--------------->8---

> Ultimately, the "real" fix would be to teach Gnus to handle all possible
> responses from IMAP servers, probably using a proper parser.

> But if you have access to the dovecot config in your case, you might try
> removing the mail_attachment_detection_options setting above, if it's
> set. If it's not set, then you might be getting the "unintentional
> feature".

Thank you very much for your help. Indeed the add-flags-on-save option
was enabled in the config file. Gentoo's infrastructure team has agreed
to disable it for now, which made the problem go away.

> I suppose in the interim I could mess with `nnimap-transform-headers'
> and try to add some bookkeeping so that multiple FETCH responses for
> the same article UID get merged together. On the other hand, this
> feature only seems to relate to has/hasnoattachment flags, which Gnus
> doesn't handle anyway -- we could safely drop those lines altogether.

> (Though it sure would be nice to handle hasattachment flags, that's
> something that users have requested in the past...)

Maybe drop the extra FETCH responses for now, and add proper bookkeeping
later when implementing the requested feature?



Reply | Threaded
Open this post in threaded view
|

bug#35443: 27.0.50; Gnus (nnimap) shows "ghost" messages in summary buffer

Eric Abrahamsen-2
Ulrich Mueller <[hidden email]> writes:

>>>>>> On Thu, 09 May 2019, Eric Abrahamsen wrote:
>
>> Okay, here's what I was told on irc:
>
>> <tss> looks like it's the new feature that adds $HasAttachment or
>>       $HasNoAttachment flags to mails. they're actually supposed to be added
>>       only during mail delivery, and there's a setting needed to enable them:
>>       mail_attachment_detection_options = add-flags-on-save
>> <tss> but .. there's an "unintentional feature" :) that they also get added
>>       during some FETCH commands if they're not already there
>> <tss> but even without this, imap protocol allows sending FETCH FLAGS updates
>>       (or any updates really) as a response to any command. and with
>>       concurrent imap access dovecot does this.
>> <tss> for example if another client adds a \Seen flag, that same thing could
>>       happen
>> <jkt> either way, a client should really be able to process these
>>       "unexpeceted" FETCH updates, that's what the standard is about
>> --8<---------------cut here---------------end--------------->8---
>
>> Ultimately, the "real" fix would be to teach Gnus to handle all possible
>> responses from IMAP servers, probably using a proper parser.
>
>> But if you have access to the dovecot config in your case, you might try
>> removing the mail_attachment_detection_options setting above, if it's
>> set. If it's not set, then you might be getting the "unintentional
>> feature".
>
> Thank you very much for your help. Indeed the add-flags-on-save option
> was enabled in the config file. Gentoo's infrastructure team has agreed
> to disable it for now, which made the problem go away.

Well that was lucky.

>> I suppose in the interim I could mess with `nnimap-transform-headers'
>> and try to add some bookkeeping so that multiple FETCH responses for
>> the same article UID get merged together. On the other hand, this
>> feature only seems to relate to has/hasnoattachment flags, which Gnus
>> doesn't handle anyway -- we could safely drop those lines altogether.
>
>> (Though it sure would be nice to handle hasattachment flags, that's
>> something that users have requested in the past...)
>
> Maybe drop the extra FETCH responses for now, and add proper bookkeeping
> later when implementing the requested feature?

Okay, I'll work something up in the next week that goes halfway with
this.

Glad it's partially fixed.

Eric



Reply | Threaded
Open this post in threaded view
|

bug#35443: 27.0.50; Gnus (nnimap) shows "ghost" messages in summary buffer

Lars Ingebrigtsen
Eric Abrahamsen <[hidden email]> writes:

>> Maybe drop the extra FETCH responses for now, and add proper bookkeeping
>> later when implementing the requested feature?
>
> Okay, I'll work something up in the next week that goes halfway with
> this.

Did you get any further in fixing this nnimap parsing bug?

--
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no



Reply | Threaded
Open this post in threaded view
|

bug#35443: 27.0.50; Gnus (nnimap) shows "ghost" messages in summary buffer

Eric Abrahamsen-2
Lars Ingebrigtsen <[hidden email]> writes:

> Eric Abrahamsen <[hidden email]> writes:
>
>>> Maybe drop the extra FETCH responses for now, and add proper bookkeeping
>>> later when implementing the requested feature?
>>
>> Okay, I'll work something up in the next week that goes halfway with
>> this.
>
> Did you get any further in fixing this nnimap parsing bug?

No I didn't! I looked at it for a while, had an internal conflict about
just dropping the lines vs actually doing something with them, imagined
all the work that would go into a more fully-featured parser, then got
distracted and forgot about it.

I still hope that in some distant future we could have a real parser
consuming these buffers. So far as I know the only in-emacs options are
in cedet -- wisent and the other one -- but I've never been able to make
them work. And now it sounds like cedet might not even stay in-tree?

Anyway, I'll make a real todo for dropping the lines, though it seems a
shame...

Eric



Reply | Threaded
Open this post in threaded view
|

bug#35443: 27.0.50; Gnus (nnimap) shows "ghost" messages in summary buffer

Eric Abrahamsen-2
In reply to this post by Ulrich Mueller-2
Lars Ingebrigtsen <[hidden email]> writes:

> Eric Abrahamsen <[hidden email]> writes:
>
>>> Maybe drop the extra FETCH responses for now, and add proper bookkeeping
>>> later when implementing the requested feature?
>>
>> Okay, I'll work something up in the next week that goes halfway with
>> this.
>
> Did you get any further in fixing this nnimap parsing bug?

Here's a whack at it. I tried to make sure that it would handle unwanted
FETCH responses whether they came before or after (or in the middle of)
the wanted FETCH responses, but I'm not in love with checking the header
regexp this way.

Because this IMAP server feature is very closely focused on adding a
flag in case of attachment (and because Gnus never explicitly requests
this flag, though I'd sure like to in the future), another more targeted
approach would be to simply delete any lines containing
$Has\(No\)?Attachment, assuming that these FETCH responses will only
take up one line.

I've configured my local Dovecot with the offending setting, and will
test it out for a bit.

Eric


imap-header-transform-fix.diff (3K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

bug#35443: 27.0.50; Gnus (nnimap) shows "ghost" messages in summary buffer

Lars Ingebrigtsen
In reply to this post by Eric Abrahamsen-2
Eric Abrahamsen <[hidden email]> writes:

> No I didn't! I looked at it for a while, had an internal conflict about
> just dropping the lines vs actually doing something with them, imagined
> all the work that would go into a more fully-featured parser, then got
> distracted and forgot about it.

OK, I'll take a look at it...

> I still hope that in some distant future we could have a real parser
> consuming these buffers. So far as I know the only in-emacs options are
> in cedet -- wisent and the other one -- but I've never been able to make
> them work. And now it sounds like cedet might not even stay in-tree?

I had to add a parsing feature to wisent, and that was kinda more
painful than you'd expect.  :-/

But I haven't heard anything about dumping cedet from the tree.  Is that
the plan?

--
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no



Reply | Threaded
Open this post in threaded view
|

bug#35443: 27.0.50; Gnus (nnimap) shows "ghost" messages in summary buffer

Lars Ingebrigtsen
In reply to this post by Eric Abrahamsen-2
Eric Abrahamsen <[hidden email]> writes:

>> Did you get any further in fixing this nnimap parsing bug?
>
> Here's a whack at it. I tried to make sure that it would handle unwanted
> FETCH responses whether they came before or after (or in the middle of)
> the wanted FETCH responses, but I'm not in love with checking the header
> regexp this way.

Well, I think it's OK...

> Because this IMAP server feature is very closely focused on adding a
> flag in case of attachment (and because Gnus never explicitly requests
> this flag, though I'd sure like to in the future), another more targeted
> approach would be to simply delete any lines containing
> $Has\(No\)?Attachment, assuming that these FETCH responses will only
> take up one line.

That sounds a bit brittle -- I'm sure there'll be other extensions like
this in the future to the IMAP protocol.

> I've configured my local Dovecot with the offending setting, and will
> test it out for a bit.

Great!

--
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no



Reply | Threaded
Open this post in threaded view
|

bug#35443: 27.0.50; Gnus (nnimap) shows "ghost" messages in summary buffer

Eric Abrahamsen-2
In reply to this post by Lars Ingebrigtsen

On 06/23/19 14:13 PM, Lars Ingebrigtsen wrote:

> Eric Abrahamsen <[hidden email]> writes:
>
>> No I didn't! I looked at it for a while, had an internal conflict about
>> just dropping the lines vs actually doing something with them, imagined
>> all the work that would go into a more fully-featured parser, then got
>> distracted and forgot about it.
>
> OK, I'll take a look at it...
>
>> I still hope that in some distant future we could have a real parser
>> consuming these buffers. So far as I know the only in-emacs options are
>> in cedet -- wisent and the other one -- but I've never been able to make
>> them work. And now it sounds like cedet might not even stay in-tree?
>
> I had to add a parsing feature to wisent, and that was kinda more
> painful than you'd expect.  :-/

Oh, I thought that was most of the point of wisent to begin with. No
wonder I couldn't get it to do anything. In theory, do you have any
recommendations in the parsing direction.

> But I haven't heard anything about dumping cedet from the tree. Is that
> the plan?

No, I was skimming your threads about cleaning up the build process and
got the impression that it was unmaintained. I probably misinterpreted.

On 06/23/19 14:23 PM, Lars Ingebrigtsen wrote:

> Eric Abrahamsen <[hidden email]> writes:
>
>>> Did you get any further in fixing this nnimap parsing bug?
>>
>> Here's a whack at it. I tried to make sure that it would handle unwanted
>> FETCH responses whether they came before or after (or in the middle of)
>> the wanted FETCH responses, but I'm not in love with checking the header
>> regexp this way.
>
> Well, I think it's OK...

Cool.

>> Because this IMAP server feature is very closely focused on adding a
>> flag in case of attachment (and because Gnus never explicitly requests
>> this flag, though I'd sure like to in the future), another more targeted
>> approach would be to simply delete any lines containing
>> $Has\(No\)?Attachment, assuming that these FETCH responses will only
>> take up one line.
>
> That sounds a bit brittle -- I'm sure there'll be other extensions like
> this in the future to the IMAP protocol.

Sure, I'll stick with this.



Reply | Threaded
Open this post in threaded view
|

bug#35443: 27.0.50; Gnus (nnimap) shows "ghost" messages in summary buffer

Lars Ingebrigtsen
Eric Abrahamsen <[hidden email]> writes:

>> I had to add a parsing feature to wisent, and that was kinda more
>> painful than you'd expect.  :-/
>
> Oh, I thought that was most of the point of wisent to begin with. No
> wonder I couldn't get it to do anything. In theory, do you have any
> recommendations in the parsing direction.

Yes, that's the point of wisent, but as with any lex-like thing I've
dealt with, it's...  painful.  :-)  At least to me.

>> But I haven't heard anything about dumping cedet from the tree. Is that
>> the plan?
>
> No, I was skimming your threads about cleaning up the build process and
> got the impression that it was unmaintained. I probably misinterpreted.

We are all the maintainer of cedet now.  :-)

--
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no



12