Quantcast

make-thread with lambda form instead of function symbol

classic Classic list List threaded Threaded
17 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

make-thread with lambda form instead of function symbol

Eric Abrahamsen-2
Okay, one more thread question...

I'm trying to do something that seems like it would be a normal use
case: spawn a series of threads which call the same function using
different external processes. Practically what this means is that I want
to pass a function-plus-argument form to make-thread, not a function
symbol. Something like:

(let* ((results)
       (sources '(source1 source2))
       (threads
        (mapcar
         (lambda (s)
           (make-thread
            (funcall
             (lambda ()
               (push (get-stuff-from-source s) results)))))
         sources)))
  (mapc #'thread-join threads)
  results)

The (funcall (lambda () thing was the only way I could get anything but
nil out of the thread functions. I think I'm fooling myself, though: so
far as I can tell, `get-stuff-from-source' is fully evaluated before the
thread is made, and nothing at all happens during the #'thread-join
loop.

Is it possible to give make-thread anything but a function symbol? The
only other thing I could think of was looping over the sources and
making ad-hoc symbols:

(fset (make-symbol (format "%s-dummy-function" (source-name source)))
      (lambda () (push (get-stuff-from-source source) results)))

That seems ugly, but perhaps not that bad.

Thanks in advance,
Eric


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: make-thread with lambda form instead of function symbol

npostavs
On Sun, Apr 16, 2017 at 12:05 PM, Eric Abrahamsen
<[hidden email]> wrote:

> I'm trying to do something that seems like it would be a normal use
> case: spawn a series of threads which call the same function using
> different external processes. Practically what this means is that I want
> to pass a function-plus-argument form to make-thread, not a function
> symbol. Something like:
>
> (let* ((results)
>        (sources '(source1 source2))
>        (threads
>         (mapcar
>          (lambda (s)
>            (make-thread
>             (funcall
>              (lambda ()
>                (push (get-stuff-from-source s) results)))))
>          sources)))
>   (mapc #'thread-join threads)
>   results)
>
> The (funcall (lambda () thing was the only way I could get anything but
> nil out of the thread functions. I think I'm fooling myself, though: so
> far as I can tell, `get-stuff-from-source' is fully evaluated before the
> thread is made, and nothing at all happens during the #'thread-join
> loop.

Just drop the funcall: (make-thread (lambda () ...)) should do what
you want, assuming you have lexical-binding set (if not, you can
construct a lambda-list with backquote or similar).

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: make-thread with lambda form instead of function symbol

Eric Abrahamsen-2
Noam Postavsky <[hidden email]> writes:

> On Sun, Apr 16, 2017 at 12:05 PM, Eric Abrahamsen
> <[hidden email]> wrote:
>> I'm trying to do something that seems like it would be a normal use
>> case: spawn a series of threads which call the same function using
>> different external processes. Practically what this means is that I want
>> to pass a function-plus-argument form to make-thread, not a function
>> symbol. Something like:
>>
>> (let* ((results)
>>        (sources '(source1 source2))
>>        (threads
>>         (mapcar
>>          (lambda (s)
>>            (make-thread
>>             (funcall
>>              (lambda ()
>>                (push (get-stuff-from-source s) results)))))
>>          sources)))
>>   (mapc #'thread-join threads)
>>   results)
>>
>> The (funcall (lambda () thing was the only way I could get anything but
>> nil out of the thread functions. I think I'm fooling myself, though: so
>> far as I can tell, `get-stuff-from-source' is fully evaluated before the
>> thread is made, and nothing at all happens during the #'thread-join
>> loop.
>
> Just drop the funcall: (make-thread (lambda () ...)) should do what
> you want, assuming you have lexical-binding set (if not, you can
> construct a lambda-list with backquote or similar).

Thanks for the response. The first thing I learned here was that
lexical-binding was nil in *scratch*, but t in the source file I'm
writing -- not a great testing setup.

I'm still not seeing the results I expect, though. Here's a dumb test:

(setq lexical-binding t)

(let ((threads
       (mapcar
        (lambda (el)
          (make-thread
           (lambda ()
             (push (cl-incf el) results))))
        '(1 2 3)))
      results)
  (mapc #'thread-join threads)
  results)

This gives me nil.

(Incidentally, if I put this in a function and edebug it, it tells me
edebug will stop at the next break point, and then enters a level of
recursive editing I can't escape from: C-M-c gives me "No catch for tag:
exit, nil".)

Should the above example work?

Thanks again,
Eric


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: make-thread with lambda form instead of function symbol

npostavs
On Sun, Apr 16, 2017 at 2:11 PM, Eric Abrahamsen
<[hidden email]> wrote:

>
> (setq lexical-binding t)
>
> (let ((threads
>        (mapcar
>         (lambda (el)
>           (make-thread
>            (lambda ()
>              (push (cl-incf el) results))))
>         '(1 2 3)))
>       results)
>   (mapc #'thread-join threads)
>   results)
>
> This gives me nil.
>
> (Incidentally, if I put this in a function and edebug it, it tells me
> edebug will stop at the next break point, and then enters a level of
> recursive editing I can't escape from: C-M-c gives me "No catch for tag:
> exit, nil".)
>
> Should the above example work?

No, check the compile warnings:

a.el:7:33:Warning: reference to free variable ‘results’
a.el:9:7:Warning: assignment to free variable ‘results’

Not sure about the edebug thing, probably it doesn't handle
cross-thread stepping.

The below returns (2 3 4) or sometimes (3 4 2).

(let* ((results nil)
       (threads
        (mapcar
         (lambda (el)
           (make-thread
            (lambda ()
              (push (cl-incf el) results))))
         '(1 2 3))))
  (mapc #'thread-join threads)
  results)

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: make-thread with lambda form instead of function symbol

Eric Abrahamsen-2
Noam Postavsky <[hidden email]> writes:

> On Sun, Apr 16, 2017 at 2:11 PM, Eric Abrahamsen
> <[hidden email]> wrote:
>>
>> (setq lexical-binding t)
>>
>> (let ((threads
>>        (mapcar
>>         (lambda (el)
>>           (make-thread
>>            (lambda ()
>>              (push (cl-incf el) results))))
>>         '(1 2 3)))
>>       results)
>>   (mapc #'thread-join threads)
>>   results)
>>
>> This gives me nil.
>>
>> (Incidentally, if I put this in a function and edebug it, it tells me
>> edebug will stop at the next break point, and then enters a level of
>> recursive editing I can't escape from: C-M-c gives me "No catch for tag:
>> exit, nil".)
>>
>> Should the above example work?
>
> No, check the compile warnings:
>
> a.el:7:33:Warning: reference to free variable ‘results’
> a.el:9:7:Warning: assignment to free variable ‘results’

Aha! That makes sense now that I see it. The thread function starts
executing immediately, so of course it needs to be able to see
"results". Thanks for that.

> Not sure about the edebug thing, probably it doesn't handle
> cross-thread stepping.

I can imagine it would be complicated.

Thanks again,
Eric


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: make-thread with lambda form instead of function symbol

Eric Abrahamsen-2
In reply to this post by Eric Abrahamsen-2
Eric Abrahamsen <[hidden email]> writes:

> Okay, one more thread question...

Well, here's the long story: I'm messing with Gnus's nnir search
routines, trying to make a general search query language that can be
translated into backend-specific queries depending on which groups
you've selected to search. The idea is you could mark both an IMAP group
and an nnmaildir group (indexed by notmuch), enter a single search
query, the query would be transformed into backend-appropriate versions,
and you'd get one search group containing messages returned by backends.

So then I figured: if we're searching different backends at once, and
each backend does its heavy lifting in an external process, then threads
would allow those processes to do their work concurrently, and results
could come back faster.

(If I can get this to work, I hope to look at getting Gnus to fetch new
news concurrently, as well.)

But! My current implementation is causing Emacs to segfault. The actual
function looks like this:

(defun nnir-run-query (specs)
  (let* ((results [])
         (threads
          (mapcar
           (lambda (x)
             (let* ((server (car x))
                    (search-engine (nnir-server-to-search-engine server)))
               (make-thread
                (lambda ()
                  (setq results
                        (vconcat
                         (nnir-run-search
                          search-engine
                          server
                          (cdr (assq 'nnir-query-spec specs))
                          (cadr x))
                         results))))))
           (cdr (assq 'nnir-group-spec specs)))))
    (mapc #'thread-join threads)
    results))

I'm testing with a single IMAP server query (ie, only one thread is
being created). The nnir-run-search function for the IMAP backend ends
up calling nnimap-send-command--> nnimap-wait-for-response-->
nnheader-accept-process-output--> accept-process-output, so I'm
expecting that's where the thread yields.

I'm attaching the traceback here. Dunno if it's something I've done
wrong...

Thanks,
Eric


log.txt (6K) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: make-thread with lambda form instead of function symbol

npostavs
On Sun, Apr 16, 2017 at 9:12 PM, Eric Abrahamsen
<[hidden email]> wrote:

>
> (defun nnir-run-query (specs)
>   (let* ((results [])
>          (threads
>           (mapcar
>            (lambda (x)
>              (let* ((server (car x))
>                     (search-engine (nnir-server-to-search-engine server)))
>                (make-thread
>                 (lambda ()
>                   (setq results
>                         (vconcat
>                          (nnir-run-search
>                           search-engine
>                           server
>                           (cdr (assq 'nnir-query-spec specs))
>                           (cadr x))
>                          results))))))
>            (cdr (assq 'nnir-group-spec specs)))))
>     (mapc #'thread-join threads)
>     results))
>
> I'm testing with a single IMAP server query (ie, only one thread is
> being created). The nnir-run-search function for the IMAP backend ends
> up calling nnimap-send-command--> nnimap-wait-for-response-->
> nnheader-accept-process-output--> accept-process-output, so I'm
> expecting that's where the thread yields.
>
> I'm attaching the traceback here. Dunno if it's something I've done
> wrong...

> #4  0x000000000069f953 in Faccept_process_output (process=..., seconds=..., millisec=..., just_this_one=...) at process.c:4566

Is line process.c:4566 this one?

    error ("Attempt to accept output from process %s locked to thread %s",
           SDATA (proc->name), SDATA (XTHREAD (proc->thread)->name));

I guess that indicates you might be doing something wrong (though
obviously Emacs shouldn't be crashing)

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: make-thread with lambda form instead of function symbol

Eric Abrahamsen-2
Noam Postavsky <[hidden email]> writes:

> On Sun, Apr 16, 2017 at 9:12 PM, Eric Abrahamsen
> <[hidden email]> wrote:
>>
>> (defun nnir-run-query (specs)
>>   (let* ((results [])
>>          (threads
>>           (mapcar
>>            (lambda (x)
>>              (let* ((server (car x))
>>                     (search-engine (nnir-server-to-search-engine server)))
>>                (make-thread
>>                 (lambda ()
>>                   (setq results
>>                         (vconcat
>>                          (nnir-run-search
>>                           search-engine
>>                           server
>>                           (cdr (assq 'nnir-query-spec specs))
>>                           (cadr x))
>>                          results))))))
>>            (cdr (assq 'nnir-group-spec specs)))))
>>     (mapc #'thread-join threads)
>>     results))
>>
>> I'm testing with a single IMAP server query (ie, only one thread is
>> being created). The nnir-run-search function for the IMAP backend ends
>> up calling nnimap-send-command--> nnimap-wait-for-response-->
>> nnheader-accept-process-output--> accept-process-output, so I'm
>> expecting that's where the thread yields.
>>
>> I'm attaching the traceback here. Dunno if it's something I've done
>> wrong...
>
>> #4  0x000000000069f953 in Faccept_process_output (process=..., seconds=..., millisec=..., just_this_one=...) at process.c:4566
>
> Is line process.c:4566 this one?
>
>     error ("Attempt to accept output from process %s locked to thread %s",
>            SDATA (proc->name), SDATA (XTHREAD (proc->thread)->name));
>
> I guess that indicates you might be doing something wrong (though
> obviously Emacs shouldn't be crashing)

So perhaps an existing process can't be "moved" to another thread?
Tomorrow I'll set up a notmuch-indexed server and try that -- in that
case the process should be invoked and concluded within a single
make-thread call, so maybe it won't raise this error.

But the segfault here seems to be coming from a borked attempt to
represent either the process or the thread as a string....


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: make-thread with lambda form instead of function symbol

Eli Zaretskii
In reply to this post by Eric Abrahamsen-2
> From: Eric Abrahamsen <[hidden email]>
> Date: Sun, 16 Apr 2017 18:12:17 -0700
>
> #0  0x000000000058879c in terminate_due_to_signal (sig=6, backtrace_limit=2147483647) at emacs.c:363
> #1  0x000000000061dbf4 in die (msg=0x74e3fd "STRINGP (a)", file=0x74e330 "lisp.h", line=1276) at alloc.c:7339
> #2  0x0000000000582dfd in XSTRING (a=...) at lisp.h:1276
> #3  0x0000000000582e44 in SDATA (string=...) at lisp.h:1327
> #4  0x000000000069f953 in Faccept_process_output (process=..., seconds=..., millisec=..., just_this_one=...) at process.c:4566

Please try the latest master, I hope I fixed this blunder.

In the future, when reporting backtraces from multithreaded Lisp
programs, please always say "thread apply all bt", to produce
backtraces of all the threads, not just the current one.  Otherwise,
some important information could be missing.

Thanks.

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: make-thread with lambda form instead of function symbol

Andrew Cohen
In reply to this post by Eric Abrahamsen-2


It would be great to get the searches done concurrently. This is the
last serious (IMHO) problem with gnus searches---the searching part can
take a long time. The current implementation tries to collect everything
to minimize connections to the backends (i.e. searching multiple groups
on a single backend should use a single connection) but even imap
searching gets to be a pain when several imap servers are involved.

Just a side comment: the existing nnir-run-query handles multiple groups
with different backends mostly just fine (albeit searching sequentially
rather than concurrently). The limitation is that the search query must
be common to the different backends (this is the big issue that your
general search query language would fix). I (used to) routinely combine
gmane, namazu, and imap groups in my searches (used to since I stopped
using namazu long ago and gmane search is now defunct :().

And a side-side comment: if the different backends allow different
criteria then search groups from different backends will prompt for
different criteria for each backend. Cumbersome but occasionally
helpful.

Andy


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: make-thread with lambda form instead of function symbol

Eli Zaretskii
In reply to this post by Eric Abrahamsen-2
> From: Eric Abrahamsen <[hidden email]>
> Date: Sun, 16 Apr 2017 20:45:19 -0700
>
> >> (defun nnir-run-query (specs)
> >>   (let* ((results [])
> >>          (threads
> >>           (mapcar
> >>            (lambda (x)
> >>              (let* ((server (car x))
> >>                     (search-engine (nnir-server-to-search-engine server)))
> >>                (make-thread
> >>                 (lambda ()
> >>                   (setq results
> >>                         (vconcat
> >>                          (nnir-run-search
> >>                           search-engine
> >>                           server
> >>                           (cdr (assq 'nnir-query-spec specs))
> >>                           (cadr x))
> >>                          results))))))
> >>            (cdr (assq 'nnir-group-spec specs)))))
> >>     (mapc #'thread-join threads)
> >>     results))
> >>
> >> I'm testing with a single IMAP server query (ie, only one thread is
> >> being created). The nnir-run-search function for the IMAP backend ends
> >> up calling nnimap-send-command--> nnimap-wait-for-response-->
> >> nnheader-accept-process-output--> accept-process-output, so I'm
> >> expecting that's where the thread yields.

In this chain of calls, where's the backend process started?

> >     error ("Attempt to accept output from process %s locked to thread %s",
> >            SDATA (proc->name), SDATA (XTHREAD (proc->thread)->name));
> >
> > I guess that indicates you might be doing something wrong (though
> > obviously Emacs shouldn't be crashing)
>
> So perhaps an existing process can't be "moved" to another thread?

What does it mean "moved" in this context?  How did you try to "move"
a process to another thread?

The ELisp manual says:

  ... by default a process is locked to the thread
  that created it.  When a process is locked to a thread, output from the
  process can only be accepted by that thread.

If you want to be able to accept process output from a thread other
than the one which created it, you need to call set-process-thread.

> But the segfault here seems to be coming from a borked attempt to
> represent either the process or the thread as a string....

Yes.  Should be fixed now.  But my advice is to always give your
thread meaningful names, as doing so will help you with debugging
(unless you are very good in remembering the hex addresses of your
thread objects ;-).

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: make-thread with lambda form instead of function symbol

Eric Abrahamsen-2
Eli Zaretskii <[hidden email]> writes:

>> From: Eric Abrahamsen <[hidden email]>
>> Date: Sun, 16 Apr 2017 20:45:19 -0700
>>
>> >> (defun nnir-run-query (specs)
>> >>   (let* ((results [])
>> >>          (threads
>> >>           (mapcar
>> >>            (lambda (x)
>> >>              (let* ((server (car x))
>> >>                     (search-engine (nnir-server-to-search-engine server)))
>> >>                (make-thread
>> >>                 (lambda ()
>> >>                   (setq results
>> >>                         (vconcat
>> >>                          (nnir-run-search
>> >>                           search-engine
>> >>                           server
>> >>                           (cdr (assq 'nnir-query-spec specs))
>> >>                           (cadr x))
>> >>                          results))))))
>> >>            (cdr (assq 'nnir-group-spec specs)))))
>> >>     (mapc #'thread-join threads)
>> >>     results))
>> >>
>> >> I'm testing with a single IMAP server query (ie, only one thread is
>> >> being created). The nnir-run-search function for the IMAP backend ends
>> >> up calling nnimap-send-command--> nnimap-wait-for-response-->
>> >> nnheader-accept-process-output--> accept-process-output, so I'm
>> >> expecting that's where the thread yields.
>
> In this chain of calls, where's the backend process started?
>
>> >     error ("Attempt to accept output from process %s locked to thread %s",
>> >            SDATA (proc->name), SDATA (XTHREAD (proc->thread)->name));
>> >
>> > I guess that indicates you might be doing something wrong (though
>> > obviously Emacs shouldn't be crashing)
>>
>> So perhaps an existing process can't be "moved" to another thread?
>
> What does it mean "moved" in this context?  How did you try to "move"
> a process to another thread?
>
> The ELisp manual says:
>
>   ... by default a process is locked to the thread
>   that created it.  When a process is locked to a thread, output from the
>   process can only be accepted by that thread.
>
> If you want to be able to accept process output from a thread other
> than the one which created it, you need to call set-process-thread.

Okay that certainly explains it -- I hadn't found that part of the
manual. The IMAP process in question was long-running, created well
before this thread was, so I was doing exactly the wrong thing. (All I
meant by "move" was starting the process in one thread, and accepting
its output in another.)

>> But the segfault here seems to be coming from a borked attempt to
>> represent either the process or the thread as a string....
>
> Yes.  Should be fixed now.  But my advice is to always give your
> thread meaningful names, as doing so will help you with debugging
> (unless you are very good in remembering the hex addresses of your
> thread objects ;-).

Segfault's gone, thank you. Now to figure out where to stick the call to
`set-process-thread'.

And thanks for the "thread apply all bt" trick, I'll remember that.

Eric


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: make-thread with lambda form instead of function symbol

Eric Abrahamsen-2
Eric Abrahamsen <[hidden email]> writes:

> Eli Zaretskii <[hidden email]> writes:

[...]

>> What does it mean "moved" in this context?  How did you try to "move"
>> a process to another thread?
>>
>> The ELisp manual says:
>>
>>   ... by default a process is locked to the thread
>>   that created it.  When a process is locked to a thread, output from the
>>   process can only be accepted by that thread.
>>
>> If you want to be able to accept process output from a thread other
>> than the one which created it, you need to call set-process-thread.
>
> Okay that certainly explains it -- I hadn't found that part of the
> manual. The IMAP process in question was long-running, created well
> before this thread was, so I was doing exactly the wrong thing. (All I
> meant by "move" was starting the process in one thread, and accepting
> its output in another.)
>
> Segfault's gone, thank you. Now to figure out where to stick the call to
> `set-process-thread'.

I tried unlocking all IMAP processes early on, when they're first
created, and so far it seems to be working great!


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: make-thread with lambda form instead of function symbol

Eric Abrahamsen-2
In reply to this post by Andrew Cohen
Andrew Cohen <[hidden email]> writes:

> It would be great to get the searches done concurrently. This is the
> last serious (IMHO) problem with gnus searches---the searching part can
> take a long time. The current implementation tries to collect everything
> to minimize connections to the backends (i.e. searching multiple groups
> on a single backend should use a single connection) but even imap
> searching gets to be a pain when several imap servers are involved.

Subjectively, the thread trick seems to really speed things up. I've
also been looking at some of the IMAP extensions to improve
single-server search -- MULTISEARCH could speed things up significantly,
and wouldn't be hard to implement. I'm not sure about SEARCHRES. It
doesn't look like it actually saves much time, and would be very hard to
integrate with the structure of Gnus searches.

> Just a side comment: the existing nnir-run-query handles multiple groups
> with different backends mostly just fine (albeit searching sequentially
> rather than concurrently). The limitation is that the search query must
> be common to the different backends (this is the big issue that your
> general search query language would fix). I (used to) routinely combine
> gmane, namazu, and imap groups in my searches (used to since I stopped
> using namazu long ago and gmane search is now defunct :().
>
> And a side-side comment: if the different backends allow different
> criteria then search groups from different backends will prompt for
> different criteria for each backend. Cumbersome but occasionally
> helpful.

Actually, in the implementation I'm working on now, I've removed the
additional criteria. The general query language makes selecting imap
keys unnecessary, the gmane "author" criteria can simply be transformed
from the "from" keyword (not to mention that this support is now
theoretical!), and all the other backend criteria specify groups: can't
we just take that from the group search spec?

I'll be happy to re-instate criteria if they really turn out to be
necessary, but I'd like to try to do away with them if possible.

Eric


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: make-thread with lambda form instead of function symbol

Andrew Cohen


>>>>> "Eric" == Eric Abrahamsen <[hidden email]> writes:

    >> And a side-side comment: if the different backends allow
    >> different criteria then search groups from different backends
    >> will prompt for different criteria for each backend. Cumbersome
    >> but occasionally helpful.

    Eric> Actually, in the implementation I'm working on now, I've
    Eric> removed the additional criteria. The general query language
    Eric> makes selecting imap keys unnecessary, the gmane "author"
    Eric> criteria can simply be transformed from the "from" keyword
    Eric> (not to mention that this support is now theoretical!), and
    Eric> all the other backend criteria specify groups: can't we just
    Eric> take that from the group search spec?

Eliminating the criteria sounds good---it was present just to deal with
the non-uniformity of search query formats across different  backends,
so once the universal language is in place it should go away forever.

Best,
Andy


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: make-thread with lambda form instead of function symbol

Eric Abrahamsen-2
Andrew Cohen <[hidden email]> writes:

>>>>>> "Eric" == Eric Abrahamsen <[hidden email]> writes:
>
>     >> And a side-side comment: if the different backends allow
>     >> different criteria then search groups from different backends
>     >> will prompt for different criteria for each backend. Cumbersome
>     >> but occasionally helpful.
>
>     Eric> Actually, in the implementation I'm working on now, I've
>     Eric> removed the additional criteria. The general query language
>     Eric> makes selecting imap keys unnecessary, the gmane "author"
>     Eric> criteria can simply be transformed from the "from" keyword
>     Eric> (not to mention that this support is now theoretical!), and
>     Eric> all the other backend criteria specify groups: can't we just
>     Eric> take that from the group search spec?
>
> Eliminating the criteria sounds good---it was present just to deal with
> the non-uniformity of search query formats across different  backends,
> so once the universal language is in place it should go away forever.

Excellent, that's good to hear. I'll keep working on this. Oddly, the
main slowdown now is simply finding good documentation on the various
engines' search syntax. I'll also see if there are any other search
engines worth adding.


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: make-thread with lambda form instead of function symbol

Eric Abrahamsen-2
In reply to this post by Andrew Cohen
Andrew Cohen <[hidden email]> writes:

> It would be great to get the searches done concurrently. This is the
> last serious (IMHO) problem with gnus searches---the searching part can
> take a long time. The current implementation tries to collect everything
> to minimize connections to the backends (i.e. searching multiple groups
> on a single backend should use a single connection) but even imap
> searching gets to be a pain when several imap servers are involved.

Bah, this isn't quite working. Searching more than one IMAP server
causes weird results and/or hangs. Searching other backends, or one IMAP
server plus other backends, seems to work okay. I have a hunch that this
comes from the way Gnus backends share the nntp-server-buffer variable
and swap symbols around -- I think the processes might be stepping on
each others' toes from thread to thread. nnimap-wait-for-response loops
accept-process-output on a 0.01 second timeout, so I can imagine that
two or more threads could be fighting over the meaning of
nntp-server-buffer.

Anyway, I've gone down enough garden paths here. I'm going to leave
things set up for threading, but not actually use threads for now. Once
the rest of the stuff is in place there will be time to figure out
what's going wrong.

Eric


Loading...