Re: [Emacs-diffs] scratch/new-flex-completion-style 2c75775 2/2: Score, sort and annotate flex-style completions according to match tightness

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: [Emacs-diffs] scratch/new-flex-completion-style 2c75775 2/2: Score, sort and annotate flex-style completions according to match tightness

Dmitry Gutov
On 03.02.2019 02:28, Jo�o T�vora wrote:
> +          ;; Sort again in case the completion style has propertized
> +          ;; completions with a 'completion-style-sort-order.
> +          (setq completions
> +                (sort completions
> +                      (lambda (a b)
> +                        (let ((va (get-text-property 0 'completion-style-sort-order a))
> +                              (vb (get-text-property 0 'completion-style-sort-order b)))

Shouldn't we be concerned that this overrides display-sort-function
returned by the completion table?

Maybe some sort of customizable mechanism is in order, similar to
completion-styles.

Or more simply, we just leave it to the completion table's author to set
display-sort-function to a special value that makes use of these
properties. It's not like setting it to a different value is going to
make sense in this case.

> +          ;; Annotate completions with <...>
> +          ;; the completion style annotation.

Is this for debugging?

Reply | Threaded
Open this post in threaded view
|

Re: [Emacs-diffs] scratch/new-flex-completion-style 2c75775 2/2: Score, sort and annotate flex-style completions according to match tightness

João Távora
Dmitry Gutov <[hidden email]> writes:

> On 03.02.2019 02:28, Jo�o T�vora wrote:
>> +          ;; Sort again in case the completion style has propertized
>> +          ;; completions with a 'completion-style-sort-order.
>> +          (setq completions
>> +                (sort completions
>> +                      (lambda (a b)
>> +                        (let ((va (get-text-property 0 'completion-style-sort-order a))
>> +                              (vb (get-text-property 0 'completion-style-sort-order b)))
>
> Shouldn't we be concerned that this overrides display-sort-function
> returned by the completion table?

Perhaps.  But it only does so when the completion style explicitly
wanted it.  And even so, the sort order should be stable, i.e. if the
completion style puts numerically equal values in two completions, they
should retain the backend's order.  And, obviously, it it doesn't put
any values in that property, then the backend's order is also retained.

In my experience, at least for the "flex" (also called "scatter", by the
way) completion style, the frontend's sorting takes precedence

> Maybe some sort of customizable mechanism is in order, similar to
> completion-styles.

It is already customizable: you don't have to use "flex" completion
style.  Let's first try it out and then see if we need _more_
customization.

> Or more simply, we just leave it to the completion table's author to
> set display-sort-function to a special value that makes use of these
> properties. It's not like setting it to a different value is going to
> make sense in this case.
>
>> +          ;; Annotate completions with <...>
>> +          ;; the completion style annotation.
>
> Is this for debugging?

At the moment, yes mostly.  But it could not be.  In SLY, I give users
an indication of how the scoring algorithm is working.  See it in
action:

https://raw.githubusercontent.com/joaotavora/sly/master/doc/animations/company-flex-completion.gif

João

Reply | Threaded
Open this post in threaded view
|

Re: [Emacs-diffs] scratch/new-flex-completion-style 2c75775 2/2: Score, sort and annotate flex-style completions according to match tightness

Dmitry Gutov
On 06.02.2019 13:09, João Távora wrote:

> Perhaps.  But it only does so when the completion style explicitly
> wanted it.  And even so, the sort order should be stable, i.e. if the
> completion style puts numerically equal values in two completions, they
> should retain the backend's order.  And, obviously, it it doesn't put
> any values in that property, then the backend's order is also retained.

To put it simply, we already have a way to indicate the completions
sorting: provide a particular property, or metadata. I'd prefer not to
couple completion style with sorting, because someone somewhere might
prefer to go without it (and use some other sorting options, e.g. by
frequency of use or mentions in the buffer).

> In my experience, at least for the "flex" (also called "scatter", by the
> way) completion style, the frontend's sorting takes precedence

Not sure what you mean by frontend here.

> It is already customizable: you don't have to use "flex" completion
> style.  Let's first try it out and then see if we need _more_
> customization.

I'd rather we start with the "more simply" alternative I've mentioned.
But others are welcome to add their opinions.

>> Is this for debugging?
>
> At the moment, yes mostly.  But it could not be.  In SLY, I give users
> an indication of how the scoring algorithm is working.  See it in
> action:
>
> https://raw.githubusercontent.com/joaotavora/sly/master/doc/animations/company-flex-completion.gif

It's kind of neat, but to be honest I don't see how these percentages
help a random user.

Even if they do, there's no need to couple this annotation addition to
the completion style. Just use a new annotation function that looks up
the text properties that the style adds, and adds them on.

Reply | Threaded
Open this post in threaded view
|

Re: [Emacs-diffs] scratch/new-flex-completion-style 2c75775 2/2: Score, sort and annotate flex-style completions according to match tightness

João Távora
Dmitry Gutov <[hidden email]> writes:

> On 06.02.2019 13:09, João Távora wrote:
>
>> Perhaps.  But it only does so when the completion style explicitly
>> wanted it.  And even so, the sort order should be stable, i.e. if the
>> completion style puts numerically equal values in two completions, they
>> should retain the backend's order.  And, obviously, it it doesn't put
>> any values in that property, then the backend's order is also retained.
>
> To put it simply, we already have a way to indicate the completions
> sorting: provide a particular property, or metadata.

Perhaps I'm missing something, and that is taiting all the remaining
discussion.  Aren't these properties/metadata thingies specific to the
completion table (files, buffers, code completions, etc)?

> I'd prefer not to couple completion style with sorting, because
> someone somewhere might prefer to go without it (and use some other
> sorting options, e.g. by frequency of use or mentions in the buffer).




>
>> In my experience, at least for the "flex" (also called "scatter", by the
>> way) completion style, the frontend's sorting takes precedence
>
> Not sure what you mean by frontend here.

I meant the completion style.  Though technically the completion style
is a degree short of a frontend, because it is a way to select the
completions collected by the backend (the completion table in Emacs
parlance) and pass them to the graphical frontend, be it icomplete's,
the *Completions* buffer or company.

In otherwords I should be able to mix and match

- frontends
- completion styles
- backends

Right?  Well all I'm saying is that, when using the flex/scatter
completion style, it's almost always, if not always always, a better
idea to stable-sort by flex-match score.

One good way to see this is to try my code with your company.el package,
which doesn't (hopefully _doesn't yet_) use the completion style's
sorting hints.  You can try it in Emacs Lisp. First do:

   (setq completion-styles '(flex))

Now go into the *scratch* buffer and type "undo". You'll see it's not
very useful: the first match you get is ":argument-precedence-order", a
legitimate match, for sure, but you probably meant something related to
the undo facilities.  And you have to scroll quite a bit before you to
"buffer-undo-list", which is buried deep in the many many b's that
match.

>> It is already customizable: you don't have to use "flex" completion
>> style.  Let's first try it out and then see if we need _more_
>> customization.
>
> I'd rather we start with the "more simply" alternative I've
> mentioned. But others are welcome to add their opinions.

Eventually, we can add a keybinding to resort exclusively according to
the table or exclusively according to the completion style.

>>> Is this for debugging?
>>
>> At the moment, yes mostly.  But it could not be.  In SLY, I give users
>> an indication of how the scoring algorithm is working.  See it in
>> action:
>>
>> https://raw.githubusercontent.com/joaotavora/sly/master/doc/animations/company-flex-completion.gif
>
> It's kind of neat, but to be honest I don't see how these percentages
> help a random user.

I can agree they should be improved.  It would be good to integrate some
prior art regarding flex string matching that has a good measure
of the match quality.  The one in the prototype is a empirical thing
that works quite OK for Common Lisp and myself.  `string-distance', for
instance, gives the the Levenshtein distance.  It's got a fancy name,
but it's not very useful for flex completion before

  (string-distance "foo" "barfoo") = (string-distance "foo" "frodos")

And we want "barfoo" to come first.  string-distance talks about
"deletions, insertions and substitutions required to transform a string
into another".  Perhaps if it measured also "move operations" it would
be a better measure.

Then instead of the percentage, we could show just the number of 1-char
editing operations, including moves, to turn the pattern into the
candidate.  That's a measure with actual meaning.

> Even if they do, there's no need to couple this annotation addition to
> the completion style. Just use a new annotation function that looks up
> the text properties that the style adds, and adds them on.

Again, this would only affect the specific completion table that I'm
writing this function for, right?  Or can I write metadata functions for
completion styles?

João

Reply | Threaded
Open this post in threaded view
|

Re: [Emacs-diffs] scratch/new-flex-completion-style 2c75775 2/2: Score, sort and annotate flex-style completions according to match tightness

Dmitry Gutov
On 06.02.2019 22:47, João Távora wrote:

> Perhaps I'm missing something, and that is taiting all the remaining
> discussion.  Aren't these properties/metadata thingies specific to the
> completion table (files, buffers, code completions, etc)?

metadata - yes, properties - no (those are extra elements at the end of
the list returned by completion-at-point-functions).

See completion--metadata and completion-extra-properties.

display-sort-function, for now, can only be a metadata element (though
that shouldn't be hard to change). annotation-function, however, can be
either.

So as things stand, the use of this completion style that'd see in mind,
is when a particular completion function returns a table that indicates
a particular category (for which the use of flex completion style is
enabled), and a particular display-sort-function in its metadata.

Consequently, a preexisting completion tables wouldn't be supported. But
like I said, it shouldn't be hard to change either.

> In otherwords I should be able to mix and match
>
> - frontends
> - completion styles
> - backends
>
> Right?  Well all I'm saying is that, when using the flex/scatter
> completion style, it's almost always, if not always always, a better
> idea to stable-sort by flex-match score.

I dunno, sorting takes CPU time anyways. Not sure a combination of
several sorting functions will frequently give a better result than just
one of them.

> One good way to see this is to try my code with your company.el package,
> which doesn't (hopefully _doesn't yet_) use the completion style's
> sorting hints.  You can try it in Emacs Lisp. First do:
>
>     (setq completion-styles '(flex))
>
> Now go into the *scratch* buffer and type "undo". You'll see it's not
> very useful: the first match you get is ":argument-precedence-order", a
> legitimate match, for sure, but you probably meant something related to
> the undo facilities.  And you have to scroll quite a bit before you to
> "buffer-undo-list", which is buried deep in the many many b's that
> match.

But then, if you enable company-sort-by-occurrence via
company-transformers, the completion style's sorting won't be visible
anymore anyway.

It's a question whether any other sorting approaches end up being
helpful, but maybe some hybrid functions will work, where both
occurrences and flex score are taken into account.

> Eventually, we can add a keybinding to resort exclusively according to
> the table or exclusively according to the completion style.

In the approach I'm thinking of, both options are the same sorting function.

> And we want "barfoo" to come first.  string-distance talks about
> "deletions, insertions and substitutions required to transform a string
> into another".  Perhaps if it measured also "move operations" it would
> be a better measure.

That's helpful to consider for the actual scores, and sorting.

> Then instead of the percentage, we could show just the number of 1-char
> editing operations, including moves, to turn the pattern into the
> candidate.  That's a measure with actual meaning.

And still, seeing it wouldn't help me if I'm just writing code. Sorry
for being blunt.

Sort by them, sure. But show the values to the user? Why?

>> Even if they do, there's no need to couple this annotation addition to
>> the completion style. Just use a new annotation function that looks up
>> the text properties that the style adds, and adds them on.
>
> Again, this would only affect the specific completion table that I'm
> writing this function for, right?  Or can I write metadata functions for
> completion styles?

The former, more or less. This creates a certain limitation, sure, but
it shouldn't be a problem for Eglot.

Reply | Threaded
Open this post in threaded view
|

Re: [Emacs-diffs] scratch/new-flex-completion-style 2c75775 2/2: Score, sort and annotate flex-style completions according to match tightness

Stefan Monnier
> See completion--metadata and completion-extra-properties.

But a completion style like `flex` can't affect either of those.
And I expect some people will want to put in their `completion-styles`,
so `flex` should work well for all/most completion tables and all/most
completion-at-point-functions (and calls to completing-read).

> I dunno, sorting takes CPU time anyways. Not sure a combination of several
> sorting functions will frequently give a better result than just one
> of them.

I think I agree here: the stability of the `flex` sort is likely to make
little difference in practice (more specifically, the sets of matches
which share the same flex-score will be sufficiently numerous and small
that the sorting within them is not significant).

> But then, if you enable company-sort-by-occurrence via company-transformers,
> the completion style's sorting won't be visible anymore anyway.

I think this is OK: the front-end can indeed override the
sorting chosen by the completion-style and the completion-table.
If the result is poor, it's the front-end's problem and it is in
a position to fix it.

> It's a question whether any other sorting approaches end up being helpful,
> but maybe some hybrid functions will work, where both occurrences and flex
> score are taken into account.

Not sure how that can be done in a modular way: how can we specify an
API where `flex` can provide its scores, `sort-by-occurrence` can
specify its scores, they don't know about each other, and yet the two
sets of scores get mixed in a useful way?


        Stefan


Reply | Threaded
Open this post in threaded view
|

Re: [Emacs-diffs] scratch/new-flex-completion-style 2c75775 2/2: Score, sort and annotate flex-style completions according to match tightness

João Távora
In reply to this post by Dmitry Gutov
Dmitry Gutov <[hidden email]> writes:

> Consequently, a preexisting completion tables wouldn't be
> supported. But like I said, it shouldn't be hard to change either.

Oh.  That's what I feared (I think) It would be nice to add something
seemingly orthogonal like a "completion style" to all completion UI's
and all tables, without needing to change the completion tables
(granted, it would be good to *also* not change the UI's, but I'm not
sure there a solution for that).

> I dunno, sorting takes CPU time anyways. Not sure a combination of
> several sorting functions will frequently give a better result than
> just one of them.

I dunno what you dunno.  When using flex/scatter it's *really* important
to sort by flex-match.  I suggested "stable sorting" because it's a
slightly more decent thing to completion tables.  And by sorting twice
it's not like we're changing the Big-O complexity or anything.

But I also don't mind a scheme where the style's sorting completely
takes over (and we save those CPU cycles).

> But then, if you enable company-sort-by-occurrence via
> company-transformers, the completion style's sorting won't be visible
> anymore anyway.

I don't know what company-transformers are.  On the one hand I suppose
when you want to sort by occurance, you ... want to sort by occurrence.
So if you use the symbol "foofooubarbarnbazbazdquuxquuxo" very often
you'll see it sort to the top for your "undo" example.  So maybe company
should take the style's sorting last here, dunno.

> It's a question whether any other sorting approaches end up being
> helpful, but maybe some hybrid functions will work, where both
> occurrences and flex score are taken into account.

In this case I would first sort by occurance and then style-score.
There's no easy way to make a hybrid function, so we must pick a
default.  For all the examples I've been given, it seems that
flex-sorting should always come last. Since flex is the first and only
style that carries some sorting idea with it, perhaps it's not a very
bad idea to give it _some_ priority by default.

>> Eventually, we can add a keybinding to resort exclusively according to
>> the table or exclusively according to the completion style.
>
> In the approach I'm thinking of, both options are the same sorting
> function.

How would you synthetize it?

> And still, seeing it wouldn't help me if I'm just writing code. Sorry
> for being blunt.

I'm a big boy, I can take blunt... :-)

> Sort by them, sure. But show the values to the user? Why?

In the general case, so the user has an idea _why_ something is being
automatically.  Humans generally gain confidence in algorithms if
they're shown human-comprehensible hints as to why it is is doing
something (humans generally also like this from other humans, which
explains why you ask me so many questions :-).

In this particular case, it's a very common idiom when displaying
tables/lists of something.  Perhaps in the "occurance sorting" field,
he/she will choose another word used less frequently for stylistic
purposes.

But, all that said, I'm not going to kill myself over the annotation,
nor do I think it even merits a customization option, though it's
probably not hard to do it anyway.

> The former, more or less. This creates a certain limitation, sure, but
> it shouldn't be a problem for Eglot.

Hmm? By far, Eglot isn't the only one thing I'm interested in enhancing
(actually, I've had few opportunities to actualy _use_ Eglot lately).

João


Reply | Threaded
Open this post in threaded view
|

Re: [Emacs-diffs] scratch/new-flex-completion-style 2c75775 2/2: Score, sort and annotate flex-style completions according to match tightness

Dmitry Gutov
In reply to this post by Stefan Monnier
On 12.02.2019 16:19, Stefan Monnier wrote:
>> See completion--metadata and completion-extra-properties.
>
> But a completion style like `flex` can't affect either of those.
> And I expect some people will want to put in their `completion-styles`,
> so `flex` should work well for all/most completion tables and all/most
> completion-at-point-functions (and calls to completing-read).

On the one hand, yes, but it's the abstraction we already have, and it
will allow flex to work properly at least with certain completion
tables. Which is a win.

On the other hand, a search for display-sort-function through Emacs's
sources seems to indicate that the design is kind of a failure: there
are only a couple places that set it to something, and in both cases
it's #'identity, meaning "do not re-sort".

So minibuffer.el might want to take a cue from Company and
company-transformers, to make this universal, we could add sorting as a
feature of completion styles, or as a separate user option, for maximum
flexibility. I'd simply like to avoid hardcoding extra sorting behavior,
like the patch proposes.

> I think I agree here: the stability of the `flex` sort is likely to make
> little difference in practice (more specifically, the sets of matches
> which share the same flex-score will be sufficiently numerous and small
> that the sorting within them is not significant).

Yep.

> Not sure how that can be done in a modular way: how can we specify an
> API where `flex` can provide its scores, `sort-by-occurrence` can
> specify its scores, they don't know about each other, and yet the two
> sets of scores get mixed in a useful way?

Conveying information via text properties seems like a viable approach.
We already use them to indicate matching characters, for instance.

Now, I wouldn't say it's terrible that sort-by-occurrence-and-score
would know specifically about flex (I don't think there's going to be a
lot of different styles in this flavor), but sure, like you proposed in
another email, the property can have a neutral name like completion-score.

To emphasize: there won't be a sort-by-occurrence whatever, but a
function called, say, sort-by-occurrence-and-score could take both
occurrences and flex scores into account.

Reply | Threaded
Open this post in threaded view
|

Re: [Emacs-diffs] scratch/new-flex-completion-style 2c75775 2/2: Score, sort and annotate flex-style completions according to match tightness

Dmitry Gutov
In reply to this post by João Távora
On 12.02.2019 20:21, João Távora wrote:

> Oh.  That's what I feared (I think) It would be nice to add something
> seemingly orthogonal like a "completion style" to all completion UI's
> and all tables, without needing to change the completion tables
> (granted, it would be good to *also* not change the UI's, but I'm not
> sure there a solution for that).

You have a point, sure. But I'd rather reuse the abstraction we already
have as the first step, and then maybe extend it (or add a new one),
rather add sorting code that uses these text properties to an existing
function.

> I dunno what you dunno.  When using flex/scatter it's *really* important
> to sort by flex-match.  I suggested "stable sorting" because it's a
> slightly more decent thing to completion tables.  And by sorting twice
> it's not like we're changing the Big-O complexity or anything.

Constants matter sometimes, too. Not necessarily in this case, but...

> But I also don't mind a scheme where the style's sorting completely
> takes over (and we save those CPU cycles).

That would be ideal, I think.

> I don't know what company-transformers are.

Not too hard to find out, I think.

> On the one hand I suppose
> when you want to sort by occurance, you ... want to sort by occurrence.
> So if you use the symbol "foofooubarbarnbazbazdquuxquuxo" very often
> you'll see it sort to the top for your "undo" example.  So maybe company
> should take the style's sorting last here, dunno.

OK, I was probably wrong here. Like, if only some of the completions
have occurences, the rest would be sorted by flex scores, and that...
sounds pretty good.

> In this case I would first sort by occurance and then style-score.
> There's no easy way to make a hybrid function, so we must pick a
> default.  For all the examples I've been given, it seems that
> flex-sorting should always come last. Since flex is the first and only
> style that carries some sorting idea with it, perhaps it's not a very
> bad idea to give it _some_ priority by default.

You probably mean assign the least priority to flex's scores. The flex
score sort would happen first with this approach, though.

Almost all completions will have different scores, so sorting by them
last would make all the previous sorting disappear.

>>> Eventually, we can add a keybinding to resort exclusively according to
>>> the table or exclusively according to the completion style.
>>
>> In the approach I'm thinking of, both options are the same sorting
>> function.
>
> How would you synthetize it?

Err, not the way you'd want. The table would just be designed to be used
with the completion style. And so its sorting function.

> In the general case, so the user has an idea _why_ something is being
> automatically.  Humans generally gain confidence in algorithms if
> they're shown human-comprehensible hints as to why it is is doing
> something (humans generally also like this from other humans, which
> explains why you ask me so many questions :-).

For a little while, maybe. To obtain the said confidence. And then, I'd
opt to turn it off, because I prefer to avoid too much visual clutter
(one of the reasons to use Emacs).

> In this particular case, it's a very common idiom when displaying
> tables/lists of something.  Perhaps in the "occurance sorting" field,
> he/she will choose another word used less frequently for stylistic
> purposes.
>
> But, all that said, I'm not going to kill myself over the annotation,
> nor do I think it even merits a customization option, though it's
> probably not hard to do it anyway.

You see, regarding the annotations as well, it makes more sense to reuse
the existing mechanism, or make it more flexible as well, somehow.

>> The former, more or less. This creates a certain limitation, sure, but
>> it shouldn't be a problem for Eglot.
>
> Hmm? By far, Eglot isn't the only one thing I'm interested in enhancing
> (actually, I've had few opportunities to actualy _use_ Eglot lately).

Sly's completion table could take the same approach. As long as the
completion table is under your control, it's all good.

By the way, we could also add an indirection for display-sort-function
(and maybe the -cycle- one as well) similar to the styles. So we not
only assign a style (or several of them) based on the category, but also
the sorting function. This way, sorting of buffers the right way would
still work and make sense. And at the same time, the user could change
certain categories to use both flex matching and flex score sorting.

That's more work for the user, but still. Seems kinda elegant.

Reply | Threaded
Open this post in threaded view
|

Re: [Emacs-diffs] scratch/new-flex-completion-style 2c75775 2/2: Score, sort and annotate flex-style completions according to match tightness

Stefan Monnier
In reply to this post by Dmitry Gutov
> On the one hand, yes, but it's the abstraction we already have, and it will
> allow flex to work properly at least with certain completion tables. Which
> is a win.

The problem I see with it is that performing the style-sorting via
completion-metadata or via completion-extra-properties is a hack that
works by breaking abstractions.

So, not only it will only work with some completion tables, but for that
you'll need to introduce "wrong" code in those completion tables.

I'd rather we extend the infrastructure so that completion styles can do
their own sorting.

> On the other hand, a search for display-sort-function through Emacs's
> sources seems to indicate that the design is kind of a failure: there are
> only a couple places that set it to something, and in both cases it's
> #'identity, meaning "do not re-sort".

Indeed.  Those cases *do* perform sorting, but they sort directly inside
their `all-completions` function rather than in their display-sort-function.

So yes, the design is kind of a failure, tho the alternative of having
a boolean in the metadata to say "don't sort" is no simpler.

> So minibuffer.el might want to take a cue from Company and
> company-transformers, to make this universal, we could add sorting as
> a feature of completion styles, or as a separate user option, for maximum
> flexibility.

The question is really: how/where would a completion style specify which
sorting it wants to do (based on the above-mentioned past experience,
we can assume that they'll want to do their sorting directly in
completion-all-completions and then specify "don't sort any further"),
and then how to change minibuffer.el to do that.

Currently, completion-all-completions does not return any information
about which style was used, so either we change this API so that it also
returns which style was used, or some info about the style gets added as
text-properties to influence subsequent sorting, or
completion-all-completions is changed to perform the sorting (which may
require extending its calling convention so the caller can tell it which
kind of sorting it wants).

> Now, I wouldn't say it's terrible that sort-by-occurrence-and-score would
> know specifically about flex (I don't think there's going to be a lot of
> different styles in this flavor), but sure, like you proposed in another
> email, the property can have a neutral name like completion-score.

Using scores would make it possible to combine different scoring systems
by weighting the different parts.  I'm not completely happy with the use
of text-properties, tho (e.g. ideally completion-all-completion could
return a list of pairs of the form (CANDIDATE . SCORE)).  But it has the
very significant advantage of requiring much fewer changes.


        Stefan


Reply | Threaded
Open this post in threaded view
|

Re: [Emacs-diffs] scratch/new-flex-completion-style 2c75775 2/2: Score, sort and annotate flex-style completions according to match tightness

Dmitry Gutov
On 13.02.2019 19:00, Stefan Monnier wrote:
> So, not only it will only work with some completion tables, but for that
> you'll need to introduce "wrong" code in those completion tables.

Since we seem to want to redo completion tables in a major way anyway, I
wouldn't worry about that too much, as long as the result is functional.

> I'd rather we extend the infrastructure so that completion styles can do
> their own sorting.

That's one option. Another option, which I've mentioned before, is to
create an indirection for sorting functions via the completion table's
category. Similar to the one that completions styles have.

> So yes, the design is kind of a failure, tho the alternative of having
> a boolean in the metadata to say "don't sort" is no simpler.

It's simpler but more ad-hoc, to be sure.

> The question is really: how/where would a completion style specify which
> sorting it wants to do (based on the above-mentioned past experience,
> we can assume that they'll want to do their sorting directly in
> completion-all-completions and then specify "don't sort any further"),
> and then how to change minibuffer.el to do that.

As we can see from the patch, using text properties (and actually sort
later) also seems like a viable approach.

> Currently, completion-all-completions does not return any information
> about which style was used, so either we change this API so that it also
> returns which style was used, or some info about the style gets added as
> text-properties to influence subsequent sorting, or
> completion-all-completions is changed to perform the sorting (which may
> require extending its calling convention so the caller can tell it which
> kind of sorting it wants).

The category-based approach should circumvent that problem.

With this approach, completion styles don't indicate sorting, and
whatever entity did set up the completion style to be used (the user, a
configuration package, etc) would need to set up both completion styles
and sorting functions to match.

> Using scores would make it possible to combine different scoring systems
> by weighting the different parts.  I'm not completely happy with the use
> of text-properties, tho (e.g. ideally completion-all-completion could
> return a list of pairs of the form (CANDIDATE . SCORE)).  But it has the
> very significant advantage of requiring much fewer changes.

I think text properties are fine here. In Company, we use them for
various bits of information.

Further, if your goal is to combine different scoring systems, it'll
probably be required to assign different scores (from different systems)
to each completion. The form (CANDIDATE . SCORE) won't cut it, but text
properties should.

Reply | Threaded
Open this post in threaded view
|

Re: [Emacs-diffs] scratch/new-flex-completion-style 2c75775 2/2: Score, sort and annotate flex-style completions according to match tightness

Stefan Monnier
>> So, not only it will only work with some completion tables, but for that
>> you'll need to introduce "wrong" code in those completion tables.
> Since we seem to want to redo completion tables in a major way anyway,

That's no guarantee that it will happen soon or at all, and it will have
to provide some backward compatibility.  And the more hacks we see using
the current system, the more difficult backward compatibility may become.

> I wouldn't worry about that too much, as long as the result is functional.

It'd be OK if it's indispensable, but I think there are better
alternatives that are easy to install right away.

>> I'd rather we extend the infrastructure so that completion styles can do
>> their own sorting.
> That's one option. Another option, which I've mentioned before, is to create
> an indirection for sorting functions via the completion table's
> category.  Similar to the one that completions styles have.

But those are orthogonal: the completion style can offer one kind of
sorting and that should work for all completion tables.
Completion tables can also offer their own kind of sorting (and it
should work with all completion styles).

How 'bout:

- Add global sorting config var(s?), to choose which kind of sorting to
  use, which would default to sorting based on "scores first and
  alphabetical after that".
- Let completion-category-overrides specify other choices.

This leaves some questions unanswered, tho:

- What about the distinction between cycle-sort and display-sort?
- Should the new var and new entries in completion-category-overrides
  contain directly sorting functions, or should they contain just the
  name of "sorting styles" with a separate table mapping those styles to
  actual functions (e.g. because a given style like `date` might be
  implemented differently for different completion tables?).
- Should we allow completion tables to offer several sorting choices?

In any case, in the mean time we can probably just introduce the new
sorting based on "scores first and alphabetical after that" and use it
by default.

> The category-based approach should circumvent that problem.
> With this approach, completion styles don't indicate sorting, and whatever
> entity did set up the completion style to be used (the user, a configuration
> package, etc) would need to set up both completion styles and sorting
> functions to match.

I think it'd be annoying for users to have to not only add `flex` to
their completion-styles but also change some sorting option at the same
time before that completion style becomes usable (in most cases, the
current alphabetical sorting works really poorly for `flex`).

> I think text properties are fine here.

It has its downsides, but I agree it seems like the better choice.


        Stefan