bug#44784: Combining accent is not combining

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

bug#44784: Combining accent is not combining

Juri Linkov-2
Sorry that I noticed this regression only now,
but it's a pretty bad-looking regression:

0. emacs -Q
1. insert to the *scratch* buffer:

ä́  і́  Я́

In 26.3 everything is correct, and COMBINING ACUTE ACCENT
is combined above its previous character:



Composed with the following character(s) "́" using this font:
  xft:-PfEd-DejaVu Sans Mono-normal-normal-normal-*-22-*-*-*-m-0-iso10646-1
by these glyphs:
  [0 1 1071 878 13 0 13 21 0 nil]
  [0 1 769 649 0 5 11 18 -14 [-15 -4 0]]
In 26.3 (x86_64-pc-linux-gnu, GTK+ Version 3.24.14)
Configured features:
XPM JPEG TIFF GIF PNG RSVG IMAGEMAGICK SOUND GPM DBUS GSETTINGS GLIB
NOTIFY ACL LIBSELINUX GNUTLS LIBXML2 FREETYPE M17N_FLT LIBOTF XFT ZLIB
TOOLKIT_SCROLL_BARS GTK3 X11 XDBE XIM THREADS LIBSYSTEMD LCMS2

But in 27.1 COMBINING ACUTE ACCENT is not combined with its previous character:



Composed with the following character(s) "́" using this font:
  xfthb:-PfEd-DejaVu Sans Mono-normal-normal-normal-*-22-*-*-*-m-0-iso10646-1
by these glyphs:
  [0 1 1071 878 13 0 13 16 0 nil]
  [0 1 769 649 13 4 11 17 -13 [0 0 0]]
In 27.1 (x86_64-pc-linux-gnu, GTK+ Version 3.24.20)
Configured features:
XPM JPEG TIFF GIF PNG RSVG SOUND GPM DBUS GSETTINGS GLIB NOTIFY INOTIFY
ACL LIBSELINUX GNUTLS LIBXML2 FREETYPE HARFBUZZ M17N_FLT LIBOTF XFT ZLIB
TOOLKIT_SCROLL_BARS GTK3 X11 XDBE XIM MODULES THREADS LIBSYSTEMD JSON
PDUMPER LCMS2 GMP

It seems in 26.3 it's correct because it uses XFT, but the problem in 27.1
is with HarfBuzz .  Maybe HarfBuzz doesn't support combining accents on
some characters?  It fails with the scripts above, but still has no problems
with plain Latin characters.

comb26.png (3K) Download Attachment
comb27.png (3K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

bug#44784: Combining accent is not combining

Eli Zaretskii
> From: Juri Linkov <[hidden email]>
> Date: Sat, 21 Nov 2020 21:17:09 +0200
>
> 0. emacs -Q
> 1. insert to the *scratch* buffer:
>
> ä́  і́  Я́
>
> In 26.3 everything is correct, and COMBINING ACUTE ACCENT
> is combined above its previous character:
>
>
> But in 27.1 COMBINING ACUTE ACCENT is not combined with its previous character:

On my system, this displays the same in both Emacs 26 and 27, the
latter using HarfBuzz.

> It seems in 26.3 it's correct because it uses XFT, but the problem in 27.1
> is with HarfBuzz .  Maybe HarfBuzz doesn't support combining accents on
> some characters?  It fails with the scripts above, but still has no problems
> with plain Latin characters.

That cannot be the reason.  Please try with other fonts, for
starters.  DejaVu Sans Mono is known to have problems with combining
accents, see this discussion:

  https://lists.freedesktop.org/archives/harfbuzz/2019-August/007422.html



Reply | Threaded
Open this post in threaded view
|

bug#44784: Combining accent is not combining

Andreas Schwab-2
In reply to this post by Juri Linkov-2
On Nov 21 2020, Juri Linkov wrote:

> But in 27.1 COMBINING ACUTE ACCENT is not combined with its previous character:

Do they come from the same font?

Andreas.

--
Andreas Schwab, [hidden email]
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."



Reply | Threaded
Open this post in threaded view
|

bug#44784: Combining accent is not combining

Eli Zaretskii
In reply to this post by Eli Zaretskii
Btw, the report by Emacs 27's "C-u C-x =", viz.:

  Composed with the following character(s) "́" using this font:
    xfthb:-PfEd-DejaVu Sans Mono-normal-normal-normal-*-22-*-*-*-m-0-iso10646-1
  by these glyphs:
    [0 1 1071 878 13 0 13 16 0 nil]
    [0 1 769 649 13 4 11 17 -13 [0 0 0]]

explicitly says that character composition _did_ happen here, it's
just the display that is broken.  And I guess that moving point to
these characters will show a "fat" cursor that includes both the base
character and the accent, once again showing that Emacs did compose
the accent with the preceding character.



Reply | Threaded
Open this post in threaded view
|

bug#44784: Combining accent is not combining

Juri Linkov-2
In reply to this post by Andreas Schwab-2
>> But in 27.1 COMBINING ACUTE ACCENT is not combined with its previous character:
>
> Do they come from the same font?

Yes, both use the same font:

Composed with the following character(s) "́" using this font:
  ftcrhb:-PfEd-DejaVu Sans Mono-normal-normal-normal-*-13-*-*-*-m-0-iso10646-1

The component character(s) are displayed by these fonts (glyph codes):
 ́: ftcrhb:-PfEd-DejaVu Sans Mono-normal-normal-normal-*-13-*-*-*-m-0-iso10646-1 (#x289)



Reply | Threaded
Open this post in threaded view
|

bug#44784: Combining accent is not combining

Juri Linkov-2
In reply to this post by Eli Zaretskii
> Btw, the report by Emacs 27's "C-u C-x =", viz.:
>
>   Composed with the following character(s) "́" using this font:
>     xfthb:-PfEd-DejaVu Sans Mono-normal-normal-normal-*-22-*-*-*-m-0-iso10646-1
>   by these glyphs:
>     [0 1 1071 878 13 0 13 16 0 nil]
>     [0 1 769 649 13 4 11 17 -13 [0 0 0]]
>
> explicitly says that character composition _did_ happen here, it's
> just the display that is broken.  And I guess that moving point to
> these characters will show a "fat" cursor that includes both the base
> character and the accent, once again showing that Emacs did compose
> the accent with the preceding character.

Actually, separate cursor is shown on a character and separate on its
combining character, so it seems they are not composed.

>> It seems in 26.3 it's correct because it uses XFT, but the problem in 27.1
>> is with HarfBuzz .  Maybe HarfBuzz doesn't support combining accents on
>> some characters?
>
> That cannot be the reason.  Please try with other fonts, for
> starters.  DejaVu Sans Mono is known to have problems with combining
> accents, see this discussion:

Yes, it displays correctly with other fonts, e.g. Noto Sans.
It would be a pity to switch to another font.  So this bug report
could be closed if nothing can be done in Emacs.



Reply | Threaded
Open this post in threaded view
|

bug#44784: Combining accent is not combining

Stephen Berman
On Sun, 22 Nov 2020 05:35:30 +0200 Eli Zaretskii <[hidden email]> wrote:

>> From: Juri Linkov <[hidden email]>
>> Cc: [hidden email]
>> Date: Sat, 21 Nov 2020 22:35:11 +0200
>>
>> > Btw, the report by Emacs 27's "C-u C-x =", viz.:
>> >
>> >   Composed with the following character(s) "́" using this font:
>> >     xfthb:-PfEd-DejaVu Sans Mono-normal-normal-normal-*-22-*-*-*-m-0-iso10646-1
>> >   by these glyphs:
>> >     [0 1 1071 878 13 0 13 16 0 nil]
>> >     [0 1 769 649 13 4 11 17 -13 [0 0 0]]
>> >
>> > explicitly says that character composition _did_ happen here, it's
>> > just the display that is broken.  And I guess that moving point to
>> > these characters will show a "fat" cursor that includes both the base
>> > character and the accent, once again showing that Emacs did compose
>> > the accent with the preceding character.
>>
>> Actually, separate cursor is shown on a character and separate on its
>> combining character, so it seems they are not composed.
>
> The 2 vectors you show above, let alone the "composed with" part,
> explicitly say they _were_ composed.  So something else is going on
> here.

I think it's a display problem with DejaVu Sans Mono, as you already
noted.  I also see what Juri sees, but when the cursor appears to be
over the separately displayed accent, `C-u C-x =' says the character at
point is SPC.

Steve Berman



Reply | Threaded
Open this post in threaded view
|

bug#44784: Combining accent is not combining

Eli Zaretskii
> From: Stephen Berman <[hidden email]>
> Cc: Juri Linkov <[hidden email]>,  [hidden email]
> Date: Sun, 22 Nov 2020 11:12:03 +0100
>
> > The 2 vectors you show above, let alone the "composed with" part,
> > explicitly say they _were_ composed.  So something else is going on
> > here.
>
> I think it's a display problem with DejaVu Sans Mono, as you already
> noted.  I also see what Juri sees, but when the cursor appears to be
> over the separately displayed accent, `C-u C-x =' says the character at
> point is SPC.

Right.  I think the discussion on the HarfBuzz list to which I pointed
explains what happens: the problematic fonts don't provide the
information necessary to correctly locate the accent over some
character (specifically, precomposed characters that are already
accented).  The [0 0 0] part of the second glyph in the composition
info is a telltale sign of lack of that information: if you try the
same with a font where display is correct, you will see that the first
2 components are non-zero (notably, the first component should be
negative).



Reply | Threaded
Open this post in threaded view
|

bug#44784: Combining accent is not combining

Stephen Berman
On Sun, 22 Nov 2020 17:28:08 +0200 Eli Zaretskii <[hidden email]> wrote:

>> From: Stephen Berman <[hidden email]>
>> Cc: Juri Linkov <[hidden email]>,  [hidden email]
>> Date: Sun, 22 Nov 2020 11:12:03 +0100
>>
>> > The 2 vectors you show above, let alone the "composed with" part,
>> > explicitly say they _were_ composed.  So something else is going on
>> > here.
>>
>> I think it's a display problem with DejaVu Sans Mono, as you already
>> noted.  I also see what Juri sees, but when the cursor appears to be
>> over the separately displayed accent, `C-u C-x =' says the character at
>> point is SPC.
>
> Right.  I think the discussion on the HarfBuzz list to which I pointed
> explains what happens: the problematic fonts don't provide the
> information necessary to correctly locate the accent over some
> character (specifically, precomposed characters that are already
> accented).  The [0 0 0] part of the second glyph in the composition
> info is a telltale sign of lack of that information: if you try the
> same with a font where display is correct, you will see that the first
> 2 components are non-zero (notably, the first component should be
> negative).

FTR, here's what `C-u C-x =' shows for the correct and incorrect
displays, respectively:

Composed with the following character(s) "́" using this font:
  ftcrhb:-PfEd-DejaVu Sans-normal-normal-normal-*-13-*-*-*-*-0-iso10646-1
by these glyphs:
  [0 1 1110 243 4 1 3 7 0 nil]
  [0 1 769 690 0 -5 -1 11 -7 [1 0 0]]
with these character(s):
  ́ (#x301) COMBINING ACUTE ACCENT

Composed with the following character(s) "́" using this font:
  ftcrhb:-PfEd-DejaVu Sans Mono-normal-normal-normal-*-13-*-*-*-m-0-iso10646-1
by these glyphs:
  [0 1 1110 917 8 1 8 10 0 nil]
  [0 1 769 649 8 3 8 11 -8 [0 0 0]]
with these character(s):
  ́ (#x301) COMBINING ACUTE ACCENT

Steve Berman