bug#42562: Problem with rendering Persian text still exists in minibuffer and dired

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

bug#42562: Problem with rendering Persian text still exists in minibuffer and dired

Sineau Gh
Hello everyone,
Although bug#41005 has been solved in buffers with column-number-mode enabled, it still exists in places like minibuffer (I use Ivy) and dired buffers. I have attached the bug report from emacs.

Thanks for all the beautiful work you have done with emacs 27,
Sina

*message*-20200727-234614 (30K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

bug#42562: Problem with rendering Persian text still exists in minibuffer and dired

Eli Zaretskii
> From: Sineau Gh <[hidden email]>
> Date: Mon, 27 Jul 2020 23:51:17 +0430
>
> Although bug#41005 has been solved in buffers with column-number-mode enabled, it still exists in places
> like minibuffer (I use Ivy) and dired buffers. I have attached the bug report from emacs.

Thanks, but we need a recipe to reproduce the problem, please.



Reply | Threaded
Open this post in threaded view
|

bug#42562: Problem with rendering Persian text still exists in minibuffer and dired

Eli Zaretskii
> From: Sineau Gh <[hidden email]>
> Date: Wed, 29 Jul 2020 20:20:55 +0430
> Cc: [hidden email]
>
> Sure. But I'm not sure what kind of a recipe you're looking for.
> Persian/Arabic text on is disjointed like it was on buffers with column-number-mode. I attached two
> screenshots too, if that helps.
> Please tell me if there's any specific guidelines for submitting a recipe.

You mean, just typing the text into *scratch* causes this?

If so, what system is this and how was Emacs configured?  Can you show
the data collected by "M-x report-emacs-bug"?

If you need something other than just typing the text to reproduce the
problem, pleased tell what should one do to reproduce.



Reply | Threaded
Open this post in threaded view
|

bug#42562: Problem with rendering Persian text still exists in minibuffer and dired

Sineau Gh
The problem I originally reported was not concerned with typing text, but text rendered in read-only buffers (ie. dired and minibuffer).

I have attached the report-emacs-bug too. Please note that I compiled emacs 27 myself on debian buster.

Also here's the bad news. Just now I realized that bug#41005 is not solved yet. At certain combinations of characters, the ligatures are still disjointed. Also I tried to change the font family, but that  didn't help either. This doesn't happen when I compiled emacs using --without-harfbuzz option. I don't know if it helps but I can try to find if there's a pattern to the combinations of characters I mentioned.

On Wed, 29 Jul 2020 at 23:02, Eli Zaretskii <[hidden email]> wrote:
> From: Sineau Gh <[hidden email]>
> Date: Wed, 29 Jul 2020 20:20:55 +0430
> Cc: [hidden email]
>
> Sure. But I'm not sure what kind of a recipe you're looking for.
> Persian/Arabic text on is disjointed like it was on buffers with column-number-mode. I attached two
> screenshots too, if that helps.
> Please tell me if there's any specific guidelines for submitting a recipe.

You mean, just typing the text into *scratch* causes this?

If so, what system is this and how was Emacs configured?  Can you show
the data collected by "M-x report-emacs-bug"?

If you need something other than just typing the text to reproduce the
problem, pleased tell what should one do to reproduce.

f_kd8snugw0 (30K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

bug#42562: Problem with rendering Persian text still exists in minibuffer and dired

Eli Zaretskii
> From: Sineau Gh <[hidden email]>
> Date: Thu, 30 Jul 2020 17:27:36 +0430
> Cc: [hidden email]
>
> The problem I originally reported was not concerned with typing text, but text rendered in read-only buffers
> (ie. dired and minibuffer).

Then please describe the steps to reproduce this with one such
read-only buffer.

> Also here's the bad news. Just now I realized that bug#41005 is not solved yet. At certain combinations of
> characters, the ligatures are still disjointed. Also I tried to change the font family, but that  didn't help either.
> This doesn't happen when I compiled emacs using --without-harfbuzz option. I don't know if it helps but I can
> try to find if there's a pattern to the combinations of characters I mentioned.

Here also we would need a recipe to reproduce the problem.

These problems are highly context dependent, and cannot be debugged
without a reproducer.

Thanks.



Reply | Threaded
Open this post in threaded view
|

bug#42562: Problem with rendering Persian text still exists in minibuffer and dired

Sineau Gh



> From: Sineau Gh <[hidden email]>
> Date: Thu, 30 Jul 2020 17:27:36 +0430
> Cc: [hidden email]
>
> The problem I originally reported was not concerned with typing text, but text rendered in read-only buffers
> (ie. dired and minibuffer).

Then please describe the steps to reproduce this with one such
read-only buffer.



> Also here's the bad news. Just now I realized that bug#41005 is not solved yet. At certain combinations of
> characters, the ligatures are still disjointed. Also I tried to change the font family, but that  didn't help either.
> This doesn't happen when I compiled emacs using --without-harfbuzz option. I don't know if it helps but I can
> try to find if there's a pattern to the combinations of characters I mentioned.

Here also we would need a recipe to reproduce the problem.

These problems are highly context dependent, and cannot be debugged
without a reproducer.
 
Thanks.



Upon further investigation, I realized that in my config I have used a combination of `default-frame-alist` and `set-fontset-font` that I forgot about. So here's a reproduction I hope can be useful for you. First of all the case is the same for both editable buffers and read-only ones so I'm just using a file with following content:
تست
تحقیق
به
اصالت

شرتالکو
حقیق
حقیقت
سنت
تالکو
مدرن
مدرنیزاسیون
I have tested this in three cases. Please note they are basically the same with some minor differences, but I mention them for completeness.
1- If I have not set a default font in my config, the text is rendered with broken ligatures. And if I set the font using `M-x set-frame-font` then everything is okay (even if I set it again to the default font used for Persian / Arabic text).

2- If I use something like the following in my config files:
`(add-to-list 'default-frame-alist '(font . "DejaVu Sans Mono-12"))`
on init the font is rendered broken. And if I set the font using `M-x set-frame-font` then everything is okay. Now if I set it again to DejaVu Sans Mono (the font I have used in my config file), it still shows broken text. I have to first set it to another font and then set it again to DejaVu so it renders correctly.

3- If I use the following line in my config:
`(set-frame-font "DejaVu Sans Mono-12" t t)`
on init the font is broken. And if I set the font using `M-x set-frame-font` then everything is okay (like previous cases). But I can't set it to DejaVu Sans Mono using `M-x set-frame-font` in any way. That means I first tried to change the font to something different and then back to DejaVu and still it didn't work.

I should note that I have tested this with various fonts and the result is the same. Also I have tested this in Org and Fundamental modes and the result is the same. Also I didn't add `set-fontset-font` to the mix but I supposed it's not going to make a difference.

I hope this helps, and excuse me if it's not. Thanks for your patience.
Reply | Threaded
Open this post in threaded view
|

bug#42562: Problem with rendering Persian text still exists in minibuffer and dired

Sineau Gh
Hi again,
Here's some more info I have gathered. If I haven't set the default font in my config file, `M-x set-frame-font` only works if there's already an open buffer in the frame. And if I visit a new file, the ligatures are rendered broken again, although the font is the same. So I have to change the font again.

On Thu, 30 Jul 2020 at 23:42, Sineau Gh <[hidden email]> wrote:



> From: Sineau Gh <[hidden email]>
> Date: Thu, 30 Jul 2020 17:27:36 +0430
> Cc: [hidden email]
>
> The problem I originally reported was not concerned with typing text, but text rendered in read-only buffers
> (ie. dired and minibuffer).

Then please describe the steps to reproduce this with one such
read-only buffer.



> Also here's the bad news. Just now I realized that bug#41005 is not solved yet. At certain combinations of
> characters, the ligatures are still disjointed. Also I tried to change the font family, but that  didn't help either.
> This doesn't happen when I compiled emacs using --without-harfbuzz option. I don't know if it helps but I can
> try to find if there's a pattern to the combinations of characters I mentioned.

Here also we would need a recipe to reproduce the problem.

These problems are highly context dependent, and cannot be debugged
without a reproducer.
 
Thanks.



Upon further investigation, I realized that in my config I have used a combination of `default-frame-alist` and `set-fontset-font` that I forgot about. So here's a reproduction I hope can be useful for you. First of all the case is the same for both editable buffers and read-only ones so I'm just using a file with following content:
تست
تحقیق
به
اصالت

شرتالکو
حقیق
حقیقت
سنت
تالکو
مدرن
مدرنیزاسیون
I have tested this in three cases. Please note they are basically the same with some minor differences, but I mention them for completeness.
1- If I have not set a default font in my config, the text is rendered with broken ligatures. And if I set the font using `M-x set-frame-font` then everything is okay (even if I set it again to the default font used for Persian / Arabic text).

2- If I use something like the following in my config files:
`(add-to-list 'default-frame-alist '(font . "DejaVu Sans Mono-12"))`
on init the font is rendered broken. And if I set the font using `M-x set-frame-font` then everything is okay. Now if I set it again to DejaVu Sans Mono (the font I have used in my config file), it still shows broken text. I have to first set it to another font and then set it again to DejaVu so it renders correctly.

3- If I use the following line in my config:
`(set-frame-font "DejaVu Sans Mono-12" t t)`
on init the font is broken. And if I set the font using `M-x set-frame-font` then everything is okay (like previous cases). But I can't set it to DejaVu Sans Mono using `M-x set-frame-font` in any way. That means I first tried to change the font to something different and then back to DejaVu and still it didn't work.

I should note that I have tested this with various fonts and the result is the same. Also I have tested this in Org and Fundamental modes and the result is the same. Also I didn't add `set-fontset-font` to the mix but I supposed it's not going to make a difference.

I hope this helps, and excuse me if it's not. Thanks for your patience.
Reply | Threaded
Open this post in threaded view
|

bug#42562: Problem with rendering Persian text still exists in minibuffer and dired

Sineau Gh
This can't be the problem, because even if I don't set a font on my
config, the font which selected automatically by emacs is also rendered
disjointed. But to make sure this isn't the case, I removed two
compatible fonts that are automatically selected (namely, Vazir and
Noto, both of which have great Persian support). The default font is now DejaVu
Mono, which is again rendered disjointed.

On Fri, 31 Jul 2020 at 09:44, Eli Zaretskii <[hidden email]> wrote:
> From: Sineau Gh <[hidden email]>
> Date: Fri, 31 Jul 2020 00:35:13 +0430
> Cc: [hidden email]
>
> Here's some more info I have gathered. If I haven't set the default font in my config file, `M-x set-frame-font`
> only works if there's already an open buffer in the frame. And if I visit a new file, the ligatures are rendered
> broken again, although the font is the same. So I have to change the font again.

If the problem comes and goes as you change fonts, then the problem is
not with Emacs, it is with the fonts you are using: they should
support Arabic shaping.  Emacs by default chooses suitable fonts for
the Arabic script, but if you force Emacs to use certain font, you can
disrupt the automatic font selection algorithm and choose a font that
doesn't support Arabic shaping.

The problems reported earlier existed with any font, and didn't
disappear when a font was changed.
Reply | Threaded
Open this post in threaded view
|

bug#42562: Problem with rendering Persian text still exists in minibuffer and dired

Eli Zaretskii
> From: Sineau Gh <[hidden email]>
> Date: Fri, 31 Jul 2020 10:08:43 +0430
> Cc: [hidden email]
>
> This can't be the problem, because even if I don't set a font on my
> config, the font which selected automatically by emacs is also rendered
> disjointed. But to make sure this isn't the case, I removed two
> compatible fonts that are automatically selected (namely, Vazir and
> Noto, both of which have great Persian support). The default font is now DejaVu
> Mono, which is again rendered disjointed.

Is this in "emacs -Q"?  If so, then this is contrary to what everyone
else sees with Emacs 27, so there must be some other factor at work
here.

In any case, I repeat the previous request: please provide a recipe
for reproducing the problem, starting with "emacs -Q".  For example,
if this happens in Dired, then the recipe could be something like
this:

  . create a directory with files named .... (here show the exact
    names of the file names to create)
  . start Dired on that directory
  . observe that file names ... (state the problematic file names) are
    rendered incorrectly (here please attach a screenshot of the
    incorrect display, and another one of how these file names should
    look when rendered correctly)

If some other non-default setting should be used, such as
column-number-mode or something else, please include those settings in
the recipe.

Such a recipe is required to debug the problem.  Based on previous
experience, once a recipe is provided, the solution is found very
quickly, but before the recipe is provided there's almost no progress
in investigating the problem.

Thanks.



Reply | Threaded
Open this post in threaded view
|

bug#42562: Problem with rendering Persian text still exists in minibuffer and dired

Sineau Gh
Oh sorry for the trouble. I haven't tested it with emacs -Q (rookie
here). In that case dired and editable buffers are working as
expected.  Here's a reproduction recipe for *Completion Buffer* and
modeline:

- create a folder with following files:

تحقیق
تست
مدرن.txt
تحقیق.org

- visit a file using C-x C-f.
- use TAB for *Completion Buffer*.
- first character of all file names are disjointed.
- type ت to filter the filenames, now the second characters are
  disjointed.
- now open one of the files. in modeline first and last characters of
  the filename are disjointed.

On Fri, 31 Jul 2020 at 10:36, Eli Zaretskii <[hidden email]> wrote:
> From: Sineau Gh <[hidden email]>
> Date: Fri, 31 Jul 2020 10:08:43 +0430
> Cc: [hidden email]
>
> This can't be the problem, because even if I don't set a font on my
> config, the font which selected automatically by emacs is also rendered
> disjointed. But to make sure this isn't the case, I removed two
> compatible fonts that are automatically selected (namely, Vazir and
> Noto, both of which have great Persian support). The default font is now DejaVu
> Mono, which is again rendered disjointed.

Is this in "emacs -Q"?  If so, then this is contrary to what everyone
else sees with Emacs 27, so there must be some other factor at work
here.

In any case, I repeat the previous request: please provide a recipe
for reproducing the problem, starting with "emacs -Q".  For example,
if this happens in Dired, then the recipe could be something like
this:

  . create a directory with files named .... (here show the exact
    names of the file names to create)
  . start Dired on that directory
  . observe that file names ... (state the problematic file names) are
    rendered incorrectly (here please attach a screenshot of the
    incorrect display, and another one of how these file names should
    look when rendered correctly)

If some other non-default setting should be used, such as
column-number-mode or something else, please include those settings in
the recipe.

Such a recipe is required to debug the problem.  Based on previous
experience, once a recipe is provided, the solution is found very
quickly, but before the recipe is provided there's almost no progress
in investigating the problem.

Thanks.
Reply | Threaded
Open this post in threaded view
|

bug#42562: Problem with rendering Persian text still exists in minibuffer and dired

Eli Zaretskii
> From: Sineau Gh <[hidden email]>
> Date: Fri, 31 Jul 2020 11:02:44 +0430
> Cc: [hidden email]
>

> تحقیق
> تست
> مدرن.txt
> تحقیق.org

> - visit a file using C-x C-f.
> - use TAB for *Completion Buffer*.
> - first character of all file names are disjointed.
> - type ت to filter the filenames, now the second characters are
>   disjointed.
> - now open one of the files. in modeline first and last characters of
>   the filename are disjointed.

Thanks.  Now everything is clear: this is due to the basic limitation
of how Emacs displays text with different faces: we render each run of
character in the same face separately from characters in a different
face.  So if the face changes in the middle of a word, we render the
two parts of the word separately, and that breaks Arabic shaping.

Fixing this would need significant changes in the low-level code that
handles text layout, which is one of the basics of the current display
engine.  Patches are welcome.



Reply | Threaded
Open this post in threaded view
|

bug#42562: Problem with rendering Persian text still exists in minibuffer and dired

Eli Zaretskii
> From: Sineau Gh <[hidden email]>
> Date: Fri, 31 Jul 2020 18:02:19 +0430
> Cc: [hidden email]
>
> I have one more question. You have hinted in a previous email that
> setting the font for emacs may cause it to use a font that doesn't
> support arabic shaping. Are you suggesting that I can't use commands
> like `set-frame-font` or `default-aframe-list` like before? Or is
> there something else in my config that messes up the ligatures?

You can use those commands, but be careful which fonts you name there.
Especially if you use set-fontset-font for Persian characters.