Corrupted display in gnome-terminal

[vincent@…]

With the attached mbox file (1 message), the index is not correctly displayed in gnome-terminal (UTF-8): the second 'B' is below the 'e' instead of being below the 'v'. As a consequence, there an additional 'E' on the following line in the index (with a 80-column terminal).

This also leads to inconsistent display in complex cases (I couldn't reproduce this), where the cursor may not be in front of the current message. For instance, when the user wants to delete a message, a wrong message can be deleted.

Note: the "From:" header is invalid, but this comes from a spam, and Mutt shouldn't behave badly even in these cases.
>How-To-Repeat:
Open the attached mbox file with:
    mutt -F /dev/null -f gnome-term
>Fix:
Unknown

[Alain Bench <veronatif@…>]

Salut Vincent, et merci.

 On Tuesday, May 2, 2006 at 12:20:56 +0200, Vincent Lef=E8vre wrote:

> the index is not correctly displayed in gnome-terminal (UTF-8): the
> second 'B' is below the 'e' instead of being below the 'v'. As a
> consequence, there an additional 'E' on the following line in the
> index (with a 80-column terminal).

    I do not understand at all the description, sorry. Some screen
copy might perhaps help me? I tried on PuTTY in UTF-8, and get garbage,
which is not a surprise given that the "From:" seems to be raw Chinese
BIG5. Setting $assumed_charset=3Dbig5 displays what looks to me like
correct ideograms.


> the "From:" header is invalid [...] Mutt shouldn't behave badly even
> in these cases.

    I agree. The current design of letting unknown raw bytes reach
directly the screen, in the hope that the terminal will know how to cope
with them, is clearly not very good. And especially unfriendly to UTF-8
terms. I believe /2218 is yet another case asking for inclusion of the
assumed_charset feature.


Bye!	Alain.
--=20
Mutt muttrc tip to send mails in best adapted first necessary and suffici=
ent
charset (version for Western Latin-1/Latin-9/CP-850/CP-1252 terminal user=
s):
set send_charset=3D"us-ascii:iso-8859-1:iso-8859-15:windows-1252:utf-8"

[Vincent Lefevre <vincent@…>]

On 2006-05-06 01:45:03 +0200, Alain Bench wrote:
>   On Tuesday, May 2, 2006 at 12:20:56 +0200, Vincent Lef=3DE8vre wrote:
>  > the index is not correctly displayed in gnome-terminal (UTF-8): the
>  > second 'B' is below the 'e' instead of being below the 'v'. As a
>  > consequence, there an additional 'E' on the following line in the
>  > index (with a 80-column terminal).
> =20
>      I do not understand at all the description, sorry. Some screen
>  copy might perhaps help me?

I get something like that:

q:Quit  d:Del  u:Undel  s:Save  m:Mail  r:Reply  g:Group  ?:Help         =
      =20
   1 N   May 01 ???@?=D7=A1B?Q?t?,B? (   3) 0123456789abcdefghijklmnopqrs=
tuvwxyzABCD
E

instead of:

q:Quit  d:Del  u:Undel  s:Save  m:Mail  r:Reply  g:Group  ?:Help         =
      =20
   1 N   May 01 ???@?=D7=A1B?Q?t?B? (   3) 0123456789abcdefghijklmnopqrst=
uvwxyzABCDE


>  I tried on PuTTY in UTF-8, and get garbage, which is not a surprise
>  given that the "From:" seems to be raw Chinese BIG5. Setting
>  $assumed_charset=3D3Dbig5 displays what looks to me like correct
>  ideograms.

I don't mind not getting correct ideograms. But it shouldn't corrupt
the display (i.e. the other lines of the index).

>  > the "From:" header is invalid [...] Mutt shouldn't behave badly even
>  > in these cases.
> =20
>      I agree. The current design of letting unknown raw bytes reach
>  directly the screen, in the hope that the terminal will know how to co=
pe
>  with them, is clearly not very good. And especially unfriendly to UTF-=
8
>  terms. I believe /2218 is yet another case asking for inclusion of the
>  assumed_charset feature.

In fact, I already have set assumed_charset=3D"us-ascii", and this didn't
solve the problem. IIRC, assumed_charset doesn't handle headers.

--=20
Vincent Lef=C3=A8vre <vincent@vinc17.org> - Web: <http://www.vinc17.org/>
100% accessible validated (X)HTML - Blog: <http://www.vinc17.org/blog/>
Work: CR INRIA - computer arithmetic / SPACES project at LORIA

[Alain Bench <veronatif@…>]

On Saturday, May 6, 2006 at 21:05:02 +0200, Vincent Lef=E8vre wrote:

>> I do not understand at all the description
> something like that:

    Understood. For some reason I was searching the 'v' and 'e' in the
subject, not in the word "save" of the help line.


> I already have set assumed_charset=3D"us-ascii", and this didn't solve
> the problem. IIRC, assumed_charset doesn't handle headers.

    Assumed_charset of course handles headers (just try big5) as well as
bodies. But the original patch had a weakness in the ?-masking of
unconvertables in headers. When assumed to be Ascii, the Chinese "From:"
is full of unconvertables. Those going unmasked to screen might still
corrupt the display. Tamotsu fixed the weakness long ago in his own
revision of the patch. I'm a little lost in the versions now: What patch
do you use? Tamo: What's the current assumed_charset patch?

    Note: The Ascii assumed and well masked Chinese name should
theoretically be displayed as "???@???B?Q?t???" in the index. Which, if
not as nice as BIG5 ideograms, is absolutely guaranteed to not corrupt
anything. Pas m=EAme cette chochotte de Gnome.


Bye!	Alain.
--=20
Give your computer's unused idle processor cycles to a scientific goal:
The Folding@home project at <URL:http://folding.stanford.edu/>.

[TAKAHASHI Tamotsu <ttakah@…>]

* Sun May  7 2006 Alain Bench <veronatif@free.fr>
> I'm a little lost in the versions now: What patch do you use?
> Tamo: What's the current assumed_charset patch?

IIRC, I have not modified the patch for more than a year:
http://marc.theaimsgroup.com/?l=mutt-dev&m=111416753329430&w=2
http://www10.plala.or.jp/sanrinsha/tamo/patch-1.5.9.tt+tamo.assumed.3

-- 
tamo

[Alain Bench <veronatif@…>]

Hello Tamotsu!

 On Sunday, May 7, 2006 at 14:55:02 +0200, Tamotsu Takahashi wrote:

> * Sun May  7 2006 Alain Bench <veronatif@free.fr>
>> What's the current assumed_charset patch?
> http://www10.plala.or.jp/sanrinsha/tamo/patch-1.5.9.tt+tamo.assumed.3

    Much thanks! I tried it, and the practice met the theory: Vincent's
example "From:" with $assumed_charset=us-ascii is masked exactly as
expected. Masked by question marks on any $charset terminals, or by
replacement chars U+FFFD on UTF-8 terms. Display and layout are not
anymore corrupted.

    Just to repeat: I think the assumed_charset patch should be
committed. It's both a great feature, and a fix for such screen layout
breakage (unavoidable in straight Mutt). It enhances the fine control
the user has on what's shown him. That's the sort of feature Mutt needs,
and should like. And I see not a single valid argument against it.


Bye!	Alain.
-- 
Thus, quite literally, what Alain said is 100% accurate

[Vincent Lefevre <vincent@…>]

On 2006-05-07 12:55:03 +0200, Alain Bench wrote:
>      Assumed_charset of course handles headers (just try big5) as well =
as
>  bodies. But the original patch had a weakness in the ?-masking of
>  unconvertables in headers. When assumed to be Ascii, the Chinese "From=
:"
>  is full of unconvertables. Those going unmasked to screen might still
>  corrupt the display. Tamotsu fixed the weakness long ago in his own
>  revision of the patch. I'm a little lost in the versions now: What pat=
ch
>  do you use?

I'm currently using this one: patch-1.5.5.1.tt.compat.1-cl

--=20
Vincent Lef=E8vre <vincent@vinc17.org> - Web: <http://www.vinc17.org/>
100% accessible validated (X)HTML - Blog: <http://www.vinc17.org/blog/>
Work: CR INRIA - computer arithmetic / SPACES project at LORIA

[Vincent Lefevre <vincent@…>]

On 2006-05-07 14:55:02 +0200, TAKAHASHI Tamotsu wrote:
>  http://www10.plala.or.jp/sanrinsha/tamo/patch-1.5.9.tt+tamo.assumed.3

But then, is there a separate file_charset patch?

--=20
Vincent Lef=E8vre <vincent@vinc17.org> - Web: <http://www.vinc17.org/>
100% accessible validated (X)HTML - Blog: <http://www.vinc17.org/blog/>
Work: CR INRIA - computer arithmetic / SPACES project at LORIA

[Alain Bench <veronatif@…>]

On Tuesday, May 9, 2006 at 23:35:02 +0200, Vincent Lef=E8vre wrote:

> I'm currently using this one: patch-1.5.5.1.tt.compat.1-cl

    This one has the said weakness. Tamo's version will work better.


Bye!=09Alain.
--=20
Software should be written to deal with every conceivable error
=09RFC 1122 / Robustness Principle

[TAKAHASHI Tamotsu <ttakah@…>]

* Wed May 10 2006 Vincent Lefevre <vincent@vinc17.org>
>  On 2006-05-07 14:55:02 +0200, TAKAHASHI Tamotsu wrote:
>  >  http://www10.plala.or.jp/sanrinsha/tamo/patch-1.5.9.tt+tamo.assumed.3
>  
>  But then, is there a separate file_charset patch?

Yes; It is now $attach_charset: http://bugs.mutt.org/1106
http://www.emaillab.org/mutt/1.5.10/patch-1.5.10.tt.attach_charset.1.gz
(via http://www.emaillab.org/mutt/download1510.html )

You might be intersted in
http://www.emaillab.org/mutt/1.5.10/patch-1.5.10.tt.file_charset_compat.1.gz
as well. This allows you to use $file_charset as an alias to $attach_charset.

http://www.emaillab.org/mutt/1.5.10/patch-1.5.10.mutt-j.multi-charset.2.gz

-- 
tamo

[brendan]

Salut Vincent, et merci.
 On 2006-05-06 01:45:03 +0200, Alain Bench wrote:
  On Saturday, May 6, 2006 at 21:05:02 +0200, Vincent Lef=E8vre wrote:
 * Sun May  7 2006 Alain Bench <veronatif@free.fr>
 On 2006-05-07 12:55:03 +0200, Alain Bench wrote:
 On 2006-05-07 14:55:02 +0200, TAKAHASHI Tamotsu wrote:
 * Wed May 10 2006 Vincent Lefevre <vincent@vinc17.org>
  On Tuesday, May 9, 2006 at 23:35:02 +0200, Vincent Lef=E8vre wrote:
 Hello Tamotsu!
assumed_charset applied.

[anonymous]

gnome-term