illumos gate

It would help me if you could provide specific detail about what locale you were using, and sample Japanese text to work with. I suspect that the wcwidth() routine is broken. When I implemented this, I was operating on the belief that wcwidth was only needed for EUC locales, but perhaps we need support for it in UTF-8 as well?

I'm not sure where I can find wcwidth() data ... I don't think its in the unicode data files that I was given. :-(

Yeah, so looking at this more, we missed it. So did the POSIX committee apparently -- there is no standard way to provide this data to localedef -- its not part of the standard localedef grammar!

I am going to propose therefore a local extension, which should hopefully keep us from having to touch all the CLDR data files.

The local extension would be a new input file format, to be processed by localedef, which would contain the list of widths. This input file would be specified on the command line with a new switch (optional) to localedef.

One implication of this is that we will need to write the width data somewhere -- we already have a few bits reserved in the ctype data for this, so that's the logical place to do it.

I don't think this would be too hard to do.

My locale is ja_JP.UTF-8, but the problem should also manifest in English locales. As for sample text, try:

これは例です。

If you paste this string into a terminal emulator and then try to move the cursor back over it, you should be able to observe that the cursor can't move back to the beginning, and that attempting to insert characters with the cursor in the middle of the string results in strange behavior.

I'm not familiar with the intricate details of how this is handled, but your explanation sounds likely to me. I have seen a few old mentions around the web of wcwidth() problems previously existing in other platforms causing similar issues in UTF-8 locales. What kind of wcwidth() data would be needed to resolve the issue?

While looking at this issue, I noticed the following (not directly related to the issue itself) - __ct_rune_t is typedef'ed int for both LP64 and LP32 despite what the comment says (if I'm reading it correctly): http://src.illumos.org/source/xref/illumos-gate/usr/src/lib/libc/port/locale/runetype.h#88

BTW, this affect not only CJK locales, the utility I'm trying to import relies on wcwidth() as well to pretty-format its output. FreeBSD has the width defined as a ranges of character codes, we could use it.. Garrett, any hints on properly handling this?

Hi Yuri,

Could you please let me know how long roughly it could take to fix the issue?

Thank you.

Garrett D'Amore wrote:

Yeah, so looking at this more, we missed it. So did the POSIX committee
apparently -- there is no standard way to provide this data to localedef --
its not part of the standard localedef grammar!

It seems to me that you can specify the widths between 'WIDTH' and 'END WIDTH'
in the charmap file as described at the end of this section:
http://pubs.opengroup.org/onlinepubs/009695399/basedefs/xbd_chap06.html#tag_06_04

Lauri Tirkkonen wrote:

It seems to me that you can specify the widths between 'WIDTH' and 'END WIDTH'
in the charmap file

It seems that the CLDR tools to generate charmap files don't bake width data in them, though I would think that the data exists in the CLDR data files somewhere.

I've written an implementation of WIDTH / END WIDTH localedef support, along with a very rich level of character widths taken from an external source and the Unicode characters -- this works for all encodings.

I'll be posting it shortly. Good news - no changes in libc.

I also updated localedef(1)'s man page, which was ancient beyond belief.