Transliteration on Unicode LATIN LETTERS "WITH STROKE"

Question

Feeding the rule "NFD; [:Nonspacing Mark:] Remove; NFC" into the ICU Transliterator demo, the character Ø (\u00d8 == LATIN CAPITAL LETTER O WITH STROKE) remains as-is (i.e. the STROKE is not stripped).

In the list of non-marking spaces (Category Mn), I cannot find anything named COMBINING DIAGONAL STROKE akin to the COMBINING SHORT STROKE OVERLAY (\u0335) or COMBINING LONG STROKE OVERLAY (\u0336).

However, I do find COMBINING SHORT SOLIDUS OVERLAY (\u0337) and COMBINING LONG SOLIDUS OVERLAY (\u0338). They appear similar, but render as much thicker lines in my browser when combined with o and O.

The Unicode data I accessed for \u00d8 does not provide a decomposition for that character.

At the same time, the ICU Collator Demo will collate each of ø, o, Ø, O, o\u0337 and O\u0338 to the same code point using a Primary (Level = 1 = Base Letter) Collator.

Does this mean that the locale of Collator used in the Demo has been set up to identify the base character in a way where the Unicode spec is silent?

If so, do I need to a custom Rule Based Transliterator if I want to strip the STROKE from LATIN [CAPITAL, SMALL] LETTER * characters on transliteration?

Answer 1

See the following. The Latin-ASCII transliterator went into ICU 4.6. As you noted, the collation demo uses UCA / CLDR tailorings which have O versus slashed-O as base letter differences, this is not the same question as whether there's a decomposition. "w" doesn't decompose into "v + v" either. The decompositions have to do with whether there were existing encodings which represent characters in two different ways.

• Java library for text normalization

• UTF-8 to ASCII using ICU Library

• http://unicode.org/repos/cldr/trunk/common/transforms/Latin-ASCII.xml

Answer 2

Yes. For some reason, the letter Ø does not have a decomposition, so you have to handle it manually.

Answer 3

This transform along with replaceAll works even for removing the Ø and other characters.

String id = "Accents-Any;NFD;[:Nonspacing Mark:] Remove; NFC";
System.out.println(latin.replaceAll("[^\\w]",""));