Étiquette : encodage

L’encodage des chaînes de caractère en Java et comment gérer l’internationalisation

Quelques notes pour mieux comprendre les notions de conversion de charset et d’internationalisation en java.

Comprendre la base : le char

Il faut identifier ce que contient le type primaire char et son grand frère le Character :

The char data type (and therefore the value that a Character object encapsulates) are based on the original Unicode specification, which defined characters as fixed-width 16-bit entities. The Unicode standard has since been changed to allow for characters whose representation requires more than 16 bits. The range of legal code points is now U+0000 to U+10FFFF, known as Unicode scalar value. (Refer to the definition of the U+n notation in the Unicode standard.)

The set of characters from U+0000 to U+FFFF is sometimes referred to as the Basic Multilingual Plane (BMP). Characters whose code points are greater than U+FFFF are called supplementary characters. The Java 2 platform uses the UTF-16 representation in char arrays and in the String and StringBuffer classes. In this representation, supplementary characters are represented as a pair of char values, the first from the high-surrogates range, (\uD800-\uDBFF), the second from the low-surrogates range (\uDC00-\uDFFF).

A char value, therefore, represents Basic Multilingual Plane (BMP) code points, including the surrogate code points, or code units of the UTF-16 encoding. An int value represents all Unicode code points, including supplementary code points. The lower (least significant) 21 bits of int are used to represent Unicode code points and the upper (most significant) 11 bits must be zero. Unless otherwise specified, the behavior with respect to supplementary characters and surrogate char values is as follows:

* The methods that only accept a char value cannot support supplementary characters. They treat char values from the surrogate ranges as undefined characters. For example, Character.isLetter(‘\uD840’) returns false, even though this specific value if followed by any low-surrogate value in a string would represent a letter.
* The methods that accept an int value support all Unicode characters, including supplementary characters. For example, Character.isLetter(0x2F81A) returns true because the code point value represents a letter (a CJK ideograph).

In the Java SE API documentation, Unicode code point is used for character values in the range between U+0000 and U+10FFFF, and Unicode code unit is used for 16-bit char values that are code units of the UTF-16 encoding. For more information on Unicode terminology, refer to the Unicode Glossary.

extrait de http://java.sun.com/javase/6/docs/api/java/lang/Character.html

Ensuite comprendre la String

Ensuite comprendre qu’une String est encodée en UTF-16, la problématique du charset est à  gérer en entrée et en sortie du code java pas en interne… :

A String represents a string in the UTF-16 format in which supplementary characters are represented by surrogate pairs (see the section Unicode Character Representations in the Character class for more information). Index values refer to char code units, so a supplementary character uses two positions in a String.

The String class provides methods for dealing with Unicode code points (i.e., characters), in addition to those for dealing with Unicode code units (i.e., char values).

extrait de http://java.sun.com/javase/6/docs/api/java/lang/String.html

Le lien entre le flux externe et la String

Ensuite comprendre comment extraire une String d’un flux (avec le bon encodage évidemment) :

http://java.sun.com/docs/books/tutorial/i18n/text/stream.html

Il faut penser à optimiser leur exemple avec :
BufferedReader in = new BufferedReader(new InputStreamReader(System.in));

et enfin i18n

Le tuto java sur l’internationalisation (i18n) est clair et se suffit à lui même.