Java: Why are Chinese/Japanese characters displaying as '???' when I append data to a text file, whilst running the .jar independant of Netbeans?

Question

I had no problem getting these characters (eg. 戦う) to display properly while I was running the program within Netbeans. Earlier, I added the parameter -J-Dfile.encoding=UTF-8 to the netbeans.conf file.

Essentially, I am creating a program to append data onto the end of a text file. Whilst running through Netbeans, the data would be saved correctly and the characters would show up when I opened the file. However, when I run the .jar independent of Netbeans (or run a built .exe) it just saves the characters to the file as '????'.

The code below shows the method that appends data onto the file. boxValue[] stores the String data that I am appending. When running the program through Netbeans, the file output would look like this when I opened the file:

食べる to eat - Plain: Present たべる 5 食べる

Running the program independently, without Netbeans, would produce this in the text file:

??? to eat - Plain: Present ??? 5 ???

private void prepareFile(String[] boxValue, boolean ruVerbIN, String addressIN){

 try
 {     
     int counter = 0; 
     int counter2;

     if(ruVerbIN == false)
     {
         counter2 = 63;
     }
     else
     {
         counter2 = 55;
     }

    File wordFile = new File(addressIN);
    FileWriter fileWriter = new FileWriter(wordFile, true);
    BufferedWriter buffer = new BufferedWriter(fileWriter);
    PrintWriter printWriter = new PrintWriter(buffer);

    while(counter <= counter2)
    {     
        printWriter.println(boxValue[counter]);        
        counter++;       
    }
    printWriter.close();
    counter = 0;
    outputTextBox.setText("Export successful.");
 }
 catch(IOException e)
 {     
     outputTextBox.setText("There was an error. Are you sure you entered the directory correctly? For example:" + "\n\n" + "\"C:\\\\Users\\\\Jayden\\\\Documents\\\\FLTR\\\\FLTR Data\\\\Japanese_Words.csv\"");
 }

}

you might need to pass `UTF-8`, follow [this](http://stackoverflow.com/questions/4192186/setting-a-utf-8-in-java-and-csv-file) — Ankur Singhal, Jan 21 '15 at 09:03

score 2 · Accepted Answer · answered Jan 21 '15 at 09:09

2

Have you tried to read the file again in your program? On windows-machines, the default encoding is CP-1252 and not UTF-8.

You can also use the BufferedWriter with a given charset

FileWriter fw = new FileWriter(new OutputStreamWriter(new FileOutputStream(file), "UTF-8"));

or with Java7

    Charset charset = Charset.forName("UTF-8");
    BufferedWriter writer = Files.newBufferedWriter(p.resolve("foo.txt"), charset)

answered Jan 21 '15 at 09:09

Steffen

341
1
12

1

"On windows-machines, the default encoding is CP-1252" Not always, it depends on the system locale – yet another reason to not use the defaults. – Karol S Jan 22 '15 at 11:29

score 1 · Answer 2 · answered Jan 21 '15 at 09:08

The documentation for FileWriter:

Convenience class for writing character files. The constructors of this class assume that the default character encoding and the default byte-buffer size are acceptable. To specify these values yourself, construct an OutputStreamWriter on a FileOutputStream.

So the encoding used is dependent on the runtime environment.

Always explicitly provide an encoding:

try(OutputStream out = new FileOutputStream(file);
    Writer writer = new OutputStreamWriter(out, StandardCharsets.UTF_8) {
  // I/O
} catch (IOException e) {
  // error handling
}

Java: Why are Chinese/Japanese characters displaying as '???' when I append data to a text file, whilst running the .jar independant of Netbeans?

2 Answers2