How to handle non-ASCII strings properly in C?

Question

My idea was to code a Hangman-like game in C. I want it to be able to use German words with umlauts (eg: ä, ü, ö) and also Greek words (completely non-ASCII characters).

My compiler and my terminal can handle Unicode well. Displaying the strings works well.

But how should I do operations on these strings? For the German language I could maybe handle the 6 upper- and lowercase accented characters by taking care of these cases in the functions. But considering Greek it seems like impossible.

I wrote this test code. It outputs the string, the length of the string (of course wrong, because the UTF-8 sequences take the place of two characters), and the value of the individual characters of the string in plain text and hex.

#include <stdio.h>
#include <string.h>

int main() {
    printf("123456789\n");
    char aTestString[] = "cheese";
    printf("%s ist %d Zeichen lang\n", aTestString, strlen(aTestString));
        
    for (int i = 0; i < strlen(aTestString); i++) {
        printf("( %c )", aTestString[i]);   // char als char
        printf("[ %02X ]", aTestString[i]); // char in hexadezimal
    }

    printf("\n123456789\n");
    char aTestString2[] = "Käse";
    printf("%s has %d characters\n", aTestString2, strlen(aTestString2));
        
    for (int i = 0; i < strlen(aTestString2); i++) {
        printf("( %c )", aTestString2[i]);  // char als char
        printf("[ %02X ]", aTestString2[i]); // char in hexadezimal
    }
    
    printf("\n123456789\n");    
    char aTestString3[] = "λόγος";
    printf("%s has %d characters\n", aTestString3, strlen(aTestString3));

    for (int i = 0; i < strlen(aTestString3); i++) {
        printf("( %c )", aTestString3[i]);  // char als char
        printf("[ %02X ]", aTestString3[i]); // char in hexadezimal
    }
}

For example, what is the recommended way to count the Unicode characters, or to see whether a specific Unicode character (that is, code point) is in the string? I am quite sure there must some simple solution because such characters are often used in passwords for example.

Here the output of the test program:

123456789
cheese has 6 character
( c )[ 63 ]( h )[ 68 ]( e )[ 65 ]( e )[ 65 ]( s )[ 73 ]( e )[ 65 ]
123456789
Käse has 5 characters
( K )[ 4B ](  )[ FFFFFFC3 ](  )[ FFFFFFA4 ]( s )[ 73 ]( e )[ 65 ]
123456789
λόγος has 10 characters
(  )[ FFFFFFCE ](  )[ FFFFFFBB ](  )[ FFFFFFCF ](  )[ FFFFFF8C ](  )[ FFFFFFCE ](  )[ FFFFFFB3 ](  )[ FFFFFFCE ](  )[ FFFFFFBF ](  )[ FFFFFFCF ](  )[ FFFFFF82 ]

Answer 1

C's multi-byte string utilities are useful in this case. Using mbrlen, for example, one way to find the number of characters in a string (albeit probably a very naive one that I just bodged together right now) is this:

#include <stdio.h>
#include <wchar.h>
#include <locale.h>

size_t string_size(const char *s)
{
    mbstate_t state = {0};
    size_t len = 0;
    for (; *s != '\0'; ++len)
    {
        unsigned c_len;
        for (c_len = 1; mbrlen(s+c_len-1, 1, &state) == -2; ++c_len) {}
        s += c_len;
    }
    return len;
}

int main(void)
{
    setlocale(LC_ALL, "en_US.utf8");
    const char *s = "zß水";
    printf("%zu\n", string_size(s));
}

// Output: 4

Using the same function mbrlen, you could also extract individual characters through finding their lengths. There are also functions to convert between multibyte characters and wide characters if you want to work with that.

Answer 2

To handle Unicode characters correctly, you use wide characters wchar_t and wide string functions instead of regular characters char and string functions. The wide character and string functions are prefixed with a w (e.g., wprintf, wcslen, wcsstr) and are designed to handle multibyte characters properly.

An example of printing out a phrase with unicode characters:

#include <stdio.h>
#include <wchar.h>
#include <locale.h>

int main() {
    setlocale(LC_ALL, ""); // Set the locale to handle wide characters

    wchar_t phrase[] = L"Hello, 世界!"; // Unicode phrase

    wprintf(L"%ls\n", phrase); // Print the Unicode phrase

    return 0;
}