Mac device access works in C, but equivalent code in Java/JNA not

Question

We work with a serial device that is attached to a Mac on USB and need to configure the DTR/RTS-line settings. Technically this involves usage of open(3), ioctl(3).

We implemented this in C and it worked. Below is a very simplified snippet showing the core part.

Then we migrated the code to Java/JNA and run into the problem that the ported code did not work, though it is basically a line-by-line conversion of the C code.

The question is where do we go wrong?

Symptom of the failure in Java is errno=25 'Inappropriate ioctl for device' returned from the call to ioctl(). Since it works in C it seems that we do somwthing wrong in JNA.

What we did:

• Checked the values from the header constants. Note that the C-code generates Java-compatible constant definitions that are used in the Java code.
• Checked the signature of ioctl(). Seems to be right according to the man-page and include files.
• Guessed that the problem is that the ioctl code for TIOCMSET is not properly passed since it is negative.

We use JNA 5.5.0.

Here comes the C code. The snippet simply reads the settings of the lines and writes them back unmodified for demo purposes. Here's the code (note the hard-coded device name).

int main(int argc, char **argv)
{
    // Print constant values.
    printf( "long TIOCMGET = 0x%x;\n", TIOCMGET );
    printf( "long TIOCMSET = 0x%x;\n", TIOCMSET );
    printf( "int O_RDWR = 0x%x;\n", O_RDWR );
    printf( "int O_NDELAY = 0x%x;\n", O_NDELAY );
    printf( "int O_NOCTTY = 0x%x;\n", O_NOCTTY );

    int value = O_RDWR|O_NDELAY|O_NOCTTY;
    printf( "value=%x\n", value );
    int portfd = open("/dev/tty.usbmodem735ae091", value);
    printf( "portfd=%d\n", portfd );

    int lineStatus;
    printf( "TIOCMGET %x\n", TIOCMGET );
    int rc = ioctl( portfd, TIOCMGET, &lineStatus );
    printf( "rc=%d, linestatus=%x\n", rc, lineStatus );

    rc = ioctl( portfd, TIOCMSET, &lineStatus );
    printf( "rc=%d, linestatus=%x\n", rc, lineStatus );

    if ( rc == -1 )
        printf( "Failure\n" );
    else
        printf( "Success\n" );

    if ( portfd != -1 )
        close( portfd );

    return 0;
}

Output of the above is:

long TIOCMGET = 0x4004746a;
long TIOCMSET = 0x8004746d;
int O_RDWR = 0x2;
int O_NDELAY = 0x4;
int O_NOCTTY = 0x20000;
value=20006
portfd=3
TIOCMGET 4004746a
rc=0, linestatus=6
rc=0, linestatus=6
Success

Here's the Java implementation:

public class Cli
{
    /**
     * Java mapping for lib c
     */
    public interface MacCl extends Library {
        String NAME = "c";
        MacCl INSTANCE = Native.load(NAME, MacCl.class);

        int open(String pathname, int flags);
        int close(int fd);
        int ioctl(int fd, long param, LongByReference request);
        String strerror( int errno );
    }

    private static final MacCl C = MacCl.INSTANCE;

    private static PrintStream out = System.err;

    public static void main( String[] argv )
    {
        long TIOCMGET = 0x4004746a;
        long TIOCMSET = 0x8004746d;
        int O_RDWR = 0x2;
        int O_NDELAY = 0x4;
        int O_NOCTTY = 0x20000;

        int value = O_RDWR|O_NDELAY|O_NOCTTY;
        out.printf( "value=%x\n", value );
        int portfd = C.open(
                "/dev/tty.usbmodem735ae091",
                value );
        out.printf( "portfd=%d\n", portfd );

        LongByReference lineStatus = new LongByReference();

        int rc = C.ioctl( portfd, TIOCMGET, lineStatus );
        out.printf(
                "rc=%d, linestatus=%d\n", rc, lineStatus.getValue() );

        rc = C.ioctl( portfd, TIOCMSET, lineStatus );
        out.printf(
                "rc=%d errno='%s'\n",
                rc,
                C.strerror( Native.getLastError() ) );

        if ( rc == -1 )
            out.print( "Failure." );
        else
            out.print( "Success." );

        if ( portfd != -1 )
            C.close( portfd );
    }
}

Java output is:

value=20006
portfd=23
rc=0, linestatus=6
rc=-1 errno='Inappropriate ioctl for device'
Failure.

Answer 1

A review of the ioctl.h header file for the commands you are using reveals that it expects an int as the third argument:

#define TIOCMSET    _IOW('t', 109, int) /* set all modem bits */
#define TIOCMGET    _IOR('t', 106, int) /* get all modem bits */

You correctly define a 4-byte int in your C code and pass a reference to it, which works:

int lineStatus;
int rc = ioctl( portfd, TIOCMGET, &lineStatus );
rc = ioctl( portfd, TIOCMSET, &lineStatus );

However, in your Java code, you define an 8-byte long reference to pass:

LongByReference lineStatus = new LongByReference();
int rc = C.ioctl( portfd, TIOCMGET, lineStatus );
rc = C.ioctl( portfd, TIOCMSET, lineStatus );

The "get" appears to work because the native code is only filling in 4 of the 8 bytes and it happens to be the low order bytes, but you may be corrupting the stack with the overallocation.

As @nyholku points out in the comments, in addition from switching from long to int you may need to pass the int (rather than the pointer) to the TIOCMSET version of the command. There is conflicting documentation but the examples I see in the wild favor your pointer implementation.

So your code should include:

IntByReference lineStatus = new IntByReference();
int rc = C.ioctl( portfd, TIOCMGET, lineStatus );
// Possible, per the page @nyholku linked:
rc = C.ioctl( portfd, TIOCMSET, lineStatus.getValue() );
// Probable, per the man pages and other examples:
rc = C.ioctl( portfd, TIOCMSET, lineStatus );

You do say the C version without the pointer "works" but only in the sense that it does not throw an error. To confirm what "works", you should read the bytes again to make sure whatever you set actually stuck.

Answer 2

We checked many times the third argument. Must a pointer be passed or not? The docs we found -- Man page man -s 4 tty on the Mac -- really documents to pass a pointer. So there seem to be differences between Unix implementations.

Finally we found the solution by printing the passed value using printf( "%xl", ... );. and the resulting value was 0xffffffff8004746d. So we got an unexpected sign extension.

And the problem is the line

        long TIOCMSET = 0x8004746d;

The literal constant is defined as an int literal that gets implicitly converted to long with sign extension. Since 0xffffffff8004746d is not equal to 0x8004746d' this explains the error message inappropriate ioctl for device. When we changed the line above to

        long TIOCMSET = 0x8004746dL; // Note the 'L' at the end.

everything worked perfectly. On other Unixes we did not have the problem because the TIO... constants happened to be positive.