Marshaling complex C structs with c unions for C#

Question

i am desperate to get a complex c datatype correctly marshaled for C#. I already read all the other posts regarding that topic and i am running out of ideas although it seems to me to be quite close to the solution.

The main issue is that the c-struct is having a union of two different struct types. On with only basic types and one including arrays, which causes trouble.

I have created an example to showcase the situation. The struct worring me is called dataStreamConfiguration. The c code looks like this, the struct in question is at the bottom of the example c-code:

#include "stdint.h"
#include "stddef.h"

typedef enum viewCapEnum {
    X = 0,
}viewCapEnum;

typedef struct fraction{
    uint8_t nominator;
    uint8_t denominator;
}fraction;

typedef struct comSize{
    fraction A;
    fraction B;
}comSize;

typedef enum someEnum{
    A = 0,
    B,
    C,
    D
}someEnum;

typedef struct someSize{
    fraction X;
    fraction Y;
}someSize;


typedef struct featTemplateCap{
    someEnum A;
    someSize Size;
}featTemplateCap;

typedef struct featTypeCap{
    someEnum AB;
    someSize CD;
}featTypeCap;


typedef struct viewCap{
 uint8_t A;
 uint8_t B;
 size_t  BCount;
 viewCapEnum ViewCapEnum[50];
 comSize MinComSize;
 size_t CapaCount;
 featTemplateCap TemplCap[14];
 size_t TypeCapaCount;
 featTypeCap FeatTypeCapa[14];
 uint8_t GCount;
}viewCap;

typedef struct featX{
    uint16_t A;
    uint16_t B;
    int16_t  C;
    int16_t  D;
}featX;


typedef struct pathCap{
    uint8_t Count;
    uint8_t Size;
    featX   Feat;
}pathCap;


typedef struct dataStreamConfiguration{
  size_t FeatureSelector;
  union {
    viewCap  AsViewCap;
    pathCap  AsPathCap;
  }dataStream;
}dataStreamConfiguration;

The marshalling of datatypes between C and the C# world is working for almoust all but this dataStreamConfiguration struct. So I got the following code, where instead of mapping (somehow) a union to c# both datatypes have been put one after another. So clearly this was not working correctly. It looked like that:

public unsafe struct UInt32Struct {
    public UInt32 value;
}

public unsafe struct fraction{
    public Byte nominator;
    public Byte denominator;
}

public unsafe struct comSize{
    public fraction A;
    public fraction B;
}

public unsafe struct someSize{
    public fraction X;
    public fraction Y;
}


public unsafe struct featTemplateCap{
    public UInt32 A;
    public someSize Size;
}

public unsafe struct featTypeCap{
    public UInt32   AB;
    public someSize CD;
}


public unsafe struct viewCap{
 public Byte A;
 public Byte B;
 public UInt16 BCount;
 [MarshalAs(UnmanagedType.ByValArray, SizeConst = 50] 
 public UInt32Struct[] ViewCapEnum;
 public comSize MinComSize;
 public UInt16 CapaCount;
 [MarshalAs(UnmanagedType.ByValArray, SizeConst = 14] 
 public featTemplateCap[] TemplCap;
 public UInt16 TypeCapaCount;
 [MarshalAs(UnmanagedType.ByValArray, SizeConst = 14] 
 public featTypeCap FeatTypeCapa[14];
 public Byte GCount;
}

public unsafe struct featX{
    public UInt16 A;
    public UInt16 B;
    public Int16  C;
    public Int16  D;
}


public unsafe struct pathCap{
    public Byte Count;
    public Byte Size;
    public featX Feat;
}

public unsafe struct dataStreamConfiguration{
    public UInt16 FeatureSelector;
    public viewCap AsViewCap;
    public pathCap AsPathCap;
}

So to get the union to c# I came across the LayoutKind.Explicit and did following:

[StructLayout(LayoutKind.Explicit)]
public unsafe struct dataStreamConfiguration{
    [FieldOffset(0)]
    public UInt16 FeatureSelector;
    [FieldOffset(2)]
    public viewCap AsViewCap;
    [FieldOffset(2)]
    public pathCap AsPathCap;
}

This was not working due to the alignment of the object types, which are incorrectly aligned or overlapped by non-object fields.. I googled a lot. Adjusted the alignment to 4 by [StructLayout(LayoutKind.Explicit, Pack=4)]. However, 4,8,16,32, whatever alignment i have choosen, I got the same error during runtime - incorrectly aligned or overlapped issue.

Next thing I did - I felt quite lucky about - was to unroll all the arrays in C# datatype for all the arrays in viewCap struct. As I have read that this might cause alignment issues. Well, It didn't work. And I found that the memory has been modified, so I could not find the values I have seen in C appearing now in C#. Most of the values in C# are 0. Ok. To get rid of this memory modification stuff I put in C# to all other structs [StructLayout(LayoutKind.Sequential)] to keep the order of elements as they are in C. Sadly it didn't help much, I could not find the values of the c-struct in c# either. However, it was finally working, when I got rid of the union and deleted either AsViewCap or AsPathCap (my weak moment of blind rage). Ok, but that was not the solution.

Last help was having a try with IntPtr, so i have created a new struct called dataStreamConfigurationPtr:

public unsafe struct dataStreamConfigurationPtr{
    public UInt16 FeatureSelector;
    public void* Ptr;
}

[StructLayout(LayoutKind.Sequential)]
public unsafe struct dataStreamConfiguration{
    public UInt16 FeatureSelector;
    public viewCap AsViewCap;
    public pathCap AsPathCap;
}

Instead of having an overlapping memory with StructLayout.Explicit I used an void* to point to the unmanaged memory location. For this I used the old struct definition to get the memory and instead having a union I took the first version where both types are laid out one over another. The idea was to use it like that:

MyFunction(dataStreamConfigurationPtr X, int Status){
    
    //Create obj and  appropraite space for data
    dataStreamConfiguration DataStream = new dataStreamConfiguration();
    DataStream.FeatureSelector = X.FeatureSelector;
    
    unsafe{
        IntPtr Ptr = new IntPtr(&X.Ptr);
        DataStream.AsViewCap = Marshal.PtrToStructure<viewCap>(Ptr);
        DataSteram.AsPathCap = Marshal.PtrToStructure<pathCap>(Ptr);
    }
    
    WCFCallback(DataStream, Status);
    
}

Now the IntPtr is pointing to the right memory, however, this works only for the first item of the structs. So for viewCap the first item A has its correct data whereas item B, BCount,.. all the other item seem to have at least misalligned values or accidental values.. I am quite desperate what to do now, i feel i am so close to a solution but have no idea how to get the other data of the struct from c to c#.

Any suggestions and comments are highly welcome!

Best regards, Tobias

Answer 1

I would assume you have two use cases and want to interprete the union part based FeatureSelector either as AsViewCap or as AsPathCap on C# side. That means I assume that you don't intend to do type punning.

One could create two structs then on the managed C# side:

public struct dataStreamConfigurationAsViewCap
{
    public UInt64 FeatureSelector;
    public viewCap AsViewCap;
}

public struct dataStreamConfigurationAsPathCap
{
    public UInt64 FeatureSelector;
    public pathCap AsPathCap;
}

You could then only inspect FeatureSelector first and based on the result either interprete it dataStreamConfigurationAsViewCap or as dataStreamConfigurationAsPathCap.

Size You have several variables with size_t on C side (FeatureSelector, BCount, CapaCount, TypeCapaCount) which you map all to UInt16, which is wrong. UInt16 is the minimum size in the C standard, but the usual implementations especially on platforms running .NET are larger, see also this nice answer. For example on my macOS machine it is 8 bytes.

Maybe it would be a good idea to start with a smaller test case and expand it step by step so that you can identify such problems. And when you encounter a problem, you can more easily create a minimal, complete and testable example.

One approach in this direction could be the following:

Small Test Case

some.h

#ifndef some_h
#define some_h

#include <stdio.h>

typedef struct viewCap {
    uint8_t A;
    uint8_t B;
} viewCap;

typedef struct pathCap {
    uint16_t X;
    uint16_t Y;
    size_t Num;
} pathCap;

typedef struct dataStreamConfiguration {
    size_t FeatureSelector;
    union {
        viewCap  AsViewCap;
        pathCap  AsPathCap;
    } dataStream;
} dataStreamConfiguration;

dataStreamConfiguration *dscViewCap(void);
dataStreamConfiguration *dscPathCap(void);
extern void free_struct(dataStreamConfiguration *ptr);


#endif /* some_h */

some.c

#include "some.h"
#include <stdlib.h>
#include <stdint.h>
#include <stddef.h>


dataStreamConfiguration *dscViewCap(void) {
    dataStreamConfiguration *dsc = calloc(1, sizeof(dataStreamConfiguration));
    dsc->FeatureSelector = 0;
    dsc->dataStream.AsViewCap.A = 42;
    dsc->dataStream.AsViewCap.B = 84;
    return dsc;
}

dataStreamConfiguration *dscPathCap(void) {
    dataStreamConfiguration *dsc = calloc(1, sizeof(dataStreamConfiguration));
    dsc->FeatureSelector = 1;
    dsc->dataStream.AsPathCap.X = 0xAAAA;
    dsc->dataStream.AsPathCap.Y = 0x5555;
    dsc->dataStream.AsPathCap.Num = 0x3333333333333333;
    return dsc;
}


void free_dsc(dataStreamConfiguration *ptr) {
    free(ptr);
}

int main(void) {
    return 0;
}

UnionFromC.cs

using System.Runtime.InteropServices;

namespace UnionFromC
{
    public static class Program
    {
        public struct viewCap
        {
            public Byte A;
            public Byte B;
        }

        public struct pathCap
        {
            public UInt16 X;
            public UInt16 Y;
            public UInt64 Num;
        } 
        
        public struct dataStreamConfigurationAsViewCap
        {
            public UInt64 FeatureSelector;
            public viewCap AsViewCap;
        }
        
        public struct dataStreamConfigurationAsPathCap
        {
            public UInt64 FeatureSelector;
            public pathCap AsPathCap;
        }


        [DllImport("StructLib", EntryPoint = "dscViewCap")]
        private static extern IntPtr NativeDSCViewCap();

        [DllImport("StructLib", EntryPoint = "dscPathCap")]
        private static extern IntPtr NativeDSCPathCap();

        [DllImport("StructLib", EntryPoint = "free_dsc")]
        private static extern void NativeFreeDSC(IntPtr ptr);

        static void Main()
        {
            IntPtr _intPtrViewCap = NativeDSCViewCap();
            var viewDSC = Marshal.PtrToStructure<dataStreamConfigurationAsViewCap>(_intPtrViewCap);
            Console.WriteLine("\nInterpretation as view cap:");
            Console.WriteLine($"  FeatureSelector: {viewDSC.FeatureSelector}");
            Console.WriteLine($"  A: {viewDSC.AsViewCap.A}");
            Console.WriteLine($"  B: {viewDSC.AsViewCap.B}");
            NativeFreeDSC(_intPtrViewCap);
            
            IntPtr _intPtrPathCap = NativeDSCPathCap();
            var pathDSC = Marshal.PtrToStructure<dataStreamConfigurationAsPathCap>(_intPtrPathCap);
            Console.WriteLine("\nInterpretation as path cap:");
            Console.WriteLine($"  FeatureSelector: {pathDSC.FeatureSelector}");
            Console.WriteLine($"  A: {pathDSC.AsPathCap.X:X4}");
            Console.WriteLine($"  B: {pathDSC.AsPathCap.Y:X4}");
            Console.WriteLine($"  Num: {pathDSC.AsPathCap.Num:X8}");
            NativeFreeDSC(_intPtrPathCap);
        }
    }
}

Output of the Test

Interpretation as view cap:
  FeatureSelector: 0
  A: 42
  B: 84

Interpretation as path cap:
  FeatureSelector: 1
  A: AAAA
  B: 5555
  Num: 3333333333333333

Answer 2

This is one solution how to get data from C/C++ to C#. Here I will describe what I did wrong and what one has to take care of.

To recall in our minds my requirement has been (still is) that any data represented as union in C/C++, needs to be represented as such in C#. This means for following structure:

typedef struct dataStreamConfiguration{
  size_t FeatureSelector;
  union {
    viewCap  AsViewCap;
    pathCap  AsPathCap;
  }dataStream;
}dataStreamConfiguration;

Any data inside AsViewCap must have its kind of representation in AsPathCap, because its simple the same memory. If one of these two are modified, the other is also. To handle C/C++ unions in C# you need to provide a memory layout. As Stephan Schlecht already meantioned it is vital to know the alignment! My project is compiled for 32bit the alignment is at the 4 Byte boarder. Thus my initial layout in my question was simply wrong. You need to check the layout in your C/C++ project and adjust it in the C# struct definion properly: Here is my corrected code, both union members start at 4th Byte:

[StructLayout(LayoutKind.Explicit)]
public unsafe struct dataStreamConfiguration{
    [FieldOffset(0)]
    public UInt16 FeatureSelector;
    [FieldOffset(4)]
    public viewCap AsViewCap;
    [FieldOffset(4)]
    public pathCap AsPathCap;
}

Doing this you are half way there! Yes! But there is one more thing. With this change the Code will compile but, you will get an exception at runtime. Yes, at runtime. Quite quick but it is so. The error message something like: "object-field at offset 4 is incorrectly aligned or overlapped by a non-object field" C# is bugging is because in C# there are basic types like Integer etc. and reference types.

These reference types can give us errors if we do not handel them correctly. C# has a very nice working marshalling, but in case of unions, its up to you to make it as nice as it can get.

Explenation: What went wrong in my code is that the struct viewCap is having arrays, marshalled by the C# marshaller. The marshaller is doing his duty and creating an array. However, an array is a reference type and created on the heap. What you will get on the stack (data transfer C++ <-> C#) is the reference address to the array on the heap. Hmpf! Thus the second structure in the union with its basic types would overlapping the address and thus invalidating the reference. Gladly the runtime environment stops us from doing that :-) Moreover, C# is defragmenting the memory. If the struct layout is not efficient regarding memory usage C# will reoder the content. You can avoid this by annotating the struct which the layout kind: Sequential.

Remember: If you got an array inside a type which is part of a union (C/C++), you cannot use the C# marshaller! Use Layoutkind Explicit for unions and Sequential for structs!

Solution: There might be several other solutions I am not aware of, but what is 100% working is to unroll the arrays. Yes, it's a lot of work. But it works!

So the final struct looks like this:

[StructLayout(LayoutKind.Sequential)]
public unsafe struct viewCap{
 public Byte A;
 public Byte B;
 public UInt16 BCount;
 public UInt32Struct ViewCapEnum_0;
 public UInt32Struct ViewCapEnum_1;
 public UInt32Struct ViewCapEnum_2;
 public UInt32Struct ViewCapEnum_3;
 [...]
 public comSize MinComSize;
 public UInt16 CapaCount;
 public featTemplateCap TemplCap_0;
 public featTemplateCap TemplCap_1;
 public featTemplateCap TemplCap_2;
 public featTemplateCap TemplCap_3;
 [...]
 public UInt16 TypeCapaCount;
 public featTypeCap FeatTypeCapa_0;
 public featTypeCap FeatTypeCapa_1;
 public featTypeCap FeatTypeCapa_2;
 public featTypeCap FeatTypeCapa_3;
 [...]
 public Byte GCount;
}

Happy runtime environment, happy life!

Yes, for this specific solution you would need to adjust the code which is using the arrays. However, its bulletproof and understandable how this works under the hood which makes it easy for maintenance. As my code is generated, an unrolled array is no big deal.