Fixing "Cannot marshal 'return value'" exception

Question

I call a method from .so lib written in C using C#. That's the C# code that is used:

    [DllImport("liblab.so")]
    
    static extern char[] entrance(
        [MarshalAs(UnmanagedType.LPWStr)]string path, 
        [MarshalAs(UnmanagedType.LPWStr)]string command);

    static void Main(string[] args)
    {
            command = Console.ReadLine();
            output = entrance(path, command).ToString();
            Console.WriteLine(output);
 }

(path is a hardcoded string there).

And that's the code that is called in C:

char* entrance(char* path, char* command1){
struct state* fs_state = setup(path);
if (fs_state != NULL){
    fgets_wrapper(command1, LINE_MAX, stdin);
    struct commands command = parse_command(command1);
    return execute_operation(command, fs_state);
}
else return "No data";
}

Calling from C# function entrance causes Unhandled exception. System.Runtime.InteropServices.MarshalDirectiveException: Cannot marshal 'return value': Invalid managed/unmanaged type combination. error. I tried different encodings, but neither of them worked.

How can I solve that?

If in C# `char*` is a `string` for parameters (that's logical), why not also use `string` for the returned type? Why a `char[]`? That's not the same thing, isn't it? [PInvoke char* in C DLL handled as String in C#. Issue with null characters](https://stackoverflow.com/questions/15273783/pinvoke-char-in-c-dll-handled-as-string-in-c-issue-with-null-characters) | [How To P/Invoke char*\[\] in C#](https://stackoverflow.com/questions/25137788/how-to-p-invoke-char-in-c-sharp) — , Jun 02 '21 at 22:45
So yes, I use ToString() here to convert char[] return type to string. Do you think it's better to use MarshalAs for `entrance` in DllImport? — coder-coder, Jun 02 '21 at 22:54
I'm not really advanced in P/Invoke and only use a few predefined WinAPI signatures, but as said it looks to me like you need: `static extern string entry (...)` which you will use without ToString, because the return type is the same as the parameters, and you have to be consistent, using the same type for all, right? — , Jun 02 '21 at 22:57
Does this answer your question? [Returning a string from PInvoke?](https://stackoverflow.com/questions/5298268/returning-a-string-from-pinvoke) and [PInvoke for C function that returns char *](https://stackoverflow.com/questions/370079/pinvoke-for-c-function-that-returns-char) — , Jun 02 '21 at 23:07
How does execute_operation work. Does it return a string literal, or does it return a dynamically allocates string? — David Heffernan, Jun 03 '21 at 08:35
Assuming the strings are ANSI on the C side then the declaration should be `[DllImport("liblab.so")] [return: MarshalAs(UnmanagedType.LPStr)] static extern string entrance( [MarshalAs(UnmanagedType.LPStr)] string path, [MarshalAs(UnmanagedType.LPStr)] string command);` — Charlieface, Jun 03 '21 at 11:43
@Charlieface, thanks, that's how I really solved that problem! Write this as an answer, I'll accept it. — coder-coder, Jun 06 '21 at 21:25

score 1 · Accepted Answer · answered Jun 07 '21 at 09:25

The strings in the C code appear to be ANSI.

So the C# import declaration should be

[DllImport("liblab.so")]
[return: MarshalAs(UnmanagedType.LPStr)]
static extern string entrance(
    [MarshalAs(UnmanagedType.LPStr)] string path,
    [MarshalAs(UnmanagedType.LPStr)] string command);

Fixing "Cannot marshal 'return value'" exception

1 Answers1