How do I pass a closure through raw pointers as an argument to a C function?

Question

I'm working with WinAPI in Rust and there are some functions (like EnumWindows()) which require a callback. The callback usually accepts an additional argument (of type LPARAM which is an alias for i64), which you can use to pass some custom data to the callback.

I have sent Vec<T> objects as LPARAM to the WinAPI callbacks and it worked fine. For instance "unpacking" an lparam value to a Vec<RECT> looked like this in my case:

unsafe extern "system" fn enumerate_callback(hwnd: HWND, lparam: LPARAM) -> BOOL {
    let rects = lparam as *mut Vec<RECT>;
}

Instead of passing a vector, I now have to pass a closure. I can not use a function pointer as my closure has to capture some variables, which would not be accessible if I used a function. In C++, I would use std::function<> for my particular task, I think that in Rust the corresponding abstraction is a closure.

My code for unpacking looks like this:

unsafe extern "system" fn enumerate_callback(hwnd: HWND, lparam: LPARAM) -> BOOL {
    let cb: &mut FnMut(HWND) -> bool = &mut *(lparam as *mut c_void as *mut FnMut(HWND) -> bool);
    // ...
}

SSCCE:

use std::os::raw::c_void;

fn enum_wnd_proc(some_value: i32, lparam: i32) {
    let closure: &mut FnMut(i32) -> bool =
        unsafe { (&mut *(lparam as *mut c_void as *mut FnMut(i32) -> bool)) };

    println!("predicate() executed and returned: {}", closure(some_value));
}

fn main() {
    let sum = 0;
    let mut closure = |some_value: i32| -> bool {
        sum += some_value;
        sum >= 100
    };

    let lparam = (&mut closure as *mut c_void as *mut FnMut(i32) -> bool) as i32;
    enum_wnd_proc(20, lparam);
}

(Playground)

I get these errors:

error[E0277]: expected a `std::ops::FnMut<(i32,)>` closure, found `std::ffi::c_void`
 --> src/main.rs:5:26
  |
5 |         unsafe { (&mut *(lparam as *mut c_void as *mut FnMut(i32) -> bool)) };
  |                          ^^^^^^^^^^^^^^^^^^^^^ expected an `FnMut<(i32,)>` closure, found `std::ffi::c_void`
  |
  = help: the trait `std::ops::FnMut<(i32,)>` is not implemented for `std::ffi::c_void`
  = note: required for the cast to the object type `dyn std::ops::FnMut(i32) -> bool`

error[E0606]: casting `&mut [closure@src/main.rs:12:23: 15:6 sum:_]` as `*mut std::ffi::c_void` is invalid
  --> src/main.rs:17:19
   |
17 |     let lparam = (&mut closure as *mut c_void as *mut FnMut(i32) -> bool) as i32;
   |                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^

error[E0606]: casting `*mut dyn std::ops::FnMut(i32) -> bool` as `i32` is invalid
  --> src/main.rs:17:18
   |
17 |     let lparam = (&mut closure as *mut c_void as *mut FnMut(i32) -> bool) as i32;
   |                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |
   = help: cast through a thin pointer first

error[E0277]: expected a `std::ops::FnMut<(i32,)>` closure, found `std::ffi::c_void`
  --> src/main.rs:17:19
   |
17 |     let lparam = (&mut closure as *mut c_void as *mut FnMut(i32) -> bool) as i32;
   |                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^ expected an `FnMut<(i32,)>` closure, found `std::ffi::c_void`
   |
   = help: the trait `std::ops::FnMut<(i32,)>` is not implemented for `std::ffi::c_void`
   = note: required for the cast to the object type `dyn std::ops::FnMut(i32) -> bool`

I would like to know:

1. Is there a way to pass a function/closure to a different function and perform those "C-like" casts?
2. What is the proper way to cast a closure to a i64 value for passing it to that callback?

I'm using the stable version of Rust.

Answer 1

First, some logical errors with the code:

1. It is not correct to cast pointers to i32 on many platforms (like 64-bit). Pointers may use all of those bits. Truncating a pointer and then calling a function at the truncated address will lead to Really Bad Things. Generally you want to use a machine-sized integer (usize or isize).

2. The sum value needs to be mutable.

The meat of the problem is that closures are concrete types that take up a size unknown to the programmer, but known to the compiler. The C function is limited to taking a machine-sized integer.

Because closures implement one of the Fn* traits, we can take a reference to the closure's implementation of that trait to generate a trait object. Taking a reference a trait leads to a fat pointer that contains two pointer-sized values. In this case, it contains a pointer to the data that is closed-over and a pointer to a vtable, the concrete methods that implement the trait.

In general, any reference to or Box of a dynamically-sized type type is going to generate a fat pointer.

On a 64-bit machine, a fat pointer would be 128 bits in total, and casting that to a machine-sized pointer would again truncate the data, causing Really Bad Things to happen.

The solution, like everything else in computer science, is to add more layers of abstraction:

use std::os::raw::c_void;

fn enum_wnd_proc(some_value: i32, lparam: usize) {
    let trait_obj_ref: &mut &mut FnMut(i32) -> bool = unsafe {
        let closure_pointer_pointer = lparam as *mut c_void;
        &mut *(closure_pointer_pointer as *mut _)
    };
    println!(
        "predicate() executed and returned: {}",
        trait_obj_ref(some_value)
    );
}

fn main() {
    let mut sum = 0;
    let mut closure = |some_value: i32| -> bool {
        println!("I'm summing {} + {}", sum, some_value);
        sum += some_value;
        sum >= 100
    };

    let mut trait_obj: &mut FnMut(i32) -> bool = &mut closure;
    let trait_obj_ref = &mut trait_obj;

    let closure_pointer_pointer = trait_obj_ref as *mut _ as *mut c_void;
    let lparam = closure_pointer_pointer as usize;

    enum_wnd_proc(20, lparam);
}

We take a second reference to the fat pointer, which creates a thin pointer. This pointer is only one machine-integer in size.

Maybe a diagram will help (or hurt)?

Reference -> Trait object -> Concrete closure
 8 bytes       16 bytes         ?? bytes

Because we are using raw pointers, it is now the programmers responsibility to make sure that the closure outlives where it is used! If enum_wnd_proc stores the pointer somewhere, you must be very careful to not use it after the closure is dropped.

---

As a side note, using mem::transmute when casting the trait object:

use std::mem;
let closure_pointer_pointer: *mut c_void = unsafe { mem::transmute(trait_obj) };

Produces a better error message:

error[E0512]: transmute called with types of different sizes
  --> src/main.rs:26:57
   |
26 |     let closure_pointer_pointer: *mut c_void = unsafe { mem::transmute(trait_obj) };
   |                                                         ^^^^^^^^^^^^^^
   |
   = note: source type: &mut dyn std::ops::FnMut(i32) -> bool (128 bits)
   = note: target type: *mut std::ffi::c_void (64 bits)

Error E0512.

---

See also

• Pass a Rust trait to C
• Rust FFI passing trait object as context to call callbacks on
• How do I create a Rust callback function to pass to a FFI function?
• How do I convert a Rust closure to a C-style callback?