Segmentation fault when using C callback user data to store a boxed Rust closure

Question

I am creating a Rust wrapper around a C API. One function in this C API sets a callback and accepts a void pointer which will be passed to the callback. It stores a reference to the callback and user data for later, so I am using the last code section from this answer.

Here is my code. The Test::trigger_callback(...) function is meant to emulate the C library calling the callback.

extern crate libc;

use libc::c_void;
use std::mem::transmute;

struct Test {
    callback: extern "C" fn(data: i32, user: *mut c_void) -> (),
    userdata: *mut c_void,
}

extern "C" fn c_callback(data: i32, user: *mut libc::c_void) {
    unsafe {
        println!("Line {}. Ptr: {}", line!(), user as u64);
        let func: &mut Box<FnMut(i32) -> ()> = transmute(user);
        println!("Line {}. Data: {:?}", line!(), data);
        (*func)(data);
        println!("Line {}", line!());
    }
}

impl Test {
    fn new<F>(func: F) -> Test
    where
        F: FnMut(i32) -> (),
        F: 'static,
    {
        let func = Box::into_raw(Box::new(Box::new(func)));
        println!("Line: {}, Ptr: {}", line!(), func as u64);

        Test {
            callback: c_callback,
            userdata: func as *mut c_void,
        }
    }

    fn trigger_callback(&self, data: i32) {
        (self.callback)(data, self.userdata);
    }
}

fn main() {
    let test = Test::new(|data: i32| {
        println!("Inside callback! Data: {}", data);
    });

    test.trigger_callback(12345);
}

As mentioned in the linked answer, Boxing the closure is meant to store it on the heap so that a pointer to it will be valid for an arbitrarily long time, and then Boxing that Box is because it's a fat pointer, but needs to be converted to a regular pointer so that it can be cast to a void pointer.

When run, this code prints out:

Line: 29, Ptr: 140589704282120
Line 13. Ptr: 140589704282120
Line 15. Data: 12345
Segmentation fault (core dumped)

It segfaults when trying to call the closure inside of the extern "C" function.

Why? As far as I understand it, putting the closure in a Box and then using Box::into_raw(...) should store it on the heap and 'leak' the memory, so the pointer should be valid for as long as the program is running. What part of this is wrong?

@Shepmaster I looked at the linked answers, and they aren't relevant after my removal of `String`s. I looked at your "Omnibus", and it's not relevant due to lack of information about callbacks. So I will keep this question open for now. Thanks for the resources! — ItsAmy, Jun 20 '18 at 20:12

Shepmaster · Accepted Answer · 2018-06-20T20:36:22.207

Box::into_raw(Box::new(Box::new(func)));

This does not produce the type you think it does:

   = note: expected type `()`
              found type `*mut std::boxed::Box<F>`

You assume it's a trait object:

let func: &mut Box<FnMut(i32) -> ()> = transmute(user);

Instead, make the input value into a trait object when you box it. I advocate for explicit lines with comments explaining each step:

// Trait object with a stable address
let func = Box::new(func) as Box<FnMut(i32)>;
// Thin pointer
let func = Box::new(func);
// Raw pointer
let func = Box::into_raw(func);

Box<FnMut(i32) -> ()>

The return type of () is redundant; use Box<FnMut(i32)>

let func: &mut Box<FnMut(i32) -> ()> = transmute(user);

Try really hard to avoid transmute. There are usually smaller tools to use:

extern "C" fn c_callback(data: i32, user: *mut libc::c_void) {
    let user = user as *mut Box<FnMut(i32)>;
    unsafe {
        (*user)(data);
    }
}

Avoid restating the same type all over. Introduce a type alias:

type CallbackFn = Box<FnMut(i32)>;

let user = user as *mut CallbackFn;

let func = Box::new(func) as CallbackFn;

Segmentation fault when using C callback user data to store a boxed Rust closure

1 Answers1

Linked