Borrowed value does not live long enough in testing methods

Question

I am quite new to Rust (trying to move from Golang). So I've written a piece of code in Go and decided to try to write it in Rust.

The actual code is below:

use rand::Rng;
use std::vec::Vec;
use std::thread::{self, JoinHandle};

pub struct WorkManager {
    pub threshold: i32,
}

impl WorkManager {
    pub fn new(trs: i32) -> Self {
        Self { threshold: trs }
    }

    pub fn run_job<T: Send + Sync>(&self, input: &'static Vec<T>, f: fn(&T) -> T) -> Vec<T> {
        if input.len() > self.threshold as usize {
            let mut guards: Vec<JoinHandle<Vec<T>>> = vec!();
            for chunk in input.chunks(self.threshold as usize) {
                let chunk = chunk.to_owned();
                let g = thread::spawn(move || chunk.iter().map(|x| f(x)).collect());
                guards.push(g);
            };
            let mut result: Vec<T> = Vec::with_capacity(input.len());
            for g in guards {
                result.extend(g.join().unwrap().into_iter());
            }
            result
        } else {
            return input.iter().map(|x| f(x)).collect::<Vec<T>>();
        }
    }
}
#[cfg(test)]
mod tests {
    use super::*;
    const TRS_TEST: i32 = 10;

    fn testing_function(a: &u32) -> u32 {
        return 3 * a;
    }

    #[test]
    fn test_single() {
        let wm: WorkManager = WorkManager::new(TRS_TEST);
        let values: Vec<u32> = rand::thread_rng()
            .sample_iter(&Uniform::from(0..20))
            .take(TRS_TEST as usize - 1)
            .collect();
        assert_eq!(
            wm.run_job(&values, testing_function),
            values
                .iter()
                .map(|x| testing_function(x))
                .collect::<Vec<u32>>()
        );
    }
}

So the problem is when I run cargo test:

error[E0597]: `values` does not live long enough
  --> src/lib.rs:50:24
   |
50 |             wm.run_job(&values, testing_function),
   |             -----------^^^^^^^-------------------
   |             |          |
   |             |          borrowed value does not live long enough
   |             argument requires that `values` is borrowed for `'static`
...
56 |     }
   |     - `values` dropped here while still borrowed

Firstly, I was not using 'static' in function declaration, but I led to the problem with I've read a lot about lifecycle, but I seem to be missing somethings, any ideas, please?

I hope you're aware you're reinventing the wheel [`rayon`](https://docs.rs/rayon) and doing it less well, are you? — Chayim Friedman, Mar 13 '22 at 02:15
Do you understand what the `'static` lifetime means, and why the compiler forces you to use it here? Do you understand why is this requirement not fulfilled? — Chayim Friedman, Mar 13 '22 at 02:17
@ChayimFriedman, Yes, I am aware :) As for static - doesn't it mean that the lifetime is just the whole lifetime of a program? Guess it does. I believe it forces me to use it because of borrowing it inside the cycle in `run_job` function (and probably because of spawn function). I do not understand, actually, that's why I am asking this question! — Akado2009, Mar 13 '22 at 02:25

Chayim Friedman · Accepted Answer · 2022-03-14T08:20:13.003

It looks like you wanted the following line:

let chunk = chunk.to_owned();

To create Vec<T> from the &[T] type of chunk. But it did not work. What happened here is rather bizarre.

You wanted to hit the following impl:

impl<T: Clone> ToOwned for [T]

Let's pretend we are the compiler. We look for a method named to_owned() for the type &[T]. How does this process work? You can read more about this in What are Rust's exact auto-dereferencing rules?, but here's what happens in this case:

Since we've passed &[T], we first loopkup for a method that takes &[T]. There is such one - <[T] as ToOwned>::to_owned() takes &[T], since ToOwned::to_owned() takes &self, and self is [T]. Is this method implemented? Well, yes, if T: Clone. Does this hold? No - we don't have this bound. Move on.
Look for a method that takes &&[T]. Does it exist? Yes, <&[T] as ToOwned>::to_owned(). Is it implemented? There is a blanket impl<T: Clone> ToOwned for T, so it is implemented if &[T]: Clone holds. Does it? Well, yes: every shared reference is Copy, and every Copy is also Clone (Copy has Clone as a supertrait).

So we found <&&[T] as ToOwned>::to_owned(). What does this method returns? It returns &[T] (since <T: Clone as ToOwned>::Owned == T, and in our case, T is &[T]). In conclusion, instead of creating Vec<T>, we ended up creating &[T] - or just copying our references, not fixing the issue at all!

If we'd used to_vec() instead, which is more explicit, the compiler's error would also be more helpful:

error[E0277]: the trait bound `T: Clone` is not satisfied
   --> src/lib.rs:31:35
    |
31  |                 let chunk = chunk.to_vec();
    |                                   ^^^^^^ the trait `Clone` is not implemented for `T`
    |
note: required by a bound in `slice::<impl [T]>::to_vec`
help: consider further restricting this bound
    |
27  |     pub fn run_job<T: Send + Sync + std::clone::Clone>(&self, input: &Vec<T>, f: fn(&T) -> T) -> Vec<T> {
    |                                   +++++++++++++++++++

Applying this suggestion doesn't end the story, though we advanced:

error[E0310]: the parameter type `T` may not live long enough
   --> src/lib.rs:32:25
    |
27  |     pub fn run_job<T: Send + Sync + Clone>(&self, input: &Vec<T>, f: fn(&T) -> T) -> Vec<T> {
    |                    -- help: consider adding an explicit lifetime bound...: `T: 'static +`
...
32  |                 let g = thread::spawn(move || chunk.iter().map(|x| f(x)).collect());
    |                         ^^^^^^^^^^^^^ ...so that the type `[closure@src/lib.rs:32:39: 32:83]` will meet its required lifetime bounds...
    |
note: ...that is required by this bound

error[E0310]: the parameter type `T` may not live long enough
   --> src/lib.rs:32:25
    |
27  |     pub fn run_job<T: Send + Sync + Clone>(&self, input: &Vec<T>, f: fn(&T) -> T) -> Vec<T> {
    |                    -- help: consider adding an explicit lifetime bound...: `T: 'static +`
...
32  |                 let g = thread::spawn(move || chunk.iter().map(|x| f(x)).collect());
    |                         ^^^^^^^^^^^^^ ...so that the type `Vec<T>` will meet its required lifetime bounds...
    |
note: ...that is required by this bound

This happens because T can contain a lifetime indirectly, e.g. it can be &'something_non_static i32. Adding the bound T: 'static + ... indeed fixes all errors, and it now works!

Is it the best we can do? Definitely no. We're copying our data, and we don't really need it - we just did that in order to satisfy the borrow checker!

Since we don't really need our data to be 'static and we're just forced into it because we're using threads, our best bet is to use crossbeams' scoped threads (they're also going to be part of std):

pub fn run_job<T: Send + Sync>(&self, input: &[T], f: impl Fn(&T) -> T + Sync) -> Vec<T> {
    if input.len() > self.threshold as usize {
        crossbeam::scope(|scope| {
            let mut guards = vec![];
            for chunk in input.chunks(self.threshold as usize) {
                let g = scope.spawn(|_| chunk.iter().map(|x| f(x)).collect::<Vec<_>>());
                guards.push(g);
            }
            guards.into_iter().flat_map(|g| g.join().unwrap()).collect()
        })
        .unwrap()
    } else {
        input.iter().map(|x| f(x)).collect()
    }
}

Note I also fixed some unidiomatic parts: e.g. passing &Vec is an antipattern - pass a slice instead. Also less type annotations, more iterators, impl Fn instead of fn, etc..

Scoped threads allow us to share non-'static data at the cost of our threads be forced to finish before we return (it'll join() implicitly before the scope object is dropped). Since we do that anyway, this does not matter.

But we're just reinventing the wheel. There is a library that automagically splits the work into threads, and does that way more efficiently than our implementation, using advanced work-stealing techniques - rayon! And using it is as simple as:

pub fn run_job<T: Send + Sync>(&self, input: &[T], f: impl Fn(&T) -> T + Send + Sync) -> Vec<T> {
    use rayon::prelude::*;
    input.par_iter().map(f).collect()
}

Yep, that's all it takes (note that it does not guarantee order, however)!

Thank you, appreciate your answer. One of the best I've got so far! Both of your solutions are working (not considering rayon to be a solution, since I am not able to use it due to the task). Also thank you for the explanation on what's going on actually with `chunks` and `threads`! Also yes, crossbeam seems to be the thing I was looking for with non-static -- `Scoped threads allow us to share non-'static data at the cost of our threads be forced to finish before we return`, great catch! — Akado2009, Mar 13 '22 at 13:35

score 1 · Answer 2 · answered Mar 13 '22 at 02:52

1

So, in the function you're trying to call, the pointer / reference is marked with a lifetime of static. That means, that that pointer has to be alive for the entire program. So, I don't really know how to completely fix it here, because the .chunks method requires static lifetime :( Sorry

answered Mar 13 '22 at 02:52

eboatwright

33
5

No problem, appreciate your effort! – Akado2009 Mar 13 '22 at 13:29

Borrowed value does not live long enough in testing methods

2 Answers2