Join iterator of &str

Question

How do I convert an Iterator<&str> to a String, interspersed with a constant string such as "\n"? For instance, given:

let xs = vec!["first", "second", "third"];
let it = xs.iter();

One may produce a string s by collecting into a Vec<&str> and joining the result:

let s = it
    .map(|&x| x)
    .collect::<Vec<&str>>()
    .join("\n");

However, this unnecessarily allocates memory for a Vec<&str>.

Is there a more direct method?

Apologies - my original answer removed the iterator but your question is asking how to join an iterator and not allocate the extra vector. — Simon Whitehead, May 08 '19 at 04:41
Looks like the `itertools` crate [doesn't allocate the vector](https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=e208a26d869de89f6b998e6bcf6b5a77) — Simon Whitehead, May 08 '19 at 05:09
Note that depending on the exact characteristics of your iterator, collecting into a vector of slices and then joining could actually be faster than using Websterix's method or `itertools`, since `SliceConcatExt::join` can calculate the needed size for the full string ahead of time and thus definitely doesn't need to reallocate during accumulation; whereas the other methods may have to reallocate the string. You should definitely benchmark. — Sebastian Redl, May 08 '19 at 06:04
@SebastianRedl But `collect::>` would need to reallocate, but it's a lot smaller than the string buffer, so i guess that would be a faster? — chpio, May 08 '19 at 09:37
@chpio It has to allocate, but not reallocate if the iterator gives a good size hint. — Sebastian Redl, May 08 '19 at 12:45
[The `once` plus `skip` trick works nicely for this](https://play.rust-lang.org/?version=stable&mode=debug&edition=2021&gist=b5eedf3f4537a263435b35942f7a5334). See also [this answer](https://stackoverflow.com/a/66951473/4039050). — schneiderfelipe, Jan 14 '22 at 15:17
Sorry, but I dont understand the `.map(|&x| x)` part... why is that? — debuti, Jul 18 '22 at 14:29
I also agree with @MattJoiner, this is not a duplicate and there are better answers to this question in 2023. I voted to reopen. — Ivan Gabriele, Apr 29 '23 at 23:16

chpio · Answer 1 · 2019-05-08T09:21:09.110

You could use the itertools crate for that. I use the intersperse helper in the example, it is pretty much the join equivalent for iterators.

cloned() is needed to convert &&str items to &str items, it is not doing any allocations. It can be eventually replaced by copied() when rust@1.36 gets a stable release.

use itertools::Itertools; // 0.8.0

fn main() {
    let words = ["alpha", "beta", "gamma"];
    let merged: String = words.iter().cloned().intersperse(", ").collect();
    assert_eq!(merged, "alpha, beta, gamma");
}

Playground

Akiner Alkan · Answer 2 · 2019-05-08T08:44:53.927

22

You can do it by using fold function of the iterator easily:

let s = it.fold(String::new(), |a, b| a + b + "\n");

The Full Code will be like following:

fn main() {
    let xs = vec!["first", "second", "third"];
    let it = xs.into_iter();

    // let s = it.collect::<Vec<&str>>().join("\n");

    let s = it.fold(String::new(), |a, b| a + b + "\n");
    let s = s.trim_end();

    println!("{:?}", s);
}

Playground

EDIT: After the comment of Sebastian Redl I have checked the performance cost of the fold usage and created a benchmark test on playground.

You can see that fold usage takes significantly more time for the many iterative approaches.

Did not check the allocated memory usage though.

edited May 08 '19 at 08:44

answered May 08 '19 at 05:15

Akiner Alkan

6,145
3
32
68

17

The reason this is slow is because you're using + to create two new Strings on every iteration. If you use a single string ([playground](https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=f2740a132f69b5d41d14a3472c98674e)) it can work better than collect and join ([playground](https://play.rust-lang.org/?version=stable&mode=release&edition=2018&gist=5fa149bbd78707d7ff4e257d38318361)). – mdonoughe Dec 21 '20 at 07:15
added `black_box` and create the vec for each test individually (because of cache warming) ([playgroud](https://play.rust-lang.org/?version=nightly&mode=release&edition=2018&gist=9316d8202ee375f35305c8ec938f5f47)). Playground isn't that good for benchmarking due to massive variance in latency/duration, but the fold variant seems to be slightly slower (over multiple runs). – chpio Apr 19 '21 at 12:23
1

[v2](https://play.rust-lang.org/?version=nightly&mode=release&edition=2018&gist=6d7a6e7427a715d853ce32440a0ad59e) with `black_box(xs).iter().copied()` takes now twice as long for collect+join over fold (the `black_box(xs)` doesn't matter, `xs` is the same). <3 for microbenchmarking – chpio Apr 19 '21 at 12:54
I guess they optimize something like `vec!["hey"; 100_000].into_iter().collect>` to just return the original vec?! – chpio Apr 19 '21 at 13:02
1

Yes, [they do](https://doc.rust-lang.org/stable/src/alloc/vec/spec_from_iter.rs.html#41) – chpio Apr 19 '21 at 13:14
@mcdonoughe could this be improved with `String::with_capacity(xs.len())`? – Sasha Kondrashov Jul 29 '22 at 00:18
1

Another solution is `let mut it = xs.into_iter(); let first = it.next().unwrap_or("").to_owned(); let r = it.fold(first, |a, b| a + "\n" + b);` Then you end up with a `String` instead of `&str` – d2weber Mar 16 '23 at 20:43

score 6 · Answer 3 · answered May 10 '23 at 23:01

6

With itertools, you have not only intersperse() but also join():

use itertools::Itertools;

let s = it.join("\n");

It is more general than intersperse() (it accepts any Display-implementing type) but therefore may be slower (I didn't benchmark though).

answered May 10 '23 at 23:01

Chayim Friedman

47,971
5
48
77

score -3 · Answer 4 · answered May 08 '19 at 05:15

-3

there's relevant example in rust documentation: here.

let words = ["alpha", "beta", "gamma"];

// chars() returns an iterator
let merged: String = words.iter()
                          .flat_map(|s| s.chars())
                          .collect();
assert_eq!(merged, "alphabetagamma");

You can also use Extend trait:

fn f<'a, I: Iterator<Item=&'a str>>(data: I) -> String {
    let mut ret = String::new();
    ret.extend(data);
    ret
}

answered May 08 '19 at 05:15

Laney

1,571
9
7

6

This answer does not reproducing the OP's needs. OP is asking about interspersed with some constant string (e.g. "\n")?. – Akiner Alkan May 08 '19 at 05:29
1

also this should work without `flat_map`, as *String* already implements `Extend<&str>`. – chpio May 08 '19 at 09:16

Join iterator of &str

4 Answers4

Linked