Is there a more efficient way to concatenate strings?

Question

I have written an example of Go code which is sends query to postgres and send result to the pager:

package main

import (
    "fmt"
    "database/sql"
    _ "github.com/lib/pq"
    "log"
    "os/exec"
    "strings"
    "os"
)

func main() {
    connstr := "user=postgres dbname=postgres sslmode=disable"
    db, err := sql.Open("postgres", connstr)
    if err != nil { log.Fatal(err) }

    rows, err := db.Query("SELECT schemaname, relname, seq_scan FROM pg_stat_all_tables ORDER BY 1 LIMIT 10")
    if err != nil { log.Fatal(err) }
    defer rows.Close()

    var buf string
    for rows.Next() {
        var s, r string
        var ss int
        if err := rows.Scan(&s, &r, &ss); err != nil { log.Fatal(err) }
        buf = fmt.Sprintf("%s %s %d\n", buf + s, r, ss)
    }
    
    cmd := exec.Command("less")
    cmd.Stdin = strings.NewReader(buf)
    cmd.Stdout = os.Stdout
    
    err = cmd.Run()
    if err != nil { log.Fatal(err) }
}

But the following line:

buf = fmt.Sprintf("%s %s %d\n", buf + s, r, ss)

looks rude for me and I'm not sure this is a right way. Is there way to achieve result in more elegant way? May be it's possible with some kind of buffers and io.Readers?

UPD. When I asked this question a few year ago, I was a Golang newbie. Now, with current experience I see no problems with this code and the question should be closed.

What exactly is the problem with your code and in what way do you want to improve it? — Adrian, Feb 27 '18 at 20:18
If `fmt.Sprintf` results in the desired output, there is no problem! — Michael Hampton, Feb 27 '18 at 20:30
I think used way is not performance-wise and Golang must have something better than that )))) — lesovsky, Feb 28 '18 at 04:33

maksadbek · Accepted Answer · 2018-02-28T09:56:21.743

Strings in Go are immutable and every time when you assign a new value to a variable it must create a new string and copy contents of existing one to it.

You can use bytes.Buffer instead of string to avoid recreation on each iteration.

package main

import (
    "fmt"
    "database/sql"
    _ "github.com/lib/pq"
    "log"
    "os/exec"
    "strings"
    "os"
    "bytes"
)

func main() {
    connstr := "user=postgres dbname=postgres sslmode=disable"
    db, err := sql.Open("postgres", connstr)
    if err != nil { log.Fatal(err) }

    rows, err := db.Query("SELECT schemaname, relname, seq_scan FROM pg_stat_all_tables ORDER BY 1 LIMIT 10")
    if err != nil { log.Fatal(err) }
    defer rows.Close()

    var buf = new(bytes.Buffer)
    for rows.Next() {
        var s, r string
        var ss int
        if err := rows.Scan(&s, &r, &ss); err != nil { log.Fatal(err) }
        buf.WriteString(fmt.Sprintf("%s %s %d\n", s, r, ss))
    }

    cmd := exec.Command("less")
    cmd.Stdin = buf
    cmd.Stdout = os.Stdout

    err = cmd.Run()
    if err != nil { log.Fatal(err) }
}

Btw, the string builder was added in Go 1.10 https://godoc.org/strings#Builder

Read more about string concatenation benchmarks: http://herman.asia/efficient-string-concatenation-in-go

Interesting how fast is string builder work relatively to other concatenations ways? Anyway bytes.Buffer and strings.Builder look preferred. — lesovsky, Feb 28 '18 at 04:31
"cmd.Stdin = buf" is wrong, because of cannot use buf (type bytes.Buffer) as type io.Reader in assignment: bytes.Buffer does not implement io.Reader (Read method has pointer receiver) — lesovsky, Feb 28 '18 at 06:48
everything ok with `cmd.Stdin = strings.NewReader(buf.String())` and strings.Builder — lesovsky, Feb 28 '18 at 07:06

jrefior · Answer 2 · 2018-02-27T21:14:19.493

The only problem with what you're doing now is that the concatenation operator + is an extremely inefficient way to combine a lot of strings into one bigger string.

How inefficient? Well here's a benchmark I did testing three approaches:

BenchmarkMultiplyBasic                    300000              4240 ns/op
BenchmarkMultiplyJoinBasic                200000              9942 ns/op
BenchmarkMultiplyConcatenationsBasic       10000            170523 ns/op

The last one is the concatenate operator + and shows truly miserable performance compared to a couple alternatives.

Here's one approach that will be more efficient, in a simplified runnable example:

package main

import(
    "fmt"
    "strconv"
    "strings"
)

type result struct {
    s, r        string
    ss          int
}

func main() {
    a := []result{
        {"twas", "brillig", 1},
        {"and", "the", 2},
        {"slithy", "toves", 3},
    }
    outstrings := make([]string, 0)
    for _, part := range a {
        outstrings = append(outstrings, part.s, part.r, strconv.Itoa(part.ss))
    }
    out := strings.Join(outstrings, ` `)
    fmt.Printf("%s\n", out)
}

prints

twas brillig 1 and the 2 slithy toves 3

How to most efficiently combine strings is a common question on StackOverflow and has been answered many times. See this top-voted question/answer for Go: How to efficiently concatenate strings in Go?.

yes, that is why I wrote here... concatenation with "+" looks suspicious, and due to I'm newbie I simply don't know language details and thought Golang should have more beautiful ways to concatenate strings. — lesovsky, Feb 28 '18 at 04:27

Is there a more efficient way to concatenate strings?

2 Answers2