How to find a sibling file when the os.Getwd() is different in different environments

Question

myprogram/
 |
 |-main.go
 |-dir1/
     |-data/
         |-datafile.json
     |-runner.go
     |-runner_test.go

In runner.go, I have a simple function that reads the datafile.json. Something like

   func GetPayload() (string, err) {
            dBytes, dErr := ioutil.ReadFile("dir1/data/datafile.json")
            if dErr != nil { return nil, dErr}
            return dBytes, nil
   }

I'm using Go in a Lambda with a structure similar to above. When the Lambda runs in its actual environment, it starts at main.go, and then invokes GetPayload() from runner.go. However, I have a test in a simple worker node machine in runner_test.go that also hits GetPayload() .

During "normal" execution (from main.go) - this works OK. However, when GetPayload() is invoked from runner_test.go, it errors, saying

open dir1/data/datafile.json no such file or directory

This makes sense, because during the test, the working directory is the directory that houses runner_test.go, which is data/, so there is no dir1 as a child of it. I've been trying to play with using os.Getwd() and getting the paths from there like:

   pwd, _ := os.Getwd()
   dBytes, dErr := ioutil.ReadFile(pwd + "dir1/data/datafile.json")

But again, that won't work, because for runner_test.go pwd is user/myname/myprogram/dir1, but from main.go, it turns up as user/myname/myprogram.

Any idea how I can find and open datafile.json from within GetPayload() in any environment? I could pass an optional parameter to GetPayload() but if possible, it'd be great to avoid that.

@HymnsForDisco , yes - this data file is static, part of the directory. Can you elaborate on how to embed it? — NateH06, Jun 18 '21 at 20:23

Hymns For Disco · Accepted Answer · 2021-06-21T17:13:25.457

3

If the file is static (meaning that it doesn't need to change after you build the program), you can embed it into the built program. This means you no longer have to worry about run-time file paths.

import (
    "embed"
)

//go:embed data/*
var dataFiles embed.FS

func GetPayload() (string, err) {
    dBytes, dErr := dataFiles.ReadFile(dataFiles, "data/datafile.json")
    if dErr != nil { return nil, dErr}
    return dBytes, nil
}

Now the files in your data/ directory are embedded in this variable dataFiles which acts as a read-only file system.

For more info:

Read more about embed in the package documentation overview.

Read my answer about "when to use embed"

edited Jun 21 '21 at 17:13

answered Jun 18 '21 at 20:38

Hymns For Disco

7,530
2
17
33

Looks like this is for go 1.16 though, and no lower? – NateH06 Jun 18 '21 at 23:48
@NateH06 That's correct. It's a new part of the build process, so you need the updated tooling in the new release to use it (plus the embed library in 1.16). – Hymns For Disco Jun 19 '21 at 00:15

score 0 · Answer 2 · answered Jun 18 '21 at 20:13

For data files that your program needs during runtime, either use a fixed directory and refer to that, or accept a command line argument or some sort of configuration that tells you where the file is.

When running unit tests, the wd is the directory containing the test file. One convention is to use a testdata/ directory under the directory containing the test, and put all data files there. That way you can refer to that file from the test by using testdata/datafile.json.

You can use a copy of the file you need during runtime as your test file, or you can use a symlink from the runtime data file to the test file under the testdata/ dir.

score 0 · Answer 3 · answered Jun 18 '21 at 20:22

For data files that your program needs during runtime, either use a fixed directory and refer to that

Someone made this suggestion, which I agree with. To that end, you can use something like this:

package main

import (
   "os"
   "path/filepath"
)

func main() {
   d, err := os.UserCacheDir()
   if err != nil {
      panic(err)
   }
   d = filepath.Join(d, "file.json")
   f, err := os.Open(d)
   if err != nil {
      panic(err)
   }
   defer f.Close()
   os.Stdout.ReadFrom(f)
}

How to find a sibling file when the os.Getwd() is different in different environments

3 Answers3