How to get source directory on secondary workers in Julia?

Question

I find using Base.source_dir() super convenient for includeing files into a Julia script because it is easy to load everything using relative paths from the script file. This doesn't work properly with parallelized code. For example, if we have a test file in /Users/username/test/ called test.jl, with the following code:

@everywhere println(pwd())
@everywhere println(Base.source_dir())

and we run julia -p 1 test/test.jl we get

/Users/username
    From worker 2:  /Users/username
/Users/username/test
    From worker 2:  nothing

what would be the most robust way to include another file that is also in the test directory?

score 2 · Accepted Answer · edited May 23 '17 at 12:14

2

I usually use something like this:

function sendto(p::Int; args...)
    for (nm, val) in args
        @spawnat(p, eval(Main, Expr(:(=), nm, val)))
    end
end    

SourceDir = Base.source_dir()    

for (idx, pid) in enumerate(workers())
    sendto(pid, SourceDir = SourceDir)
end    

@everywhere include(string(SourceDir, "/", "FileName"))

See here (Julia: How to copy data to another processor in Julia) for the source of the sendto() function as well as a lot of other useful tools for moving data between workers.

edited May 23 '17 at 12:14

Community

1
1

answered Jun 13 '16 at 21:13

Michael Ohlrogge

10,559
5
48
76

Agreed. I went ahead and changed the title since your solution should work in many situations and it will help me find it when I'll be googling for it in a couple weeks haha. – tlnagy Jun 13 '16 at 21:46
See also this post (http://stackoverflow.com/questions/27677399/julia-how-to-copy-data-to-another-processor-in-julia) which is where I got the function. It has a similar `getfrom()` function as well. I'd actually recommend changing your title back. Your new one might make the question basically a duplicate of the other one. – Michael Ohlrogge Jun 13 '16 at 21:50
I actually like the `@eval @everywhere` solution a lot in that post – tlnagy Jun 13 '16 at 21:55

score 2 · Answer 2 · answered Jun 13 '16 at 21:59

A less generalized, but arguably more elegant solution for this specific case is to the following:

source_dir = Base.source_dir()    
@eval @everywhere include(joinpath($source_dir, "test2.jl"))

which was adapted from Julia: How to copy data to another processor in Julia as pointed out by @aireties. I'll leave both as solutions.

How to get source directory on secondary workers in Julia?

2 Answers2