0

Consider following MATLAB code snippet

indices = cell(n,1);
values = rand(m,3);
% indices{...} contains a vector of integers 1,..,m

parfor i = 1:n
  currentIndices = indices{i};
  currentValues = values(currentIndices,:); % each worker needs the complete values-array
end

The indices-array says which rows of the values-array are relevant inside the loop. However, each worker gets the values array, so MATLAB gives the warning of broadcasting it to each worker. Is there a way to circumvent this?

Thank you :)

Kapa
  • 21
  • 1
  • 2
    You are saying: I need each worker to have all the data, yet MATLAB tells me that all the workers need all the data. Yes? What is the question? Circumvent what? if you want to circumvent what you want to do, then you need to simply need to "want to do" something else, I guess. – Ander Biguri May 04 '20 at 10:00
  • In any case, if you do not know why the warning is there: copying memory is the biggest bottleneck to parallel computing. MATLAB is saying "you are copying to all nodes, are you sure you want to do this? because it will be very slow and maybe there is no point of doing parallel processing if you do this". Its a fair warning I'd say. – Ander Biguri May 04 '20 at 10:03
  • If you are able to easily calculate the indices you want for each loop iteration as a function of `i`, such as `currentIndicies = [i, 2*i, 3*i]`, then you can shift this calculation into the loop to avoid broadcasting the data to each worker. But if these indices are dependent on the data, then each worker _needs_ them so there's not much you can do. – Dominic D May 04 '20 at 10:14
  • [This question of mine](https://stackoverflow.com/q/32095552/5211833) and answers thereon might be relevant to you. – Adriaan May 04 '20 at 10:23

0 Answers0