Aparapi GPU execution slower than CPU

Question

I am trying to test the performance of Aparapi. I have seen some blogs where the results show that Aparapi does improve the performance while doing data parallel operations.

But I am not able to see that in my tests. Here is what I did, I wrote two programs, one using Aparapi, the other one using normal loops.

Program 1: In Aparapi

import com.amd.aparapi.Kernel;
import com.amd.aparapi.Range;

public class App 
{
    public static void main( String[] args )
    {
        final int size = 50000000;

        final float[] a = new float[size];
        final float[] b = new float[size];

        for (int i = 0; i < size; i++) {
           a[i] = (float) (Math.random() * 100);
           b[i] = (float) (Math.random() * 100);
        }

        final float[] sum = new float[size];

        Kernel kernel = new Kernel(){
           @Override public void run() {
              int gid = getGlobalId();
              sum[gid] = a[gid] + b[gid];
           }
        };
        long t1 = System.currentTimeMillis();
        kernel.execute(Range.create(size));
        long t2 = System.currentTimeMillis();
        System.out.println("Execution mode = "+kernel.getExecutionMode());
        kernel.dispose();
        System.out.println(t2-t1);
    }
}

Program 2: using loops

public class App2 {

    public static void main(String[] args) {

        final int size = 50000000;

        final float[] a = new float[size];
        final float[] b = new float[size];

        for (int i = 0; i < size; i++) {
           a[i] = (float) (Math.random() * 100);
           b[i] = (float) (Math.random() * 100);
        }

        final float[] sum = new float[size];
        long t1 = System.currentTimeMillis();
        for(int i=0;i<size;i++) {
            sum[i]=a[i]+b[i];
        }

        long t2 = System.currentTimeMillis();
        System.out.println(t2-t1);

    }
}

Program 1 takes around 330ms whereas Program 2 takes only around 55ms. Am I doing something wrong here? I did printout the execution mode in Aparpai program and it prints that the mode of execution is GPU

Answer 1

You did not do anything wrong - execpt for the benchmark itself.

Benchmarking is always tricky, and particularly for the cases where a JIT is involved (as for Java), and for libraries where many nitty-gritty details are hidden from the user (as for Aparapi). And in both cases, you should at least execute the code section that you want to benchmark multiple times.

For the Java version, one might expect the computation time for a single execution of the loop to decrease when the loop itself it is executed multiple times, due to the JIT kicking in. There are many additional caveats to consider - for details, you should refer to this answer. In this simple test, the effect of the JIT may not really be noticable, but in more realistic or complex scenarios, this will make a difference. Anyhow: When repeating the loop for 10 times, the time for a single execution of the loop on my machine was about 70 milliseconds.

For the Aparapi version, the point of possible GPU initialization was already mentioned in the comments. And here, this is indeed the main problem: When running the kernel 10 times, the timings on my machine are

1248
72
72
72
73
71
72
73
72
72

You see that the initial call causes all the overhead. The reason for this is that, during the first call to Kernel#execute(), it has to do all the initializations (basically converting the bytecode to OpenCL, compile the OpenCL code etc.). This is also mentioned in the documentation of the KernelRunner class:

The KernelRunner is created lazily as a result of calling Kernel.execute().

The effect of this - namely, a comparatively large delay for the first execution - has lead to this question on the Aparapi mailing list: A way to eagerly create KernelRunners. The only workaround suggested there was to create an "initialization call" like

kernel.execute(Range.create(1));

without a real workload, only to trigger the whole setup, so that the subsequent calls are fast. (This also works for your example).

---

You may have noticed that, even after the initialization, the Aparapi version is still not faster than the plain Java version. The reason for that is that the task of a simple vector addition like this is memory bound - for details, you may refer to this answer, which explains this term and some issues with GPU programming in general.

As an overly suggestive example for a case where you might benefit from the GPU, you might want to modify your test, in order to create an artificial compute bound task: When you change the kernel to involve some expensive trigonometric functions, like this

Kernel kernel = new Kernel() {
    @Override
    public void run() {
        int gid = getGlobalId();
        sum[gid] = (float)(Math.cos(Math.sin(a[gid])) + Math.sin(Math.cos(b[gid])));
    }
};

and the plain Java loop version accordingly, like this

for (int i = 0; i < size; i++) {
    sum[i] = (float)(Math.cos(Math.sin(a[i])) + Math.sin(Math.cos(b[i])));;
}

then you will see a difference. On my machine (GeForce 970 GPU vs. AMD K10 CPU) the timings are about 140 milliseconds for the Aparapi version, and a whopping 12000 milliseconds for the plain Java version - that's a speedup of nearly 90 through Aparapi!

Also note that even in CPU mode, Aparapi may offer an advantage compared to plain Java. On my machine, in CPU mode, Aparapi needs only 2300 milliseconds, because it still parallelizes the execution using a Java thread pool.

Answer 2

Just add before main loop kernel execution

kernel.setExplicit(true);
kernel.put(a);
kernel.put(b);

and

kernel.get(sum);

after it.

Although Aparapi does analyze the byte code of the Kernel.run() method (and any method reachable from Kernel.run()) Aparapi has no visibility to the call site. In the above code there is no way for Aparapi to detect that that hugeArray is not modified within the for loop body. Unfortunately, Aparapi must default to being ‘safe’ and copy the contents of hugeArray backwards and forwards to the GPU device.

https://github.com/aparapi/aparapi/blob/master/doc/ExplicitBufferHandling.md