I think you could be encountering an issue when the command string is parsed.
/system/bin/sh chmod 0777 /data/playback.bin is treated as "/system/bin/sh" "chmod" "0777" "/data/playback.bin/" with the extra quotes. Not sure if that is the reason for failure, but try the following instead:
Process process = Runtime.getRuntime().exec("su");
DataOutputStream suStream = new DataOutputStream(process.getOutputStream());
suStream.writeBytes("/system/bin/sh chmod 0777 /data/playback.bin" + "\n");
suStream.writeBytes("exit\n");
suStream.flush();
suStream.close();
process.waitFor();
Note that you should get the app root permission popup after exec("su"), so if you don't then it never gets to the point of even attempting to execute your file permission change command.
EDIT - You could also try this variation of the above (based on this answer):
Process shell = Runtime.getRuntime().exec("su", null, new File("/system/bin/"));
OutputStream os = shell.getOutputStream();
os.write(("chmod 777 /data/playback.bin").getBytes("ASCII"));
os.flush();
os.close();
shell.waitFor();