The way that the system function usually works on *nix is that it calls fork and then the child calls one of the exec functions with /bin/sh -c and then the string you passed to system in the child, which turns the child process into an instance of the /bin/sh program which runs the command. The parent calls one of the wait functions, which waits for the /bin/sh to exit, which it does with the same exit status as the shell script, and then system also returns that value.
If you look at the man pages for the wait system call(s):
main 3 wait
You should get some information about what gets returns and some macro functions that help you make sense of it.
The WIFEXITED(stat_val) macro can be used to test if the program exited normally as opposed to with a signal. Normal exits involve calling the exit system call. If this function returns a non-zero value then you can use the WEXITSTATUS(stat_val) macro to get the value that it actually returned.
The WIFSIGNALED(stat_val) macro can be used to test if the program was terminated with a signal, and if so the WTERMSIG(stat_val) macro will return the signal number that caused the termination.
There are some other macros that can tell you if the process were stopped or continued, rather than terminated, but I don't think that they are overly helpful to you for this purpose, but you may want to look into them.
As far as what is actually happening in this case, it can be difficult to tell. If the fork call fails then system will be able to return -1 and set errno to reflect the error. If the fork did not fail then the error may have happened in the child and be more difficult to locate. It may be possible that on your platform system might do some tests before forking to insure that you have permission to execute the appropriate files and set errno to reflect that, but maybe not.
You should look into the perror function to print out error messages in the case that errno is set.
If the failure happens after fork and within the child then you either need to get the shell to tell you more about what is happening, or get the shell script to. This may be by including echo statements in the script similarly to using print statements in your C programs.
You should also look into the access function to test if you have permission to read and/or execute files.
If you are using Linux then you should be able to do:
strace -o my_program.strace -f ./my_program
or
ltrace -o my_program.ltrace -f -S ./my_program
and then examine the trace files (after the -o) to look at what the programs and kernel say to each other. ltrace looks at how the program talks to library function, while strace looks at system calls, but the -S tells ltrace to also look at system calls. The -f argument tells them both to trace children of the program as they are created.
I just noticed that you said that you were using ksh
As I mentioned system under a Posix system should use /bin/sh or a compatible shell. This doesn't mean that /bin/sh won't run /bin/ksh to run your script (or that the kernel won't use the #! line at the beginning of the script file to do this), but it could be a problem. There are ways to run shell scripts so that this line is not used to know which shell is to be used. The most notable is:
. myshell.sh
The period and space essentially tries to dump the text of the file into the current shell session rather than run it in another process (this is useful for setting up an environment). If you were to be doing:
int x = system(". myshell.sh");
Then that could be a problem.