You do not need to be a system app. First, create package com.android.internal.telephony in your project, and put this in a file called "ITelephony.aidl":
package com.android.internal.telephony; 
interface ITelephony {      
boolean endCall();     
void answerRingingCall();      
void silenceRinger(); 
}
Once you have that, you can use this code to end a call:
TelephonyManager telephonyManager = (TelephonyManager)context.getSystemService(Context.TELEPHONY_SERVICE);
Class clazz = Class.forName(telephonyManager.getClass().getName());
Method method = clazz.getDeclaredMethod("getITelephony");
method.setAccessible(true);
ITelephony telephonyService = (ITelephony) method.invoke(telephonyManager);
telephonyService.endCall();
You could use this inside a PhoneStateListener, for example. For this to work, you require permissions in manifest:
<uses-permission android:name="android.permission.MODIFY_PHONE_STATE" />
<uses-permission android:name="android.permission.CALL_PHONE" />
<uses-permission android:name="android.permission.READ_PHONE_STATE" />
Edit: Apologies for horrible formatting, I still can't figure out how to properly do code blocks here :/