You can do this by computing the angles.
Suppose your endpoints are (x1,y1) and (x2,y2), and your point is (x,y).
Then you create two vectors, from one endpoint to other, and one endpoint to your point:
vec1 = (x - x1, y - y1);
vec2 = (x2 - x1, y2 - y1);
Compute the dot product:
double dotp = (x-x1) * (x2-x1) + (y-y1) * (y2 - y1);
Now the dot product divided by magnitude gives you the cosine of the angle:
double theta = Math.acos((dtop) / (Math.sqrt((x-x1) * (x-x1) + (y-y1) * (y-y1))
* Math.sqrt((x2-x1) * (x2-x1) + (y2-y1) * (y2-y1))));
Now the trick is that if your angle is greater than PI / 2, you are 'out'
public static boolean check(double x, double y, double x1, double y1,
double x2, double y2) {
// vectors are (dx1, dy1) and (dx2, dy2)
double dx1 = x - x1, dx2 = x2 - x1, dy1 = y - y1, dy2 = y2 - y1;
double dotp = dx1 * dx2 + dy1 * dy2;
double theta = Math.acos(dotp / (Math.sqrt(dx1 * dx1 + dy1 * dy1)
* Math.sqrt(dx2 * dx2 + dy2 * dy2)));
theta = Math.abs(theta);
if (theta > (Math.PI / 2))
return false;
dx1 = x - x2;
dx2 = x1 - x2;
dy1 = y - y2;
dy2 = y1 - y2;
dotp = dx1 * dx2 + dy1 * dy2;
theta = Math.acos(dotp / (Math.sqrt(dx1 * dx1 + dy1 * dy1)
* Math.sqrt(dx2 * dx2 + dy2 * dy2)));
theta = Math.abs(theta);
if (theta > (Math.PI / 2))
return false;
return true;
}