The output of kde can be transformed into a raster and then from there you can extract any contour using the rasterToPolygons function. Once your points are converted to a format recognized by the sp package, you can look at any kind of intersection between the spatial objects using the gIntersection function.
You end up with two SpatialPoints objects x.inY and y.inX which contain the x points included within the 50% contour of fhaty and vice versa. The coordinates of these points may be extracted in an array using coordinates(...).
This is probably not the most elegant solution but it should work fine if the array released by the kde function is not too big.
I hope this helps.
STEP 1: convert the outputs from kde to a raster object
# for the x kde
arrayX <- expand.grid(list(fhatx$eval.points[[1]],fhatx$eval.points[[2]]))
arrayX$z <- as.vector(fhatx$estimate)
rasterX <- rasterFromXYZ(arrayX)
# for the y kde
arrayY <- expand.grid(list(fhaty$eval.points[[1]],fhaty$eval.points[[2]]))
arrayY$z <- as.vector(fhaty$estimate)
rasterY <- rasterFromXYZ(arrayY)
STEP 2: rescale the rasters between 0 and 100, then convert all cells within the 50 contour to 1. Of course the contour may be changed to 95 or another value
#for raster x
rasterX <- rasterX*100/rasterX@data@max
rasterX[rasterX[]<=50,] <- NA
rasterX[rasterX[]>50,] <- 1
#[enter image description here][1]for raster y
rasterY <- rasterY*100/rasterY@data@max
rasterY[rasterY[]<=50,] <- NA
rasterY[rasterY[]>50,] <- 1
STEP 3: extract the polygons corresponding to the 50% contour
polyX50<-rasterToPolygons(rasterX, n=16, na.rm=T, digits=4, dissolve=T)
polyY50<-rasterToPolygons(rasterY, n=16, na.rm=T, digits=4, dissolve=T)
STEP 4: convert your points to spatial objects in order to use the sp library
x.points <- SpatialPoints(x)
y.points <- SpatialPoints(y)
STEP 5: locate the points that intersect with one polygon or the other
#x points falling in fhatx 50 contour
x.inY <- gIntersection(x.points, polyY50)
#y points falling in fhatx 50 contour
y.inX <- gIntersection(y.points, polyX50)
Plot
par(mfrow=c(1,2))
plot(fhatx, cont=c(50,95), col="red")
plot(fhaty, cont=c(50,95), col="green",add=TRUE)
plot(x.points, col="red", add=T)
plot(y.points, col="green", add=T)
plot(fhatx, cont=c(50,95), col="red")
plot(fhaty, cont=c(50,95), col="green",add=TRUE)
plot(x.inY, col="red", add=T)
plot(y.inX, col="green", add=T)