SceneKit matrix transformation to match camera angle

Question

I'm building a UIPanGestureRecognizer so I can move nodes in 3D space.

Currently, I have something that works, but only when the camera is exactly perpendicular to the plane, my UIPanGestureRecognizer looks like this:

@objc func handlePan(_ sender:UIPanGestureRecognizer) {
  let projectedOrigin = self.sceneView!.projectPoint(SCNVector3Zero)

  let viewCenter = CGPoint(
    x: self.view!.bounds.midX,
    y: self.view!.bounds.midY
  )

  let touchlocation = sender.translation(in: self.view!)

  let moveLoc = CGPoint(
    x: CGFloat(touchlocation.x + viewCenter.x),
    y: CGFloat(touchlocation.y + viewCenter.y)
  )

  let touchVector = SCNVector3(x: Float(moveLoc.x), y: Float(moveLoc.y), z: Float(projectedOrigin.z))
  let worldPoint = self.sceneView!.unprojectPoint(touchVector)
  let loc = SCNVector3( x: worldPoint.x, y: 0, z: worldPoint.z )

  worldHandle?.position = loc
}

The problem happens when the camera is rotated, and the coordinates are effected by the perspective change. Here is you can see the touch position drifting:

Related SO post for which I used to get to this position: How to use iOS (Swift) SceneKit SCNSceneRenderer unprojectPoint properly

It referenced these great slides: http://www.terathon.com/gdc07_lengyel.pdf

If the grid is a plane (else you can add a plane and make it invisible) you can simply do a hittest on the plane node to get the coordinates on the plane. Works for the tap gesture but also very accurately for the pan gesture (e.g. hittest when the pan starts and hittest again everytime the gesture state changed is active). This will exclude the camera factor as well as the orientation of the plane. You know the number of tiles and their width/height so you can simply divide the coordinates by the width of a tile to get the column, similar for a row. — Xartec, Jan 27 '18 at 16:29
That works however the coordinates of the SCNVector3 on the plane are affected by the perspective distortion and the hits are detected in the wrong spot, works perfectly when the plane is perpendicular however. — JP Silvashy, Jan 27 '18 at 16:31
No, the localCoordinates (of the hitresult) on the plane will always be consistent regardless of camera settings. You can then convert the resulting position to worldspace to get the location for nodes you want to place on that position. — Xartec, Jan 27 '18 at 16:33
Maybe my issue is with translating the position, I've edited my question with a screen recording of the touch position drifting. — JP Silvashy, Jan 28 '18 at 04:11
https://developer.apple.com/documentation/scenekit/scnscenerenderer/1522929-hittest will give you more accurate reults — Xartec, Jan 28 '18 at 13:59
You can use the hitresults from the hittest to get the local coordinates of the tap on the plane directly, but also to get a proper z value for unprojecting. — Xartec, Jan 28 '18 at 14:05
I understand how the hittest works, and when I tap on the plan or items, they are correct, however view the attached gif, you can see that when the pan gesture happens the perspective affects the position of the pan gesture. — JP Silvashy, Jan 28 '18 at 14:24
Sorry, I made a mistake, I've added the code for the pan gesture recognizer! — JP Silvashy, Jan 28 '18 at 16:52
Thanks, that clears things up a bit :) As Rickster mention in the post you referenced to that approach only works if the plane is perpendicular to the camera. That is why I suggest using a hittest as it takes the camera and plane orientation out of the equation. I will post an answer with sample code shortly. — Xartec, Jan 28 '18 at 17:32
Hi @Xartec I am wondering whether its possible to do the reverse from 3D to 2D in the face tracking with Scenekit/ARkit? — swiftlearneer, Jul 23 '20 at 00:07

Xartec · Accepted Answer · 2018-01-28T18:31:07.607

The tricky part of going from 2D touch position to 3D space is obviously the z-coordinate. Instead of trying to convert the touch position to an imaginary 3D space, map the 2D touch to a 2D plane in that 3D space using a hittest. Especially when movement is required only in two direction, for example like chess pieces on a board, this approach works very well. Regardless of the orientation of the plane and the camera settings (as long as the camera doesn't look at the plane from the side obviously) this will map the touch position to a 3D position directly under the finger of the touch and follow consistently.

I modified the Game template from Xcode with an example. https://github.com/Xartec/PrecisePan/

The main parts are:

the pan gesture code:

// retrieve the SCNView
    let scnView = self.view as! SCNView
    // check what nodes are tapped
    let p = gestureRecognize.location(in: scnView)
    let hitResults = scnView.hitTest(p, options: [SCNHitTestOption.searchMode: 1, SCNHitTestOption.ignoreHiddenNodes: false])

    if hitResults.count > 0 {
        // check if the XZPlane is in the hitresults
        for result in hitResults {
            if result.node.name == "XZPlane" {
                //NSLog("Local Coordinates on XZPlane %f, %f, %f", result.localCoordinates.x, result.localCoordinates.y, result.localCoordinates.z)

                //NSLog("World Coordinates on XZPlane %f, %f, %f", result.worldCoordinates.x, result.worldCoordinates.y, result.worldCoordinates.z)
                ship.position = result.worldCoordinates
                ship.position.y += 1.5
                return;
            }
        }
    }

The addition of a XZ plane node in viewDidload:

let XZPlaneGeo = SCNPlane(width: 100, height: 100)
let XZPlaneNode = SCNNode(geometry: XZPlaneGeo)
XZPlaneNode.geometry?.firstMaterial?.diffuse.contents = UIImage(named: "grid")
XZPlaneNode.name = "XZPlane"
XZPlaneNode.rotation = SCNVector4(-1, 0, 0, Float.pi / 2)
//XZPlaneNode.isHidden = true
scene.rootNode.addChildNode(XZPlaneNode)

Uncomment the isHidden line to hide the helper plane and it will still work. The plane obviously needs to be large enough to fill the screen or at least the portion where the user is allowed to pan.

By setting a global var to hold a startWorldPosition of the pan (in state .began) and comparing it to the hit worldPosition in the state .change you can determine the delta/translation in world space and translate other objects accordingly.

I have the similar question , I need to pan the Point of View , But when Camera is rotated then I am not able to pan wherever I want (sometimes it is working on X direction some time in Y direction based on the rotation ) Here What I have tried https://stackoverflow.com/questions/66437552/sceneview-translate-move-point-of-view-camera-with-gesture/66469719#66469719 — Prashant Tukadiya, May 07 '21 at 15:57

SceneKit matrix transformation to match camera angle

1 Answers1