RealityKit's `.allowsCameraControl` tool for building `nonAR` apps

Question

RealityKit has an option for .nonAR camera, but lacks fundamental options to build 3D nonAR apps, like camera control (.allowsCameraComtrol).

Is it viable to use RealityKit a replacement for SceneKit (which hasn't been updated for the past years and will likely be desecrated soon)?

Answer 1

.allowsCameraControl equivalent

Apple has been preparing a replacement for SceneKit, and it's definitely a Reality family – RealityKit, RealityFoundation, Reality Composer (iOS/macOS), Reality Composer Pro (rather for visionOS), Reality Converter, ARQuickLook, etc. Using RealityKit 2.0, you can create both AR and VR apps. As a tool for prototyping 3D scenes, you need to use Reality Composer. And even despite the lack of such an important feature as .allowsCameraControl, you are now able to make VR apps using RealityKit.

Workaround

Nonetheless, use my code as a starting point to create your own camera control for VR in RealityKit:

import UIKit
import RealityKit

class ViewController: UIViewController {
    @IBOutlet var arView: ARView!
    let buildings = try! Experience.loadScene()
    let camera = PerspectiveCamera()
    var current_X_Angle: Float = 0.0
    var current_Y_Angle: Float = 0.0

    override func viewDidLoad() {
        super.viewDidLoad()
        arView.environment.background = .color(.systemCyan)
        arView.scene.anchors.append(buildings)
        self.camera.position.z = 1
        buildings.children[0].addChild(self.camera)
        self.gestureRecognizer()
    }
    func gestureRecognizer() {
        for gestureRecognizer in [UIPanGestureRecognizer.self,
                                  UIPinchGestureRecognizer.self] {
            if gestureRecognizer == UIPinchGestureRecognizer.self {
                let r = UIPinchGestureRecognizer(target: self,
                               action: #selector(allowCameraControl_01))
                arView.addGestureRecognizer(r)
            }
            if gestureRecognizer == UIPanGestureRecognizer.self {
                let r = UIPanGestureRecognizer(target: self,
                               action: #selector(allowCameraControl_02))
                arView.addGestureRecognizer(r)
            }
        }
    }
    @objc func allowCameraControl_01(recognizer: UIPinchGestureRecognizer) {
        switch recognizer.state {
            case .changed, .ended:
                self.camera.position.z *= 1 / Float(recognizer.scale)
                recognizer.scale = 1.0
            default: break
        }
    }
    @objc func allowCameraControl_02(recognizer: UIPanGestureRecognizer) {
        switch recognizer.state {
            case .changed, .ended:
                let translate = recognizer.translation(in: recognizer.view)
                let angle_X = Float(translate.y / 300) * .pi / 180.0
                let angle_Y = Float(translate.x / 100) * .pi / 180.0
                self.current_X_Angle += angle_X
                self.current_Y_Angle += angle_Y                
                camera.setOrientation(Transform(pitch: current_X_Angle,
                                                  yaw: current_Y_Angle,
                                                 roll: .zero).rotation,
                                      relativeTo: buildings.anchor)
            default: break
        }
    }
}

---

Here you can find SwiftUI version.

visionOS Xcode Simulator

In Xcode 15's visionOS simulator, there are Pan, Orbit and Dolly UI buttons, available in lower right corner.