Building a Portal App in ARKit: Adding Objects

This is an excerpt taken from Chapter 8, “Adding Objects to Your World”, of our book ARKit by Tutorials. This book show you how to build five immersive, great-looking AR apps in ARKit, Apple’s augmented reality framework. Enjoy!

In the previous tutorial of this series, you learned how to set up your iOS app to use ARKit sessions and detect horizontal planes. In this part, you’re going to build up your app and add 3D virtual content to the camera scene via SceneKit. By the end of this tutorial, you’ll know how to:

• Handle session interruptions
• Place objects on a detected horizontal plane

Before jumping in, download the project materials using the “Download Materials” button and load the starter project from the starter folder.

Getting Started

Now that you are able to detect and render horizontal planes, you need to reset the state of the session if there are any interruptions. ARSession is interrupted when the app moves into the background or when multiple applications are in the foreground. Once interrupted, the video capture will fail and the ARSession will be unable to do any tracking as it will no longer receive the required sensor data. When the app returns to the foreground, the rendered plane will still be present in the view. However, if your device has changed its position or rotation, the ARSession tracking will not work anymore. This is when you need to restart the session.

The ARSCNViewDelegate implements the ARSessionObserver protocol. This protocol contains the methods that are called when the ARSession detects interruptions or session errors.

Open PortalViewController.swift and add the following implementation for the delegate methods to the existing extension.

// 1
func session(_ session: ARSession, didFailWithError error: Error) {
  // 2
  guard let label = self.sessionStateLabel else { return }
  showMessage(error.localizedDescription, label: label, seconds: 3)
}

// 3
func sessionWasInterrupted(_ session: ARSession) {
  guard let label = self.sessionStateLabel else { return }
  showMessage("Session interrupted", label: label, seconds: 3)
}

// 4
func sessionInterruptionEnded(_ session: ARSession) {
  // 5
  guard let label = self.sessionStateLabel else { return }
  showMessage("Session resumed", label: label, seconds: 3)

  // 6
  DispatchQueue.main.async {
    self.removeAllNodes()
    self.resetLabels()
  }
  // 7
  runSession()
}

Let’s go over this step-by-step.

1. session(_:, didFailWithError:) is called when the session fails. On failure, the session is paused and it does not receive sensor data.
2. Here you set the sessionStateLabel text to the error message that was reported as a result of the session failure. showMessage(_:, label:, seconds:) shows the message in the specified label for the given number of seconds.
3. The sessionWasInterrupted(_:) method is called when the video capture is interrupted as a result of the app moving to the background. No additional frame updates are delivered until the interruption ends. Here you display a “Session interrupted” message in the label for 3 seconds.

4. The sessionInterruptionEnded(_:) method is called after the session interruption has ended. A session will continue running from the last known state once the interruption has ended. If the device has moved, any anchors will be misaligned. To avoid this, you restart the session.
5. Show a “Session resumed” message on the screen for 3 seconds.
6. Remove previously rendered objects and reset all labels. You will implement these methods soon. These methods update the UI, so they need to be called on the main thread.
7. Restart the session. runSession() simply resets the session configuration and restarts the tracking with the new configuration.

You will notice there are some compiler errors. You’ll resolve these errors by implementing the missing methods.

Place the following variable in PortalViewController below the other variables:

var debugPlanes: [SCNNode] = []

You’ll use debugPlanes, which is an array of SCNNode objects that keep track of all the rendered horizontal planes in debug mode.

Then, place the following methods below resetLabels():

// 1
func showMessage(_ message: String, label: UILabel, seconds: Double) {
  label.text = message
  label.alpha = 1

  DispatchQueue.main.asyncAfter(deadline: .now() + seconds) {
    if label.text == message {
      label.text = ""
      label.alpha = 0
    }
  }
}

// 2
func removeAllNodes() {
  removeDebugPlanes()
}

// 3
func removeDebugPlanes() {
  for debugPlaneNode in self.debugPlanes {
    debugPlaneNode.removeFromParentNode()
  }

  self.debugPlanes = []
}

Take a look at what’s happening:

1. You define a helper method to show a message string in a given UILabel for the specified duration in seconds. Once the specified number of seconds pass, you reset the visibility and text for the label.
2. removeAllNodes() removes all existing SCNNode objects added to the scene. Currently, you only remove the rendered horizontal planes here.
3. This method removes all the rendered horizontal planes from the scene and resets the debugPlanes array.

Now, place the following line in renderer(_:, didAdd:, for:) just before the #endif of the #if DEBUG preprocessor directive:

self.debugPlanes.append(debugPlaneNode)

This adds the horizontal plane that was just added to the scene to the debugPlanes array.

Note that in runSession(), the session executes with a given configuration:

sceneView?.session.run(configuration)

Replace the line above with the code below:

sceneView?.session.run(configuration,
                       options: [.resetTracking, .removeExistingAnchors])

Here you run the ARSession associated with your sceneView by passing the configuration object and an array of ARSession.RunOptions, with the following run options:

1. resetTracking : The session does not continue device position and motion tracking from the previous configuration.
2. removeExistingAnchors : Any anchor objects associated with the session in its previous configuration are removed.

Run the app and try to detect a horizontal plane.

Now send the app to the background and then re-open the app. Notice that the previously rendered horizontal plane is removed from the scene and the app resets the label to display the correct instructions to the user.

Hit Testing

You are now ready to start placing objects on the detected horizontal planes. You will be using ARSCNView’s hit testing to detect touches from the user’s finger on the screen to see where they land in the virtual scene. A 2D point in the view’s coordinate space can refer to any point along a line segment in the 3D coordinate space. Hit-testing is the process of finding objects in the world located along this line segment.

Open PortalViewController.swift and add the following variable.

var viewCenter: CGPoint {
  let viewBounds = view.bounds
  return CGPoint(x: viewBounds.width / 2.0, y: viewBounds.height / 2.0)
}

In the above block of code, you set the variable viewCenter to the center of the PortalViewController’s view.

Now add the following method:

// 1
override func touchesBegan(_ touches: Set<UITouch>, with event: UIEvent?) {
  // 2
  if let hit = sceneView?.hitTest(viewCenter, types: [.existingPlaneUsingExtent]).first {
    // 3
    sceneView?.session.add(anchor: ARAnchor.init(transform: hit.worldTransform))      
  }
}

Here’s what’s happening:

1. ARSCNView has touches enabled. When the user taps on the view, touchesBegan() is called with a set of UITouch objects and a UIEvent which defines the touch event. You override this touch handling method to add an ARAnchor to the sceneView.
2. You call hitTest(_:, types:) on the sceneView object. The hitTest method has two parameters. It takes a CGPoint in the view’s coordinate system, in this case the screen’s center, and the type of ARHitTestResult to search for.

   Here you use the existingPlaneUsingExtent result type which searches for points where the ray from the viewCenter intersects with any detected horizontal planes in the scene while considering the limited extent of the planes.

   The result of hitTest(_:, types:) is an array of all hit test results sorted from the nearest to the farthest. You pick the first plane that the ray intersects. You will get results for hitTest(_:, types:) any time the screen’s center falls within the rendered horizontal plane.

3. You add an ARAnchor to the ARSession at the point where your object will be placed. The ARAnchor object is initialized with a transformation matrix that defines the anchor’s rotation, translation and scale in world coordinates.

The ARSCNView receives a callback in the delegate method renderer(_:didAdd:for:) after the anchor is added. This is where you handle rendering your portal.

Adding Crosshairs

Before you add the portal to the scene, there is one last thing you need to add in the view. In the previous section, you implemented detecting hit testing for sceneView with the center of the device screen. In this section, you’ll work on adding a view to display the screen’s center so as to help the user position the device.

Open Main.storyboard. Navigate to the Object Library and search for a View object. Drag and drop the view object onto the PortalViewController.

Change the name of the view to Crosshair. Add layout constraints to the view such that its center matches its superview’s centre. Add constraints to set the width and height of the view to 10. In the Size Inspector tab, your constraints should look like this:

Navigate to the Attributes inspector tab and change the background color of the Crosshair view to Light Gray Color.

Select the assistant editor and you’ll see PortalViewController.swift on the right. Press Ctrl and drag from the Crosshair view in storyboard to the PortalViewController code, just above the declaration for sceneView.

Enter crosshair for the name of the IBOutlet and click Connect.

Build and run the app. Notice there’s a gray square view at the center of the screen. This is the crosshair view that you just added.

Now add the following code to the ARSCNViewDelegate extension of the PortalViewController.

// 1
func renderer(_ renderer: SCNSceneRenderer,
              updateAtTime time: TimeInterval) {
  // 2
  DispatchQueue.main.async {
    // 3
    if let _ = self.sceneView?.hitTest(self.viewCenter,
      types: [.existingPlaneUsingExtent]).first {
      self.crosshair.backgroundColor = UIColor.green
    } else { // 4
      self.crosshair.backgroundColor = UIColor.lightGray
    }
  }
}

Here’s what’s happening with the code you just added:

1. This method is part of the SCNSceneRendererDelegate protocol which is implemented by the ARSCNViewDelegate. It contains callbacks which can be used to perform operations at various times during the rendering. renderer(_: updateAtTime:) is called exactly once per frame and should be used to perform any per-frame logic.
2. You run the code to detect if the screen’s center falls in the existing detected horizontal planes and update the UI accordingly on the main queue.
3. This performs a hit test on the sceneView with the viewCenter to determine if the view center indeed intersects with a horizontal plane. If there’s at least one result detected, the crosshair view’s background color is changed to green.
4. If the hit test does not return any results, the crosshair view’s background color is reset to light gray.

Build and run the app.

Move the device around so that it detects and renders a horizontal plane, as shown on the left. Now move the device such that the device screen’s center falls within the plane, as shown on the right. Notice that the center view’s color changes to green.

Adding a State Machine

Now that you have set up the app for detecting planes and placing an ARAnchor, you can get started with adding the portal.

To track the state your app, add the following variables to PortalViewController:

var portalNode: SCNNode? = nil
var isPortalPlaced = false

You store the SCNNode object that represents your portal in portalNode and use isPortalPlaced to keep state of whether the portal is rendered in the scene.

Add the following method to PortalViewController:

func makePortal() -> SCNNode {
  // 1
  let portal = SCNNode()
  // 2
  let box = SCNBox(width: 1.0,
                   height: 1.0,
                   length: 1.0,
                   chamferRadius: 0)
  let boxNode = SCNNode(geometry: box)
  // 3
  portal.addChildNode(boxNode)  
  return portal
}

Here you define makePortal(), a method that will configure and render the portal. There are a few things happening here:

1. You create an SCNNode object which will represent your portal.
2. This initializes a SCNBox object which is a cube and makes a SCNNode object for the box using the SCNBox geometry.
3. You add the boxNode as a child node to your portal and return the portal node.

Here, makePortal() is creating a portal node with a box object inside it as a placeholder.

Now replace the renderer(_:, didAdd:, for:) and renderer(_:, didUpdate:, for:) methods for the SCNSceneRendererDelegate with the following:

func renderer(_ renderer: SCNSceneRenderer, didAdd node: SCNNode, for anchor: ARAnchor) {
  DispatchQueue.main.async {
    // 1
    if let planeAnchor = anchor as? ARPlaneAnchor, 
    !self.isPortalPlaced {
      #if DEBUG
        let debugPlaneNode = createPlaneNode(
          center: planeAnchor.center,
          extent: planeAnchor.extent)
        node.addChildNode(debugPlaneNode)
        self.debugPlanes.append(debugPlaneNode)
      #endif
      self.messageLabel?.alpha = 1.0
      self.messageLabel?.text = """
            Tap on the detected \
            horizontal plane to place the portal
            """
    }
    else if !self.isPortalPlaced {// 2
        // 3
      self.portalNode = self.makePortal()
      if let portal = self.portalNode {
        // 4
        node.addChildNode(portal)
        self.isPortalPlaced = true

        // 5
        self.removeDebugPlanes()
        self.sceneView?.debugOptions = []

        // 6
        DispatchQueue.main.async {
          self.messageLabel?.text = ""
          self.messageLabel?.alpha = 0
        }
      }

    }
  }
}

func renderer(_ renderer: SCNSceneRenderer,
              didUpdate node: SCNNode,
              for anchor: ARAnchor) {
  DispatchQueue.main.async {
    // 7
    if let planeAnchor = anchor as? ARPlaneAnchor,
      node.childNodes.count > 0,
      !self.isPortalPlaced {
      updatePlaneNode(node.childNodes[0],
                      center: planeAnchor.center,
                      extent: planeAnchor.extent)
    }
  }
}

Here are the changes you made:

1. You’re adding a horizontal plane to the scene to show the detected planes only if the anchor that was added to the scene is an ARPlaneAnchor, and only if isPortalPlaced equals false, which means the portal has not yet been placed.
2. If the anchor that was added was not an ARPlaneAnchor, and the portal node still hasn’t been placed, this must be the anchor you add when the user taps on the screen to place the portal.
3. You create the portal node by calling makePortal().
4. renderer(_:, didAdd:, for:) is called with the SCNNode object, node, that is added to the scene. You want to place the portal node at the location of the node. So you add the portal node as a child node of node and you set isPortalPlaced to true to track that the portal node has been added.

5. To clean up the scene, you remove all rendered horizontal planes and reset the debugOptions for sceneView so that the feature points are no longer rendered on screen.
6. You update the messageLabel on the main thread to reset its text and hide it.
7. In the renderer(_:, didUpdate:, for:) you update the rendered horizontal plane only if the given anchor is an ARPlaneAnchor, if the node has at least one child node and if the portal hasn’t been placed yet.

Finally, replace removeAllNodes() with the following.

func removeAllNodes() {
  // 1
  removeDebugPlanes()
  // 2
  self.portalNode?.removeFromParentNode()
  // 3
  self.isPortalPlaced = false
}

This method is used for cleanup and removing all rendered objects from the scene. Here’s a closer look at what’s happening:

1. You remove all the rendered horizontal planes.
2. You then remove the portalNode from its parent node.
3. Change the isPortalPlaced variable to false to reset the state.

Build and run the app; let the app detect a horizontal plane and then tap on the screen when the crosshair view turns green. You will see a rather plain-looking, huge white box.

This is the placeholder for your portal. In the next and final part of this tutorial series, you’ll add some walls and a doorway to the portal. You’ll also add textures to the walls so that they look more realistic.

Where to Go From Here?

This has been quite a ride! Here’s a summary of what you learned in this tutorial:

• You can now detect and handle ARSession interruptions when the app goes to the background.
• You understand how hit testing works with an ARSCNView and the detected horizontal planes in the scene.
• You can use the results of hit testing to place ARAnchors and SCNNode objects corresponding to them.

In the upcoming final part of this tutorial series, you’ll pull everything together, add the walls and ceiling, and add a bit of lighting to the scene!

If you enjoyed what you learned in this tutorial, why not check out our complete book, ARKit by Tutorials, available on our online store?

ARKit is Apple’s mobile AR development framework. With it, you can create an immersive, engaging experience, mixing virtual 2D and 3D content with the live camera feed of the world around you.

If you’ve worked with any of Apple’s other frameworks, you’re probably expecting that it will take a long time to get things working. But with ARKit, it only takes a few lines of code — ARKit does most of the the heavy lifting for you, so you can focus on what’s important: creating an immersive and engaging AR experience.

In this book, you’ll create five immersive and engaging apps: a tabletop poker dice game, an immersive sci-fi portal, a 3D face-tracking mask app, a location-based AR ad network, and monster truck simulation with realistic vehicle physics.

To celebrate the launch of the book, it’s currently on sale as part of our Game On book launch event. But don’t wait too long, as this deal is only good until Friday, June 8th!

If you have any questions or comments on this tutorial, feel free to join the discussion below!

The post Building a Portal App in ARKit: Adding Objects appeared first on Ray Wenderlich.