ARKit Tutorial in Swift 4 for Xcode 9 using SceneKit

ARKit Tutorial in Swift 4 for Xcode 9

In this tutorial I’m going to show you how to work with ARKit, the new Framework from Apple that allows us to easily create Augmented Reality experiences in our iOS apps. The first thing to know about ARKit is that it can be used in three major ways:

  • Inside of Xcode writing Swift or Obj-C Code
  • Inside of the Unity game engine
  • Inside of the Unreal game engine

In this tutorial, we’ll be working with the first option.

We will be writing the app shown above in Swift 4 and Xcode 9 Beta in this tutorial. If you are a game developer and comfortable working with Unity or Unreal, this tutorial is not intended for you. I recommend following Unity and Epic’s documentation respectively if you are working in one of those environments. However, if you want to learn how ARKit works under the hood, this still may be useful in understanding what’s happening when you work with the plugins provided. At the end of the day, your game engine is still wrapping the native Swift or Obj-C calls to ARKit.

Getting started

First off, create a new Xcode project and use the Augmented Reality template. (Xcode 9 Beta and above only)

From here you can run the app immediately on your device and you’ll see a spaceship floating in space. As you walk around with the phone, it should track your movements and simulate the AR effect with the ship.

From here, we’re going to replace this model with a custom one we download. Then, I’ll go over how to use the ARKit hitscan and anchor features to place this custom model on real-world objects.

If you’re already familiar with SceneKit and 3D models, I recommend skipping to the Section 2

Finding a model to work with

I like to use TurboSquid to get models to play around with when I’m prototyping games, and working with AR will be similar. So you can find free (or cheap) 3D models over here. One thing to be aware of though is that some models here are meant for non-realtime rendering, so the poly counts may be high. Any time you are planning to grab a model from TurboSquid, always check that the poly count is not too high. Anything over 10k is pretty high poly for a mobile app. As another reference point, very high end games designed to be played on cutting edge gaming PCs generally have between 50k and 150k polygons for a main character model. If you see a poly count that high it should be for an extremely detailed model, and would ideally be one of very few items in the scene. What you really want for mobile is something low poly, and if it’s a detailed model you should expect a normal map to be included. A normal map will create the illusion that there are more polys than there actually are, without a significant performance impact.

SceneKit has support for DAE models. (If you have another file format model that want to work with, I recommend getting a copy of blender and converting your models to DAE by importing and then exporting as DAE within the application.)

Turbosquid has a filter option that does not include DAE, however it works fine to just type in “DAE”. Using this filter will help you to skip the process of converting models, and instead allow us to focus on code. As a shortcut, you can use this direct link to find DAE models that are free and less than 10k polys. Every model on this list should be appropriate for use in SceneKit.

I like this tree model. We could create an AR experience where users can “plant” trees in AR and walk around through them. You can download the tree model here after signing up for a free account.

Import the DAE model in to the Xcode Project

So download the DAE version of the model and then you can drag the file in to the Xcode navigator in to your art.scnassets folder that was created from the AR app template. Once you have added the asset to your Xcode project, you should be able to preview it within Xcode hold click and drag around to rotate around your mesh. If you don’t see anything in this step there may be an issue with your model.

Section 2

Load the model in to your scene

Now that the model is in your project you can open up the ViewController.swift file created by the template and swap out the spaceship for your model. Find this line of code:

let scene = SCNScene(named: "art.scnassets/ship.scn")!

 

Change it to point to your downloaded model. In my case it’s Lowpoly_tree_sample.dae.

let scene = SCNScene(named: "art.scnassets/Lowpoly_tree_sample.dae")!

 

When running the app with this change you will not be able to see the model because it is much too large, and not positioned in front of the camera. An easy way to check the size of a model is to cut & paste the spaceship model from the default scn file in to the dae file using the SceneKit editor in Xcode. This will tell you the relative sizes, and in this case the tree is around 100 times too large.

Select the tree model in the SceneKit editor view by clicking it, and then on the right-hand side pane select the Node inspector. This is the tab with the cube icon. Here we can set the x, y, and z scale to 0.01, reducing the size of the tree by 99%. You can also set the scale in code, as well as position and rotation.

You may notice that in the default position, the tree is not visible. This is because it’s origin is at position 0, 0, 0, and it’s invisible because the camera is actually inside of the tree at this location. If you start up the AR app on a device and walk around a bit, when you turn around you’ll see the tree. But let’s just move it out some using some code. What we’ll do is change the position to be 1 unit in front of the camera in it’s default location. In order to do this we’ll need to find the tree as a node within the scene. Going back to the SceneKit editor (select the DAE file within Xcode), we can click on the tree model itself, and then again in the Node inspector there should be a name. This is the name of the mesh selected, in this case Tree_lp_11. If you’re using a different model the name may be different, or empty. Fortunately we can just type in our own name as needed.

Now that we know the name of the node, we can access it within our code.

let scene = SCNScene(named: "art.scnassets/Lowpoly_tree_sample.dae")!
let treeNode = scene.rootNode.childNode(withName: "Tree_lp_11", recursively: true)

 

The second line above does a search of the child nodes of the scene object created from the DAE file we imported, and returns a node with the name specified. Since 3D models can have deeply nested nodes, it’s often useful to recursively search through the nested heirarchy to find the mesh object, so we opt to search recursively, even though it is not neccessary for this particular model.

From here, we can simply reposition the node by moving it forward a bit. The means going in a negative direction in the z axis, as the default camera faces down the negative Z. Or in other words it’s looking at the point (0, 0, -Inf).

So to make the tree visible, let’s move it back 1 unit in the z direction. The easiest way to do this is just set the new z to -1 on the position object.

let scene = SCNScene(named: "art.scnassets/Lowpoly_tree_sample.dae")!
let treeNode = scene.rootNode.childNode(withName: "Tree_lp_11", recursively: true)
treeNode?.position.z = -1

 

This will work, but in practice what is more common is to create a new position object from scratch and assign that as the new position. This is based on taste, but it’s a common approach to avoid mutating positions of 3D objects, and instead to replace them. To do this we need to construct a SCNVector3 object with the position (0,0,-1). This code has the exact same effect, but is a better practice:

let scene = SCNScene(named: "art.scnassets/Lowpoly_tree_sample.dae")!
let treeNode = scene.rootNode.childNode(withName: "Tree_lp_11", recursively: true)
treeNode?.position = SCNVector3Make(0, 0, -1)

 

In order to modify the treeNode later, let’s keep an instance reference to it. Outside of any functions, but inside the ViewController class add an optional reference to treeNode:

class ViewController: UIViewController, ARSCNViewDelegate {
  var treeNode: SCNNode?
  ...
}

 

In the next step we’re going to want a reference to the treeNode again, so rather than doing the lookup every time, it is useful to cache the reference to the treeNode. To do this, I’ll modify the childNode call we just added inside of viewDidLoad so that it sets this to an instance variable treeNode as opposed to just the local treeNode variable:

let scene = SCNScene(named: "art.scnassets/Lowpoly_tree_sample.dae")!
self.treeNode = scene.rootNode.childNode(withName: "Tree_lp_11", recursively: true)
self.treeNode?.position = SCNVector3Make(0, 0, -1)

 

Although AR is not supported in the simulator, SceneKit is, and this is just a SceneKit model. So if you run the simulator you’ll find you can now see the tree model with a black background.

ARKit model rendering in iPhone Simulator

Using HitTest

Next, let’s modify the app so that when the user taps, it will place a new copy of the tree model to wherever he tapped. This is a very complex operation because tapping on the 2D screen needs to go through a projection matrix based on the estimated shape of the 3D scene it’s seeing through the camera. ARKit makes this complicated process fairly simple by providing the HitTest api.

So first, we implement an override for touchesBegan. This is called any time the user taps on the screen. From this we can retrieve the first touch and perform a hitTest in the ARScene. We’ll look for results that are anchors. These are basically points in space ARKit has identified and is tracking. In other words it’s a surface of some real-world object.

Once we get a result from the hitTest, we can position the treeNode to the same location as the anchor. The hit test comes back with a 4×4 matrix containing the scale, rotation, and position data. The 4th row of this matrix is the position. We can reconstruct the position using this row by accessing m41, m42, and m43 as x, y, and z respectively. Setting the position to a new SCNVector3 object with these coordinates should move the tree to that location.

override func touchesBegan(_ touches: Set<UITouch>, with event: UIEvent?) {
  guard let touch = touches.first else { return }
  let results = sceneView.hitTest(touch.location(in: sceneView), types: [ARHitTestResult.ResultType.featurePoint])
  guard let hitFeature = results.last else { return }
  let hitTransform = SCNMatrix4(hitFeature.worldTransform)
  let hitPosition = SCNVector3Make(hitTransform.m41,
                                   hitTransform.m42,
                                   hitTransform.m43)
  treeNode?.position = hitPosition;
}

 

Give this a try and you’ll find you can teleport the tree to locations in the real-world when you tap!

Xcode 9 Beta 1 vs Xcode 9 Beta 2

Note: Some users have complained that the call SCNMatrix4 function call causes a compiler error. This seems to be caused by users on Xcode 9 Beta 1. The new SCNMatrix4 initializer was added in Xcode 9 Beta 2. If you run in to this problem because you are still on Beta 1, just replace the function SCNMatrix4 with the function SCNMatrix4FromMat4 and this should fix the issue.

Similarly, we can make clones of the tree as well so we can plant our forest 🙂

override func touchesBegan(_ touches: Set<UITouch>, with event: UIEvent?) {
  guard let touch = touches.first else { return }
  let results = sceneView.hitTest(touch.location(in: sceneView), types: [ARHitTestResult.ResultType.featurePoint])
  guard let hitFeature = results.last else { return }
  let hitTransform = SCNMatrix4(hitFeature.worldTransform)
  let hitPosition = SCNVector3Make(hitTransform.m41,
                                   hitTransform.m42,
                                   hitTransform.m43)
  let treeClone = treeNode!.clone()
  treeClone.position = hitPosition
  sceneView.scene.rootNode.addChildNode(treeClone)
}

 

From here you can imagine some of the interesting things that could be done by adding interactions, animations, etc to the scene. To proceed from here with your project what you should learn is how to work with SceneKit, as those fundamentals will apply to most of your ARKit based apps.

You can find the complete source code to this tutorial on Gumroad.

Questions? Comments? Hate mail? Let me know what you thought of this tutorial on my contact page. I’ll be making the video version of this tutorial soon, so check back or subscribe on my YouTube channel to keep up to date.

Did this tutorial help you?

Support my Patreon

Your support on Patreon allows me to make better tutorials more often.

Subscribe via RSS

Core NFC Tutorial for NFC on iOS Devices

With the release of iOS 11, for the first time third-party developers are able to use the NFC reader on devices iPhone 7 and higher. This could be used for passing along identification information and a whole host of other data exchange applications from door locks to subway passes.

The technology used on iOS 11 is called Core NFC, and I’m going to go over how to use it in this tutorial in Swift 4.

Core NFC Device communications on iOS

Because Core NFC is currently read-only, functionality such as contactless payments will not be possible out of the box, there are however other applications we can use the Core NFC reading capabilities for. So let me show you how it works.

The first step to work with NFC is to enable it for your App ID in the Apple Developer Center. Create a new App ID and enable “NFC Tag Reading” as a capability for the app.

NFC Tag Reading App Service

After you do this, I recommend creating a development/distribution provisioning profile specifically for this app ID, so that the NFC reading capability will be present when you try to build.

Next, in your Xcode project add the entitlement to your projectName.entitlements file. You’ll need to right click the file and select “Open As Source Code” to manually enter this key as shown:

<key>com.apple.developer.nfc.readersession.formats</key>
<array>
  <string>NDEF</string>
</array>

If you do not have an entitlements file, you can manually create one and point to it in the settings of the project. Under “Build Settings” look for the record “Code Signing Entitlements”, and punch in the relative path to your entitlements file. In my case it’s “CoreNFC-Tutorial/CoreNFC-Tutorial.entitlements” because my project files are inside a sub-folder called “CoreNFC-Tutorial”.

Next, we need to add the usage string to your Xcode project. Open your Info.plist file and add a new record, start typing and allow it to autocomplete “Privacy – NFC Scan Usage Description”. This message will be shown to users when NFC is used, so for the value enter something useful for the user such as “NFC is needed to unlock doors.”.

Next, in our code we want to import the CoreNFC module.

import CoreNFC

Note: Core NFC is completely unavailable on the iOS simulator, and even importing the module will fail. So Core NFC is for device-only testing!

I created an NFCHelper.swift to store all my NFC-related calls, and put everything in an NFCHelper class. In the init method of the class I created a session. Core NFC requires you utilize the NFCNDEFReaderSession class in order to start listening for NFC communications. (Note the class NFCReaderSession is abstract and should not be used directly)

class NFCHelper {
  init() {
    let session =
      NFCNDEFReaderSession(delegate: self,
                           queue: nil,
                           invalidateAfterFirstRead: true)
    session.begin()
  }
}

Here we create the session, and pass in a nil dispatch queue. Doing so will cause NFCNDEFReaderSession to automatically create a serial dispatch queue.

Creating a session, we can specify a delegate in our NFCNDEFReaderSession class. I would like to use the NFCHelper class as the delegate, so we must first adhere to the delegate protocol, NFCNDEFReaderSessionDelegate. This delegate is based on an Objective-C object, so we must first also adhere to NSObject. NFCNDEFReaderSessionDelegate has two delegate methods we must implement:

func readerSession(_ session: NFCNDEFReaderSession, didInvalidateWithError error: Error)
 
func readerSession(_ session: NFCNDEFReaderSession, didDetectNDEFs messages: [NFCNDEFMessage])

These two callbacks are called when an NFC session has a validation error, or when NFC activity is detected. How we use the messages will depend on your specific application, but everything you need to know will be sent through the didDetectNDEFs callback in the form of records inside of messages. To get started, you can log the results of each message in a loop. These are NFCNDEFPayload objects, which contain an identifier, payload, type, and typeNameFormat.

func readerSession(_ session: NFCNDEFReaderSession, didDetectNDEFs messages: [NFCNDEFMessage]) {
  print("Did detect NDEFs.")
  // Loop through all messages
  for message in messages {
    for record in message.records {
      print(record.identifier)
      print(record.payload)
      print(record.type)
      print(record.typeNameFormat)
    }
  }
}

To clean this up a little bit so I can actually integrate with the front-end of the app, I created a callback specific for my application. You may want to do something similar. I added a callback variable that the implementing view can work with:

I call this when I get a payload from an NFC tag, or an error:

class NFCHelper: NSObject, NFCNDEFReaderSessionDelegate {
  ...
  var onNFCResult: ((Bool, String) -> ())?
  ...
}

I also broke out my init function to open up a session using a method, so I can restart the session from a button on my ViewController. My final code for my NFCHelper.swift file is as follows:

//
//  NFCHelper.swift
//  CoreNFC-Tutorial
//
//  Created by Jameson Quave on 6/6/17.
//  Copyright © 2017 Jameson Quave. All rights reserved.
//
 
import Foundation
import CoreNFC
 
class NFCHelper: NSObject, NFCNDEFReaderSessionDelegate {
 
  var onNFCResult: ((Bool, String) -> ())?
 
  func restartSession() {
    let session =
    NFCNDEFReaderSession(delegate: self,
                       queue: nil,
                       invalidateAfterFirstRead: true)
    session.begin()
  }
  // MARK: NFCNDEFReaderSessionDelegate
  func readerSession(_ session: NFCNDEFReaderSession, didInvalidateWithError error: Error) {
    guard let onNFCResult = onNFCResult else {
      return
    }
    onNFCResult(false, error.localizedDescription)
  }
  func readerSession(_ session: NFCNDEFReaderSession, didDetectNDEFs messages: [NFCNDEFMessage]) {
    guard let onNFCResult = onNFCResult else {
      return
    }
    for message in messages {
      for record in message.records {
        if(record.payload.count > 0) {
          if let payloadString = String.init(data: record.payload, encoding: .utf8) {
              onNFCResult(true, payloadString)
          }
        }
      }
    }
  }
 
}

I also set up a simple UI in my ViewController to demonstrate usage of this class:

//
//  ViewController.swift
//  CoreNFC-Tutorial
//
//  Created by Jameson Quave on 6/6/17.
//  Copyright © 2017 Jameson Quave. All rights reserved.
//
 
import UIKit
 
class ViewController: UIViewController {
  var helper: NFCHelper?
  var payloadLabel: UILabel!
  var payloadText = ""
  override func viewDidLoad() {
    super.viewDidLoad()
    // Add a detect button
    let button = UIButton(type: .system)
    button.setTitle("Read NFC", for: .normal)
    button.titleLabel?.font = UIFont(name: "Helvetica", size: 28.0)
    button.isEnabled = true
    button.addTarget(self, action: #selector(didTapReadNFC), for: .touchUpInside)
    button.frame = CGRect(x: 60, y: 200, width: self.view.bounds.width - 120, height: 80)
    self.view.addSubview(button)
    // Add a label to display the payload in
    payloadLabel = UILabel(frame: button.frame.offsetBy(dx: 0, dy: 220))
    payloadLabel.text = "Press Read to see payload data."
    payloadLabel.numberOfLines = 100
    self.view.addSubview(payloadLabel)
  }
  // Called by NFCHelper when a tag is read, or fails to read
  func onNFCResult(success: Bool, message: String) {
    if success {
      payloadText = "\(payloadText)\n\(message)"
    }
    else {
      payloadText = "\(payloadText)\n\(message)"
    }
    // Update UI on main thread
    DispatchQueue.main.async {
      self.payloadLabel.text = self.payloadText
    }
 
  }
  // Called when user taps Read NFC Button
  @objc func didTapReadNFC() {
    if helper == nil {
      helper = NFCHelper()
      helper?.onNFCResult = self.onNFCResult(success:message:)
    }
    payloadText = ""
    helper?.restartSession()
  }
}

From here you can integrate the rest of your app with your intended use-case. Whether it’s to identify visitors to an event, check out the stats on your Amiibo, or even process payments, the Core NFC API from Apple has finally opened up the possibilities for NFC integration on these new devices. What kind of NFC product are you working on? Let me know at jquave@gmail.com.

Full Source Code

Video Tutorial (YouTube)

Did this tutorial help you?

Support my Patreon

Your support on Patreon allows me to make better tutorials more often.

Subscribe via RSS