If this post is helpful for you, consider donating to my patreon. It helps me produce better tutorials, faster. Visit my Patreon.

Become a Patron!

ARKit Tutorial in Swift 4 for Xcode 9 using SceneKit

 

ARKit Tutorial in Swift 4 for Xcode 9

In this tutorial I’m going to show you how to work with ARKit, the new Framework from Apple that allows us to easily create Augmented Reality experiences in our iOS apps. The first thing to know about ARKit is that it can be used in three major ways:

  • Inside of Xcode writing Swift or Obj-C Code
  • Inside of the Unity game engine
  • Inside of the Unreal game engine

In this tutorial, we’ll be working with the first option.

We will be writing the app shown above in Swift 4 and Xcode 9 Beta in this tutorial. If you are a game developer and comfortable working with Unity or Unreal, this tutorial is not intended for you. I recommend following Unity and Epic’s documentation respectively if you are working in one of those environments. However, if you want to learn how ARKit works under the hood, this still may be useful in understanding what’s happening when you work with the plugins provided. At the end of the day, your game engine is still wrapping the native Swift or Obj-C calls to ARKit.

Getting started

First off, create a new Xcode project and use the Augmented Reality template. (Xcode 9 Beta and above only)

From here you can run the app immediately on your device and you’ll see a spaceship floating in space. As you walk around with the phone, it should track your movements and simulate the AR effect with the ship.

From here, we’re going to replace this model with a custom one we download. Then, I’ll go over how to use the ARKit hitscan and anchor features to place this custom model on real-world objects.

If you’re already familiar with SceneKit and 3D models, I recommend skipping to the Section 2

Finding a model to work with

I like to use TurboSquid to get models to play around with when I’m prototyping games, and working with AR will be similar. So you can find free (or cheap) 3D models over here. One thing to be aware of though is that some models here are meant for non-realtime rendering, so the poly counts may be high. Any time you are planning to grab a model from TurboSquid, always check that the poly count is not too high. Anything over 10k is pretty high poly for a mobile app. As another reference point, very high end games designed to be played on cutting edge gaming PCs generally have between 50k and 150k polygons for a main character model. If you see a poly count that high it should be for an extremely detailed model, and would ideally be one of very few items in the scene. What you really want for mobile is something low poly, and if it’s a detailed model you should expect a normal map to be included. A normal map will create the illusion that there are more polys than there actually are, without a significant performance impact.

SceneKit has support for DAE models. (If you have another file format model that want to work with, I recommend getting a copy of blender and converting your models to DAE by importing and then exporting as DAE within the application.)

Turbosquid has a filter option that does not include DAE, however it works fine to just type in “DAE”. Using this filter will help you to skip the process of converting models, and instead allow us to focus on code. As a shortcut, you can use this direct link to find DAE models that are free and less than 10k polys. Every model on this list should be appropriate for use in SceneKit.

I like this tree model. We could create an AR experience where users can “plant” trees in AR and walk around through them. You can download the tree model here after signing up for a free account.

Import the DAE model in to the Xcode Project

So download the DAE version of the model and then you can drag the file in to the Xcode navigator in to your art.scnassets folder that was created from the AR app template. Once you have added the asset to your Xcode project, you should be able to preview it within Xcode hold click and drag around to rotate around your mesh. If you don’t see anything in this step there may be an issue with your model.

Section 2

Load the model in to your scene

Now that the model is in your project you can open up the ViewController.swift file created by the template and swap out the spaceship for your model. Find this line of code:

let scene = SCNScene(named: "art.scnassets/ship.scn")!

 

Change it to point to your downloaded model. In my case it’s Lowpoly_tree_sample.dae.

let scene = SCNScene(named: "art.scnassets/Lowpoly_tree_sample.dae")!

 

When running the app with this change you will not be able to see the model because it is much too large, and not positioned in front of the camera. An easy way to check the size of a model is to cut & paste the spaceship model from the default scn file in to the dae file using the SceneKit editor in Xcode. This will tell you the relative sizes, and in this case the tree is around 100 times too large.

Select the tree model in the SceneKit editor view by clicking it, and then on the right-hand side pane select the Node inspector. This is the tab with the cube icon. Here we can set the x, y, and z scale to 0.01, reducing the size of the tree by 99%. You can also set the scale in code, as well as position and rotation.

You may notice that in the default position, the tree is not visible. This is because it’s origin is at position 0, 0, 0, and it’s invisible because the camera is actually inside of the tree at this location. If you start up the AR app on a device and walk around a bit, when you turn around you’ll see the tree. But let’s just move it out some using some code. What we’ll do is change the position to be 1 unit in front of the camera in it’s default location. In order to do this we’ll need to find the tree as a node within the scene. Going back to the SceneKit editor (select the DAE file within Xcode), we can click on the tree model itself, and then again in the Node inspector there should be a name. This is the name of the mesh selected, in this case Tree_lp_11. If you’re using a different model the name may be different, or empty. Fortunately we can just type in our own name as needed.

Now that we know the name of the node, we can access it within our code.

let scene = SCNScene(named: "art.scnassets/Lowpoly_tree_sample.dae")!
let treeNode = scene.rootNode.childNode(withName: "Tree_lp_11", recursively: true)

 

The second line above does a search of the child nodes of the scene object created from the DAE file we imported, and returns a node with the name specified. Since 3D models can have deeply nested nodes, it’s often useful to recursively search through the nested heirarchy to find the mesh object, so we opt to search recursively, even though it is not neccessary for this particular model.

From here, we can simply reposition the node by moving it forward a bit. The means going in a negative direction in the z axis, as the default camera faces down the negative Z. Or in other words it’s looking at the point (0, 0, -Inf).

So to make the tree visible, let’s move it back 1 unit in the z direction. The easiest way to do this is just set the new z to -1 on the position object.

let scene = SCNScene(named: "art.scnassets/Lowpoly_tree_sample.dae")!
let treeNode = scene.rootNode.childNode(withName: "Tree_lp_11", recursively: true)
treeNode?.position.z = -1

 

This will work, but in practice what is more common is to create a new position object from scratch and assign that as the new position. This is based on taste, but it’s a common approach to avoid mutating positions of 3D objects, and instead to replace them. To do this we need to construct a SCNVector3 object with the position (0,0,-1). This code has the exact same effect, but is a better practice:

let scene = SCNScene(named: "art.scnassets/Lowpoly_tree_sample.dae")!
let treeNode = scene.rootNode.childNode(withName: "Tree_lp_11", recursively: true)
treeNode?.position = SCNVector3Make(0, 0, -1)

 

In order to modify the treeNode later, let’s keep an instance reference to it. Outside of any functions, but inside the ViewController class add an optional reference to treeNode:

class ViewController: UIViewController, ARSCNViewDelegate {
  var treeNode: SCNNode?
  ...
}

 

In the next step we’re going to want a reference to the treeNode again, so rather than doing the lookup every time, it is useful to cache the reference to the treeNode. To do this, I’ll modify the childNode call we just added inside of viewDidLoad so that it sets this to an instance variable treeNode as opposed to just the local treeNode variable:

let scene = SCNScene(named: "art.scnassets/Lowpoly_tree_sample.dae")!
self.treeNode = scene.rootNode.childNode(withName: "Tree_lp_11", recursively: true)
self.treeNode?.position = SCNVector3Make(0, 0, -1)

 

Although AR is not supported in the simulator, SceneKit is, and this is just a SceneKit model. So if you run the simulator you’ll find you can now see the tree model with a black background.

ARKit model rendering in iPhone Simulator

Using HitTest

Next, let’s modify the app so that when the user taps, it will place a new copy of the tree model to wherever he tapped. This is a very complex operation because tapping on the 2D screen needs to go through a projection matrix based on the estimated shape of the 3D scene it’s seeing through the camera. ARKit makes this complicated process fairly simple by providing the HitTest api.

So first, we implement an override for touchesBegan. This is called any time the user taps on the screen. From this we can retrieve the first touch and perform a hitTest in the ARScene. We’ll look for results that are anchors. These are basically points in space ARKit has identified and is tracking. In other words it’s a surface of some real-world object.

Once we get a result from the hitTest, we can position the treeNode to the same location as the anchor. The hit test comes back with a 4×4 matrix containing the scale, rotation, and position data. The 4th row of this matrix is the position. We can reconstruct the position using this row by accessing m41, m42, and m43 as x, y, and z respectively. Setting the position to a new SCNVector3 object with these coordinates should move the tree to that location.

override func touchesBegan(_ touches: Set<UITouch>, with event: UIEvent?) {
  guard let touch = touches.first else { return }
  let results = sceneView.hitTest(touch.location(in: sceneView), types: [ARHitTestResult.ResultType.featurePoint])
  guard let hitFeature = results.last else { return }
  let hitTransform = SCNMatrix4(hitFeature.worldTransform)
  let hitPosition = SCNVector3Make(hitTransform.m41,
                                   hitTransform.m42,
                                   hitTransform.m43)
  treeNode?.position = hitPosition;
}

 

Give this a try and you’ll find you can teleport the tree to locations in the real-world when you tap!

Xcode 9 Beta 1 vs Xcode 9 Beta 2

Note: Some users have complained that the call SCNMatrix4 function call causes a compiler error. This seems to be caused by users on Xcode 9 Beta 1. The new SCNMatrix4 initializer was added in Xcode 9 Beta 2. If you run in to this problem because you are still on Beta 1, just replace the function SCNMatrix4 with the function SCNMatrix4FromMat4 and this should fix the issue.

Similarly, we can make clones of the tree as well so we can plant our forest 🙂

override func touchesBegan(_ touches: Set<UITouch>, with event: UIEvent?) {
  guard let touch = touches.first else { return }
  let results = sceneView.hitTest(touch.location(in: sceneView), types: [ARHitTestResult.ResultType.featurePoint])
  guard let hitFeature = results.last else { return }
  let hitTransform = SCNMatrix4(hitFeature.worldTransform)
  let hitPosition = SCNVector3Make(hitTransform.m41,
                                   hitTransform.m42,
                                   hitTransform.m43)
  let treeClone = treeNode!.clone()
  treeClone.position = hitPosition
  sceneView.scene.rootNode.addChildNode(treeClone)
}

 

From here you can imagine some of the interesting things that could be done by adding interactions, animations, etc to the scene. To proceed from here with your project what you should learn is how to work with SceneKit, as those fundamentals will apply to most of your ARKit based apps.

You can find the complete source code to this tutorial on Gumroad.

Questions? Comments? Hate mail? Let me know what you thought of this tutorial on my contact page. I’ll be making the video version of this tutorial soon, so check back or subscribe on my YouTube channel to keep up to date.

Did this tutorial help you?

Support my Patreon

Your support on Patreon allows me to make better tutorials more often.

Follow me on Twitter
Subscribe via RSS

31 thoughts on “ARKit Tutorial in Swift 4 for Xcode 9 using SceneKit

  1. Pingback: ARKit on iOS 11 | iOS Swift Tutorials by Jameson Quave

  2. Hi and thanks for the tutorial. I couldn’t get it to work this line let hitPointTransform = SCNMatrix4(anchor.worldTransform) doesn’t let me pass in those arguments. I get cannot invoke initialiser for type SCNMatrix4 with an argument list of type (matrix_float4x4). Also following your tutorial exactly the model floats in the air even after changing the position and scale. Any thoughts? Maybe you could post your project and I could dissect it 🙂

  3. Nice tutorial, thanks! I ran into a couple minor issues, so just thought I’d share how I resolved them:

    1. Xcode could not save changes to the Lowpoly_tree_sample.dae file. I first tried to change permissions on the file but that didn’t work. So I just scaled it in code (down to 0.01 to make each tree node mostly visible onscreen):

    treeNode.scale = SCNVector3Make(0.01, 0.01, 0.01)

    2. The tutorial shows creating the treeNode in viewDidLoad, so it would not be available in touchesBegan(with:). I created a property for it instead:

    var treeNode: SCNNode!

    …and then initialized it in viewDidLoad:

    treeNode = scene.rootNode.childNode(withName: “Tree_lp_11”, recursively: true)!

    Here’s a link to my ViewController.swift file with these changes:

    https://www.dropbox.com/s/mh1j2klcollgxk2/ViewController.swift?dl=0

  4. Thanks for a great introduction. I have a question about

    let hitPointTransform = SCNMatrix4(anchor.worldTransform)

    i had to use the following to get it to compile

    let hitPointTransform = SCNMatrix4FromMat4(anchor.worldTransform)

    Am I missing something?

    • I’m not sure, multiple people have said this, but when I call that function it says SCNMatrix4FromMat4 is not available in Swift, and the SCNMatrix4() constructor works fine. Do you happen to have beta 1? I’m using beta 2. That’s the only thing I can thing that might cause this difference.

  5. Cannot invoke initializer for type ‘SCNMatrix4’ with an argument list of type ‘(matrix_float4x4)’
    Overloads for ‘SCNMatrix4’ exist with these partially matching parameter lists: (float4x4), (double4x4)

    • Same reply as I gave above. This works for me, apparently SCNMatrix4FromMat needs to be used sometime. When is sometimes? I’m not sure, but guessing in Beta 1, as I am on Beta 2.

  6. I’m trying to import .dae file but I’m trying to put this .dae formats on one plane already detected. But it’s impossible always the object is above of my current position. Do you know if there are one way to change the position of a .dae model? (With .scn model woks right…)

  7. Hi, thanks for this tutorial. i’ll also shoot a message via contact but i’ll put this here too incase someone else is running into the issue. I’m not 100% sure where to put the touchesbegan override. I’m new to swift. I placed it after the viewDidLoad() function and i get an error with the node variable name “use of unresolved identifier” as well as a initializer error for SCNMatrix4.

    if i place it below the new scene lines outlined above i get an error about overrides being only specified on classes.

    any help would be appreciated, there aren’t many tutorials out yet so it’s been hard to find a similar one that uses the touchesbegan override.

  8. Just wanted to note that let hitPointTransform = SCNMatrix4(anchor.worldTransform) doesn’t work but let hitPointTransform = SCNMatrix4FromMat4(anchor.worldTransform) does.

  9. I get an error of “Use of unresolved identifier ‘treeNode’ ” once i get to the “treeNode?.position = hitpointPosition” you sure your tutorial has the proper code? everything was working up until i put the override func touchesBegan into the ViewController file. Trying to understand this and im rather lost.

  10. I’ve tried this tutorial and your code in github but I always get a black background screen with the tree in the middle. I never get this to not be nil: guard let hitFeature = results.last else { return }

    What can it be? I’m testing it in an iPhone 6 Plus

  11. Hi Jameson, thanks for the great tutorial. Regarding the recommended polygons. Do you know of, or have any additional information on this? Have you tried implementing a scene with several low poly objects or a single high poly objects? Wonder what’s the poly count of the scene for a demo like Wingnut AR. Thanks.

    • I assume that Wingnut AR was built using Unity or Unreal with graphics pipelines that do fancy tricks like displacement mapping on top of normal mapping. Ultimately all the rules of AR are roughly the same as the rules of game development on modern iOS devices, so that is where I would look to find guidance. Personally I haven’t been building complex 3D scenes, so I don’t regularly run in to the performance limitations. I will say that performance is largely driven by the number of draw calls though, more so than the # of polys. So the first thing you want to do is look at using batched materials so that multiple objects can use only a single draw call. It’s a somewhat in-depth topic, and one that is well covered online if you search around for how to minimize draw calls on iOS. Hope that helps!

  12. Hi – thanks for the tutorial. I’m using xcode v 9.0. Running on 6s simulator iOS11 – target swift 4 and getting this error:

    [Session] Unable to run the session, configuration is not supported on this device:
    2017-08-10 15:53:30.807791-0500 arkit-test-03[5510:370110] [Session] Session did fail with error: Error Domain=com.apple.arkit.error Code=100 “Unsupported configuration.” UserInfo={NSLocalizedDescription=Unsupported configuration., NSLocalizedFailureReason=The provided configuration is not supported on this device.}

    Any idea on what configuration stuff I can check?

    Thanks!

Comment