ARKit Tutorial in Swift 4 for Xcode 9

In this tutorial I’m going to show you how to work with ARKit, the new Framework from Apple that allows us to easily create Augmented Reality experiences in our iOS apps. The first thing to know about ARKit is that it can be used in three major ways:

• Inside of Xcode writing Swift or Obj-C Code
• Inside of the Unity game engine
• Inside of the Unreal game engine

In this tutorial, we’ll be working with the first option.

We will be writing the app shown above in Swift 4 and Xcode 9 Beta in this tutorial. If you are a game developer and comfortable working with Unity or Unreal, this tutorial is not intended for you. Next week we look at the Happy Wheels unblocked game and the many joys of that little gem. Till then I recommend following Unity and Epic’s documentation respectively if you are working in one of those environments. However, if you want to learn how ARKit works under the hood, this still may be useful in understanding what’s happening when you work with the plugins provided. At the end of the day, your game engine is still wrapping the native Swift or Obj-C calls to ARKit.

Getting started

First off, create a new Xcode project and use the Augmented Reality template. (Xcode 9 Beta and above only)

From here you can run the app immediately on your device and you’ll see a spaceship floating in space. As you walk around with the phone, it should track your movements and simulate the AR effect with the ship.

From here, we’re going to replace this model with a custom one we download. Then, I’ll go over how to use the ARKit hitscan and anchor features to place this custom model on real-world objects.

If you’re already familiar with SceneKit and 3D models, I recommend skipping to the Section 2

Finding a model to work with

I like to use TurboSquid to get models to play around with when I’m prototyping games, and working with AR will be similar. So you can find free (or cheap) 3D models over here. One thing to be aware of though is that some models here are meant for non-realtime rendering, so the poly counts may be high. Any time you are planning to grab a model from TurboSquid, always check that the poly count is not too high. Anything over 10k is pretty high poly for a mobile app. As another reference point, very high end games designed to be played on cutting edge gaming PCs generally have between 50k and 150k polygons for a main character model. If you see a poly count that high it should be for an extremely detailed model, and would ideally be one of very few items in the scene. What you really want for mobile is something low poly, and if it’s a detailed model you should expect a normal map to be included. A normal map will create the illusion that there are more polys than there actually are, without a significant performance impact.

SceneKit has support for DAE models. (If you have another file format model that want to work with, I recommend getting a copy of blender and converting your models to DAE by importing and then exporting as DAE within the application.)

Turbosquid has a filter option that does not include DAE, however it works fine to just type in “DAE”. Using this filter will help you to skip the process of converting models, and instead allow us to focus on code. As a shortcut, you can use this direct link to find DAE models that are free and less than 10k polys. Every model on this list should be appropriate for use in SceneKit.

I like this tree model. We could create an AR experience where users can “plant” trees in AR and walk around through them. You can download the tree model here after signing up for a free account.

Import the DAE model in to the Xcode Project

So download the DAE version of the model and then you can drag the file in to the Xcode navigator in to your art.scnassets folder that was created from the AR app template. Once you have added the asset to your Xcode project, you should be able to preview it within Xcode hold click and drag around to rotate around your mesh. If you don’t see anything in this step there may be an issue with your model.

Section 2

Load the model in to your scene

Now that the model is in your project you can open up the ViewController.swift file created by the template and swap out the spaceship for your model. Find this line of code:

 

Change it to point to your downloaded model. In my case it’s Lowpoly_tree_sample.dae.

 

When running the app with this change you will not be able to see the model because it is much too large, and not positioned in front of the camera. An easy way to check the size of a model is to cut & paste the spaceship model from the default scn file in to the dae file using the SceneKit editor in Xcode. This will tell you the relative sizes, and in this case the tree is around 100 times too large.

Select the tree model in the SceneKit editor view by clicking it, and then on the right-hand side pane select the Node inspector. This is the tab with the cube icon. Here we can set the x, y, and z scale to 0.01, reducing the size of the tree by 99%. You can also set the scale in code, as well as position and rotation.

You may notice that in the default position, the tree is not visible. This is because it’s origin is at position 0, 0, 0, and it’s invisible because the camera is actually inside of the tree at this location. If you start up the AR app on a device and walk around a bit, when you turn around you’ll see the tree. But let’s just move it out some using some code. What we’ll do is change the position to be 1 unit in front of the camera in it’s default location. In order to do this we’ll need to find the tree as a node within the scene. Going back to the SceneKit editor (select the DAE file within Xcode), we can click on the tree model itself, and then again in the Node inspector there should be a name. This is the name of the mesh selected, in this case Tree_lp_11. If you’re using a different model the name may be different, or empty. Fortunately we can just type in our own name as needed.

Now that we know the name of the node, we can access it within our code.

 

The second line above does a search of the child nodes of the scene object created from the DAE file we imported, and returns a node with the name specified. Since 3D models can have deeply nested nodes, it’s often useful to recursively search through the nested heirarchy to find the mesh object, so we opt to search recursively, even though it is not neccessary for this particular model.

From here, we can simply reposition the node by moving it forward a bit. The means going in a negative direction in the z axis, as the default camera faces down the negative Z. Or in other words it’s looking at the point (0, 0, -Inf).

So to make the tree visible, let’s move it back 1 unit in the z direction. The easiest way to do this is just set the new z to -1 on the position object.

 

This will work, but in practice what is more common is to create a new position object from scratch and assign that as the new position. This is based on taste, but it’s a common approach to avoid mutating positions of 3D objects, and instead to replace them. To do this we need to construct a SCNVector3 object with the position (0,0,-1). This code has the exact same effect, but is a better practice:

 

In order to modify the treeNode later, let’s keep an instance reference to it. Outside of any functions, but inside the ViewController class add an optional reference to treeNode:

 

In the next step we’re going to want a reference to the treeNode again, so rather than doing the lookup every time, it is useful to cache the reference to the treeNode. To do this, I’ll modify the childNode call we just added inside of viewDidLoad so that it sets this to an instance variable treeNode as opposed to just the local treeNode variable:

 

Although AR is not supported in the simulator, SceneKit is, and this is just a SceneKit model. So if you run the simulator you’ll find you can now see the tree model with a black background.

Using HitTest

Next, let’s modify the app so that when the user taps, it will place a new copy of the tree model to wherever he tapped. This is a very complex operation because tapping on the 2D screen needs to go through a projection matrix based on the estimated shape of the 3D scene it’s seeing through the camera. ARKit makes this complicated process fairly simple by providing the HitTest api.

So first, we implement an override for touchesBegan. This is called any time the user taps on the screen. From this we can retrieve the first touch and perform a hitTest in the ARScene. We’ll look for results that are anchors. These are basically points in space ARKit has identified and is tracking. In other words it’s a surface of some real-world object.

Once we get a result from the hitTest, we can position the treeNode to the same location as the anchor. The hit test comes back with a 4×4 matrix containing the scale, rotation, and position data. The 4th row of this matrix is the position. We can reconstruct the position using this row by accessing m41, m42, and m43 as x, y, and z respectively. Setting the position to a new SCNVector3 object with these coordinates should move the tree to that location.

 

Give this a try and you’ll find you can teleport the tree to locations in the real-world when you tap!

Xcode 9 Beta 1 vs Xcode 9 Beta 2

Note: Some users have complained that the call SCNMatrix4 function call causes a compiler error. This seems to be caused by users on Xcode 9 Beta 1. The new SCNMatrix4 initializer was added in Xcode 9 Beta 2. If you run in to this problem because you are still on Beta 1, just replace the function SCNMatrix4 with the function SCNMatrix4FromMat4 and this should fix the issue.

Similarly, we can make clones of the tree as well so we can plant our forest 🙂

 

From here you can imagine some of the interesting things that could be done by adding interactions, animations, etc to the scene. To proceed from here with your project what you should learn is how to work with SceneKit, as those fundamentals will apply to most of your ARKit based apps.

You can find the complete source code to this tutorial on Gumroad.

Questions? Comments? Hate mail? Let me know what you thought of this tutorial on my contact page. I’ll be making the video version of this tutorial soon, so check back or subscribe on my YouTube channel to keep up to date.