Core ML for iOS Apps in iOS 11

Core ML, announced at WWDC 2017 is a new set of APIs built by Apple for use with iOS 11 or higher devices. With Core ML, developers can incorporate machine learning models in to their mobile apps, and have the inference accelerated using the Metal APIs. This means the processing of models will be significantly faster than using other systems such as TensorFlow or Caffe2.

Notably during the WWDC keynote, it was mentioned that Core ML will support models such as those coming from Keras or Caffe. So many existing models can be converted to work on-device with the new acceleration.

Core ML supports the following key features:
– Deep neural networks
– Recurrent neural networks
– Convolutional neural networks
– Support vector machines
– Tree ensembles
– Linear models

This is all done with on-device processing, and supports iOS, macOS, watchOS, and tvOS.

Under the Core ML banner is a Model Converter, Natural Language API, and a Vision API.

Once the APIs are made public, I’ll collect the references here for the official Core ML documentation. Follow along with me and let’s start learning!

For now, we know that the APIs will support Face tracking, Face detection, Landmarks, Text detection, Rectangle detections, Barcode detection, Object tracking, and Image registration.

Sign up now and get a set of FREE video tutorials on writing iOS apps coming soon.

Subscribe via RSS

Caffe2 on iOS – Deep Learning Tutorial

Caffe2 in an iOS App Deep Learning Tutorial

At this years’s F8 conference, Facebook’s annual developer event, Facebook announced Caffe2 in collaboration with Nvidia. This framework gives developers yet another tool for building deep learning networks for machine learning. But I am super pumped about this one, because it is specifically designed to operate on mobile devices! So I couldn’t resist but start digging in immediately.

I’m still learning, but I want to share my journey in working with Caffe2. So, in this tutorial I’m going to show you step by step how to take advantage of Caffe2 to start embedding deep learning capabilities in to your iOS apps. Sound interesting? Thought so… let’s roll 🙂

Building Caffe2 for iOS

The first step here is to just get Caffe2 built. Mostly their instructions are adequate so I won’t repeat too much of it here. You can learn how to build Caffe2 for iOS here.

The last step for their iOS install process is to run, but that’s about where they leave you off with the instruction. So from here, let’s take a look at the build artifacts. The core library for Caffe2 on iOS is located inside the caffe2 folder:

  • caffe2/libCaffe2_CPU.a

And in the root folder:

  • libCAFFE2_NNPACK.a

NNPack is sorta like CUDNN for mobile, in that it accelerates the neural network operations for mobile CPUs. PThreadPool is a thread pool library.

Create an Xcode project

Now that the library was built, I created a new iOS app project in Xcode with a single-view template. From here I drag and drop the libCaffe2_CPU.a file in to my project heirarchy along with the other two libs, libCAFFE2_NNPACK.a and libCAFFE2_PTHREADPOOL.a. Select ‘Copy’ when prompted. The file is located at caffe2/build_ios/caffe2/libCaffe2_CPU.a. This pulls a copy of the library in to my project and tells Xcode I want to link against the library. We need to do the same thing with protobuf, which is located in caffe2/build_ios/third_party/protobuf/cmake/libprotobuf.a.

In my case I wanted to also include OpenCV2, which has it’s own requirements for setting up. You can learn about how to install OpenCV2 on their site. The main problem I ran in to with OpenCV2 was figuring out that I needed to create a Prefix.h file, and then in the settings of the project set the Prefix Header file to be MyAppsName/Prefix.h. In my example project I called the project DayMaker, so for me it was DayMaker/Prefix.h. Then I could put the following in the Prefix.h file so that OpenCV2 would get included before any Apple headers:

#ifdef __cplusplus
    #import <opencv2/opencv.hpp>
    #import <opencv2/stitching/detail/blenders.hpp>
    #import <opencv2/stitching/detail/exposure_compensate.hpp>

Prefix Headers for Caffe2 in Xcode

Include the Caffe2 headers

In order to actually use the library, we’ll need to pull in the right headers. Assuming you have a directory structure where your caffe2 files are a level above your project. (I cloned caffe2 in to ~/Code/caffe2 and set up my project in ~/Code/DayMaker.) You’ll need to add the following User Header Search Path in your project settings:


You’ll also need to add the following to “Header Search Paths”


Now you can also try importing some caffe2 C++ headers in order to confirm it’s all working as expected. I created a new Objective-C class to wrap the Caffe2 C++ API around. To follow along, create a new Objective-C class called Caffe2. Then rename the Caffe2.m file it creates to This causes the compiler to see this as Objective-C++ instead of just Objective-C, a requirement for making this all work.

Next, I added some Caffe2 headers to the .mm file. At this point this is my entire file:

#import "caffe2/core/context.h"
#import "caffe2/core/operator.h"
#import "Caffe2.h"
@implementation Caffe2

Next, we need to disable bitcode since the library isn’t built with bitcode support. Ideally we would include bitcode support in the build, but that would involve diving more in to the build process for caffe2, and at this point we’re just trying to get something up an running. So for now, go in to the Xcode project settings and set ‘Enable Bitcode’ to ‘No’.

According to this Github issue a reasonable place to start with a C++ interface to the Caffe2 library is this standalone app. So let’s expand the file to include some of this stuff and see if everything works on-device.

With a few tweaks we can make a class that loads up the caffe2 environment and loads in a set of predict/net files. I’ll pull in the files from Squeezenet on the Model Zoo. Copy these in to the project heirarchy, and we’ll load it up just like any iOS binary asset…

//  Caffe2.m
//  DayMaker
//  Created by Jameson Quave on 4/22/17.
//  Copyright © 2017 Jameson Quave. All rights reserved.
#import "Caffe2.h"
// Caffe2 Headers
#include "caffe2/core/flags.h"
#include "caffe2/core/init.h"
#include "caffe2/core/predictor.h"
#include "caffe2/utils/proto_utils.h"
// OpenCV
#import <opencv2/opencv.hpp>
namespace caffe2 {
    void run(const string& net_path, const string& predict_net_path) {
        caffe2::NetDef init_net, predict_net;
        CAFFE_ENFORCE(ReadProtoFromFile(net_path, &init_net));
        CAFFE_ENFORCE(ReadProtoFromFile(predict_net_path, &predict_net));
        // Can be large due to constant fills
        VLOG(1) << "Init net: " << ProtoDebugString(init_net);
        LOG(INFO) << "Predict net: " << ProtoDebugString(predict_net);
        auto predictor = caffe2::make_unique<Predictor>(init_net, predict_net);
        LOG(INFO) << "Checking that a null forward-pass works";
        Predictor::TensorVector inputVec, outputVec;
        predictor->run(inputVec, &outputVec);
        NSLog(@"outputVec size: %lu", outputVec.size());
        NSLog(@"Done running caffe2");
@implementation Caffe2
- (instancetype) init {
    self = [super init];
    if(self != nil) {
        [self initCaffe];
    return self;
- (void) initCaffe {
    int argc = 0;
    char** argv;
    caffe2::GlobalInit(&argc, &argv);
    NSString *net_path = [NSBundle.mainBundle pathForResource:@"exec_net" ofType:@"pb"];
    NSString *predict_net_path = [NSBundle.mainBundle pathForResource:@"predict_net" ofType:@"pb"];
    caffe2::run([net_path UTF8String], [predict_net_path UTF8String]);
    // This is to allow us to use memory leak checks.

Next, we can just instantiate this from the AppDelegate to test it out… (Note you’ll need to import Caffe2.h in your Bridging Header if you’re using Swift, like me.

#import "Caffe2.h"

In AppDelegate.swift:

func application(_ application: UIApplication, didFinishLaunchingWithOptions launchOptions: [UIApplicationLaunchOptionsKey: Any]?) -> Bool {
    // Instantiate caffe2 wrapper instance
    let caffe2 = Caffe2()
    return true

This for me produced some linker errors from clang:

[F operator.h:469] You might have made a build error: the Caffe2 library does not seem to be linked with whole-static library option. To do so, use -Wl,-force_load (clang) or -Wl,--whole-archive (gcc) to link the Caffe2 library.

Adding -force_load DayMaker/libCaffe2_CPU.a as an additional linker flag corrected this issue, but then it presented an issue not being able to find opencv. The DayMaker part will be your project name, or just whatever folder your libCaffe2_CPU.a file is located in. This will show up as two flags, just make sure theyre in the right order and it should perform the right concatenation of the flags.

Linker flags

Building and running the app crashes immediately with this output:

libc++abi.dylib: terminating with uncaught exception of type caffe2::EnforceNotMet: [enforce fail at conv_op_impl.h:24] X.ndim() == filter.ndim(). 1 vs 4 Error from operator:
input: "data" input: "conv1_w" input: "conv1_b" output: "conv1" type: "Conv" arg { name: "stride" i: 2 } arg { name: "pad" i: 0 } arg { name: "kernel" i: 3 }

Success! I mean, it doesn’t look like success jut yet, but this is an error coming from caffe. The issue here is just that we never set anything for the input. So let’s fix that by providing data from an image.

Caffe2 on an iOS device

Loading up some image data

Here you can add a cat jpg to the project or some similar image to work with, and load it in:

UIImage *image = [UIImage imageNamed:@"cat.jpg"];

I refactored this a bit and moved my logic out in to a predictWithImage method, as well as creating the predictor in a seperate function:

namespace caffe2 {
    void LoadPBFile(NSString *filePath, caffe2::NetDef *net) {
        NSURL *netURL = [NSURL fileURLWithPath:filePath];
        NSData *data = [NSData dataWithContentsOfURL:netURL];
        const void *buffer = [data bytes];
        int len = (int)[data length];
        CAFFE_ENFORCE(net->ParseFromArray(buffer, len));
    Predictor *getPredictor(NSString *init_net_path, NSString *predict_net_path) {
        caffe2::NetDef init_net, predict_net;
        LoadPBFile(init_net_path, &init_net);
        LoadPBFile(predict_net_path, &predict_net);
        auto predictor = new caffe2::Predictor(init_net, predict_net);
        return predictor;

The predictWithImage method is using openCV to get the GBR data from the image, then I’m loading that in to Caffe2 as the inputVector. Most of the work here is actually done in OpenCV with the cvtColor line…

- (NSString*)predictWithImage: (UIImage *)image predictor:(caffe2::Predictor *)predictor {
    cv::Mat src_img, bgr_img;
    UIImageToMat(image, src_img);
    // needs to convert to BGR because the image loaded from UIImage is in RGBA
    cv::cvtColor(src_img, bgr_img, CV_RGBA2BGR);
    size_t height = CGImageGetHeight(image.CGImage);
    size_t width = CGImageGetWidth(image.CGImage);
    caffe2::TensorCPU input;
    // Reasonable dimensions to feed the predictor.
    const int predHeight = 256;
    const int predWidth = 256;
    const int crops = 1;
    const int channels = 3;
    const int size = predHeight * predWidth;
    const float hscale = ((float)height) / predHeight;
    const float wscale = ((float)width) / predWidth;
    const float scale = std::min(hscale, wscale);
    std::vector<float> inputPlanar(crops * channels * predHeight * predWidth);
    // Scale down the input to a reasonable predictor size.
    for (auto i = 0; i < predHeight; ++i) {
        const int _i = (int) (scale * i);
        for (auto j = 0; j < predWidth; ++j) {
            const int _j = (int) (scale * j);
            inputPlanar[i * predWidth + j + 0 * size] = (float)[(_i * width + _j) * 3 + 0];
            inputPlanar[i * predWidth + j + 1 * size] = (float)[(_i * width + _j) * 3 + 1];
            inputPlanar[i * predWidth + j + 2 * size] = (float)[(_i * width + _j) * 3 + 2];
    input.Resize(std::vector<int>({crops, channels, predHeight, predWidth}));
    caffe2::Predictor::TensorVector input_vec{&input};
    caffe2::Predictor::TensorVector output_vec;
    predictor->run(input_vec, &output_vec);
    float max_value = 0;
    int best_match_index = -1;
    for (auto output : output_vec) {
        for (auto i = 0; i < output->size(); ++i) {
            float val = output->template data<float>()[i];
            if(val > 0.001) {
                printf("%i: %s : %f\n", i, imagenet_classes[i], val);
                if(val>max_value) {
                    max_value = val;
                    best_match_index = i;
    return [NSString stringWithUTF8String: imagenet_classes[best_match_index]];

The imagenet_classes are defined in a new file, classes.h. It’s just a copy from the Android example repo here.

Most of this logic was pulled and modified from bwasti’s github repo for the Android example.

With these changes I was able to simplify the initCaffe method as well:

- (void) initCaffe {
    NSString *init_net_path = [NSBundle.mainBundle pathForResource:@"exec_net" ofType:@"pb"];
    NSString *predict_net_path = [NSBundle.mainBundle pathForResource:@"predict_net" ofType:@"pb"];
    caffe2::Predictor *predictor = caffe2::getPredictor(init_net_path, predict_net_path);
    UIImage *image = [UIImage imageNamed:@"cat.jpg"];
    NSString *label = [self predictWithImage:image predictor:predictor];
    NSLog(@"Identified: %@", label);
    // This is to allow us to use memory leak checks.

So you’ll notice I’m pulling in the cat.jpg here. I used this cat pic:


The output when running on iPhone 7:

Identified: tabby, tabby cat

Hooray! It works on a device!

I’m going to keep working on this and publishing what I learn. If that sounds like something you want to follow along with you can get new posts in your email, just join my mobile development newsletter. I’ll never spam you, just keep you up-to-date with deep learning and my own work on the topic.

Thanks for reading! Leave a comment or contact me if you have any feedback 🙂

Side-note: Compiling on Mac OS Sierra with CUDA

When compiling for Sierra as a target (not the iOS build script, but just running make) I ran in to a problem in protobuf that is related to this issue. This will only be a problem if you are building against CUDA. I suppose it’s somewhat unusual to do so because most Mac computers do not have NVIDIA chips in them, but in my case I have a 2013 MBP with an NVIDIA chip that I can use CUDA with.

To resolve the problem in the most hacky way possible, I applied the changes found in that issue pull. Just updating protobuf to the latest version by building from source would probably also work… but this just seemed faster. I open up my own version of this file in /usr/local/Cellar/protobuf/3.2.0_1/include/google/protobuf/stubs/atomicops.h and just manually commented out lines 198 through 205:

// Apple.
#if __has_feature(cxx_atomic) || _GNUC_VER >= 407
#include <google/protobuf/stubs/atomicops_internals_generic_c11_atomic.h>
#else  // __has_feature(cxx_atomic) || _GNUC_VER >= 407
#include <google/protobuf/stubs/atomicops_internals_macosx.h>
#endif  // __has_feature(cxx_atomic) || _GNUC_VER >= 407

I’m not sure what the implications of this are, but it seems to be what they did in the official repo, so it must not do much harm. With this change I’m able to make the Caffe2 project with CUDA support enabled. In the official version of protobuf used by tensorflow, you can see this bit is actually just removed, so it seems to be the right thing to do until protobuf v3.2.1 is released, where this is fixed using the same approach.

Sign up now and get a set of FREE video tutorials on writing iOS apps coming soon.

Subscribe via RSS