iOS Metal Tutorial with Swift Part 5: Switching to MetalKit

Welcome the 5th part of our iOS Metal tutorial series!

In the 1st part, you learned how to get started with Metal and render a simple 2D triangle.

In the 2nd part, you learned how to setup a series of transformations to move from a triangle to a full 3D cube.

In the 3rd part, you learned how to add a texture to the cube.

In the 4th part, you learned how to add light to the scene.

In this 5th part, you’ll learn how to update your app to take advantage of the MetalKit framework. In addition, you’ll also be updating the app to use the SIMD (pronounced “sim-dee”) framework for 3D-related math.

To get the most out of this tutorial, you should have a basic understanding of 3D graphics with either Metal or OpenGL. If this is the first time you’re learning about Metal, you should go back and complete the previous parts of the series.

Without further ado, let’s get into it!

Getting Started

Start by downloading the starter project for this tutorial. This tutorial starts essentially where you ended with the previous tutorial. Also, make sure to download the additional resources required to complete this tutorial.

Do a quick build and run, just to make sure the starter project works. The result should resemble something like this:

Getting Started with MetalKit

Apple presented MetalKit at WWDC 2015 as a gateway to Metal. The framework gives you utilities that reduce the amount of boilerplate code you have to write in order to get an app running on Metal.

MetalKit provides three major pieces of functionality:

• Texture loading: Allows you to easily load image assets into Metal textures using a MTKTextureLoader.
• View management: Reduces the amount of code you need to get Metal to render something on-screen via MTKView.
• Model I/O integration: Allows you to efficiently load model assets into Metal buffers and manage mesh data using built-in containers.

In this tutorial, you’ll be focusing on texture loading and view management. Model I/O integration will be the subject of a future part in the series.

Switching to SIMD

The SIMD framework provides many common data types and functions that help when dealing with vector and matrix math. When this tutorial series first started, there were issues in Swift that prevented you from using C data types like the ones found in the SIMD and GLKit frameworks.

To overcome those issues, you ended up having to write an Objective-C wrapper to represent a 4×4 matrix and perform various GLKMath operations. It served its purpose, but it didn’t look that nice, because you needed to use a bridging header to bring it to Swift.

Thankfully, the problem was fixed in Swift 2 and you can now take full advantage of the SIMD framework, which means you can now remove all the remaining Objective-C code from the project!

Deleting the Objective-C Wrapper

Open the project and take a look at the Matrix4 class. You’re currently using this class to store your matrix data; it also provides helper methods for a couple of matrix math operations.

To get started, select Matrix.h, Matrix.m and HelloMetal-Bridging-Header.h and delete them from your project. This will understandably cause a lot of errors to show up in your project. Fear not, because you’ll be working on getting the app back to a runnable state.

Even though you deleted the bridging header file, you still need to unlink it from your project settings. Go to Build Settings and search for bridging to find the Objective-C Bridging Header setting. Highlight it and press the Delete key to clear the setting:

At this point, Objective-C is no longer part of this project. Hooray!

Replacing Matrix4 With a SIMD Data Type

Next, you’re going to replace all of your Matrix4 instances with float4x4 instances in your app. A float4x4 is a SIMD data type representing a 4×4 matrix of floats.

Since you’ll replace code throughout your entire project, you can perform a rare case of blind search and replace. Open the find navigator, click the Find text and select Replace from the dropdown. Put in Matrix4 in the search field and float4x4 in the replace field. To enable the Replace All button, you actually have to perform the search first, so press the enter-key with the search field selected:

Click the Replace All button to replace all occurrences of Matrix4 with float4x4.

Now, since float4x4 is part of the SIMD library, you’ll need to import it everywhere you use float4x4.

At the top of these four files:

• BufferProvider.swift
• MetalViewController.swift
• MySceneViewController.swift
• Node.swift

Add the following line:

import simd

There are still lots of errors, because float4x4 doesn’t have some of the methods that Matrix4 had; specifically, methods for applying transformations, creating a projection matrix and returning the number of elements in matrix.

Fixing the Remaining Errors

To fix the remaining errors, find the float4x4+Extensions.swift file under the additional resources you downloaded and add it to your project. This file contains an extension to float4x4, which adds a Swift version of those helper methods.

You’ll notice there are a few errors left; don’t panic, you’ll take care of them next.

Fixing Issues in BufferProvider.swift

Open BufferProvider.swift and under nextUniformsBuffer(_:modelViewMatrix:light:) find the following code:

memcpy(bufferPointer, modelViewMatrix.raw(), MemoryLayout<Float>.size*float4x4.numberOfElements())
    memcpy(bufferPointer + MemoryLayout<Float>.size*float4x4.numberOfElements(), projectionMatrix.raw(), MemoryLayout<Float>.size*float4x4.numberOfElements())
    memcpy(bufferPointer + 2*MemoryLayout<Float>.size*float4x4.numberOfElements(), light.raw(), Light.size())

Replace that code with:

// 1
var projectionMatrix = projectionMatrix
var modelViewMatrix = modelViewMatrix
 
// 2
memcpy(bufferPointer, &modelViewMatrix, MemoryLayout<Float>.size*float4x4.numberOfElements())
memcpy(bufferPointer + MemoryLayout<Float>.size*float4x4.numberOfElements(), &projectionMatrix, MemoryLayout<Float>.size*float4x4.numberOfElements())
memcpy(bufferPointer + 2*MemoryLayout<Float>.size*float4x4.numberOfElements(), light.raw(), Light.size())

The two main differences between these blocks of code are:

1. Matrices are now Swift structs and you need to mark them as mutable when you pass them by reference.
2. To pass matrix data to memcpy, you simply need to get a pointer to it.

Fixing Issues in Node.swift

Now, open Node.swift and find the following code under render(_ commandQueue: MTLCommandQueue, pipelineState: MTLRenderPipelineState, drawable: CAMetalDrawable, parentModelViewMatrix: float4x4, projectionMatrix: float4x4, clearColor: MTLClearColor?):

let nodeModelMatrix = self.modelMatrix()

Replace that code with:

var nodeModelMatrix = self.modelMatrix()

Under modelMatrix(), find:

let matrix = float4x4()

And replace that code with:

var matrix = float4x4()

Also, remove the question marks and the exclamation mark right below it.

The various helper methods from the float4x4 extension are modifying the struct, therefore the variables must be declared as var instead of let.

Your project should now be error free. Time for another build and run. The result should look exactly the same as before, which is to be expected!

The main difference is that you’ve now removed all the Objective-C code, and you’re now using the new SIMD data type float4x4 instead of that old Matrix4.

Exploring float4x4+Extensions.swift

Open float4x4+Extensions.swift and take a look at the methods. As you can see, this file still calls math functions from GLKMath under the hood in order to use well-written and well-tested code instead of reinventing the wheel.

This change might not seem worth it, but it’s important to use SIMD’s float4x4 because it’s a standardized solution for 3D graphics and it will allow easier integration with third-party code.

At the end of the day, it doesn’t really matter how the matrix math is done. You can use GLKit, a 3rd party extension or perhaps Apple will release their own solution down the road someday. The important thing is to have your matrices represented in the same format as the rest of them, out there in the wild! :]

MetalKit Texture Loading

Before you take a look at the functionality that MetalKit offers, open MetalTexture.swift and review how it currently loads the texture in loadTexture(_ device: MTLDevice, commandQ: MTLCommandQueue, flip: Bool):

1. First, you load the image from a file.
2. Next, you extract the pixel data from that image into raw bytes.
3. Then, you ask the MTLDevice to create an empty texture.
4. Finally, you copy the bytes data into that empty texture.

Lucky for you, MetalKit provides a great API that helps you with loading textures. Your main interaction with it will be through the MTKTextureLoader class.

You might be asking, “How much code can this API save me from writing?” The answer is pretty much everything in MetalTexture!

To switch texture loading to MetalKit, delete MetalTexture.swift from your project. Again, this will cause some errors; you’ll fix these shortly.

Fixing Issues in Cube.swift

First, open Cube.swift and find the following at the top of the file:

import Metal

Then, replace it with this:

import MetalKit

Next, add a parameter to the initializer. Find this line of code:

init(device: MTLDevice, commandQ: MTLCommandQueue) {

Then, replace it with the following:

init(device: MTLDevice, commandQ: MTLCommandQueue, textureLoader :MTKTextureLoader) {

Scroll down to the end of this initializer and find the following code:

let texture = MetalTexture(resourceName: "cube", ext: "png", mipmaped: true)
texture.loadTexture(device, commandQ: commandQ, flip: true)
 
super.init(name: "Cube", vertices: verticesArray, device: device, texture: texture.texture)

Now, replace it with the following:

let path = Bundle.main.path(forResource: "cube", ofType: "png")!
let data = NSData(contentsOfFile: path) as! Data
let texture = try! textureLoader.newTexture(with: data, options: [MTKTextureLoaderOptionSRGB : (false as NSNumber)])
 
super.init(name: "Cube", vertices: verticesArray, device: device, texture: texture)

Here’s a recap of what you’ve just done:

• You added a MTKTextureLoader parameter to the cube’s initializer.
• Then, after converting the image into NSData, you used newTextureWithData(_:options:) on the textureLoader to directly load the image into a MTLTexture.

Fixing Issues in MetalViewController.swift

Now you need to pass a texture loader to the cube when you create it.

Open MetalViewController.swift and find the following at the top of the file:

import Metal

Replace it with this:

import MetalKit

Next, add the following new property to MetalViewController:

var textureLoader: MTKTextureLoader! = nil

Finally, initialize this property by adding the line below, right after the point where you create the default device in viewDidLoad():

textureLoader = MTKTextureLoader(device: device)

Fixing Issues in MySceneViewController.swift

Now that you’ve got a default instance of a texture loader, you need to update MySceneViewController.swift to pass it to the cube.

Go to viewDidLoad() in MySceneViewController.swift and find this code:

objectToDraw = Cube(device: device, commandQ:commandQueue)

Now, replace the call to Cube() with this:

objectToDraw = Cube(device: device, commandQ: commandQueue, textureLoader: textureLoader)

Build and run the app, and you should have the exact same result as before. Again, this is the expected result.

Although the result didn’t change, you’re making positive changes to your app, under the hood. You’re now using MTKTextureLoader from MetalKit to load a texture. Compared to before, where you had to write a whole bunch of code yourself to achieve the same result.

Switching to MTKView

The idea behind MTKView is simple. In iOS, it’s a subclass of UIView, and it allows you to quickly connect a view to the output of a render pass. A MTKView will help you do the following:

• Configure the CAMetalLayer of the view.
• Control the timing of the draw calls.
• Quickly manage a MTLRenderPassDescriptor.
• Handle view resizes easily.

To use a MTKView, you can either implement a delegate for it or you can subclass it to provide the draw updates for the view. For this tutorial, you’ll go with the first option.

First, you need to change the main view’s class to be a MTKView.

Open Main.storyboard, select the view controller view, then change the class to MTKView in the Identity Inspector:

An instance of MTKView, by default, will ask for redraws periodically. So you can remove all the code that sets up a CADisplayLink.

Removing Redundant Code From MetalViewController.swift

Open MetalViewController.swift, scroll to the end of viewDidLoad() and remove the following:

timer = CADisplayLink(target: self, selector: #selector(MetalViewController.newFrame(_:)))
timer.add(to: RunLoop.main, forMode: RunLoopMode.defaultRunLoopMode)

After that, you can also remove both newFrame(_:) and gameloop(_:) functions.

Now you need to remove the code that sets up the Metal layer since the MTKView will handle that for you.

Again, in viewDidLoad(), remove the following:

metalLayer = CAMetalLayer()
metalLayer.device = device
metalLayer.pixelFormat = .bgra8Unorm
metalLayer.framebufferOnly = true
view.layer.addSublayer(metalLayer)

Adding the MTKViewDelegate Protocol

To make your MetalViewController responsible for the draw updates, it must conform to MTKViewDelegate.

Add this extension to the end of the file to implement the protocol methods:

// MARK: - MTKViewDelegate
extension MetalViewController: MTKViewDelegate {
 
  // 1
  func mtkView(_ view: MTKView, drawableSizeWillChange size: CGSize) {
    projectionMatrix = float4x4.makePerspectiveViewAngle(float4x4.degrees(toRad: 85.0),
      aspectRatio: Float(self.view.bounds.size.width / self.view.bounds.size.height),
      nearZ: 0.01, farZ: 100.0)
  }
 
  // 2
  func draw(in view: MTKView) {
    render(view.currentDrawable)
  }
 
}

Taking a look at the two protocol methods:

1. mtkView(_ view: MTKView, drawableSizeWillChange size: CGSize) runs whenever the MTKView resizes. Here, you reset the projectionMatrix based on the new size.
2. draw(in view: MTKView) is called when you need to draw a new frame to the view.

Since you’ve changed the way you call render(), you need to update the method. Find the following code:

func render() {
  if let drawable = metalLayer.nextDrawable() {
    self.metalViewControllerDelegate?.renderObjects(drawable)
  }
}

Then, replace it with this:

func render(_ drawable: CAMetalDrawable?) {
  guard let drawable = drawable else { return }
  self.metalViewControllerDelegate?.renderObjects(drawable)
}

Now that you’re responding to size changes using the delegate, you can remove the viewDidLayoutSubviews() function too.

To connect the view delegate to the view controller, add the following code to the MetalViewController class after the list of properties:

@IBOutlet weak var mtkView: MTKView! {
  didSet {
    mtkView.delegate = self
    mtkView.preferredFramesPerSecond = 60
    mtkView.clearColor = MTLClearColor(red: 0.0, green: 0.0, blue: 0.0, alpha: 1.0)
  }
}

This is a property observer and will connect the view’s delegate to the view controller whenever the outlet is set.

To make the actual connection from the storyboard to the outlet, you need to open up Main.storyboard and have MetalViewController.swift open in the assistant editor. Drag from the unconnected outlet (denoted by the empty circle in the gutter) to the view in the storyboard. When you release, they should be connected:

All that’s left to do is set the device property of the MTKView. To do that, first find these two lines in viewDidLoad():

device = MTLCreateSystemDefaultDevice()
textureLoader = MTKTextureLoader(device: device)

Below those two lines, add the following line:

mtkView.device = device

Finally, remove the following properties from the top of the class:

var metalLayer: CAMetalLayer! = nil
var timer: CADisplayLink! = nil
var lastFrameTimestamp: CFTimeInterval = 0.0

Also, find the following lines of code and remove them:

metalLayer = CAMetalLayer()
metalLayer.device = device
metalLayer.pixelFormat = .bgra8Unorm
metalLayer.framebufferOnly = true
view.layer.addSublayer(metalLayer)

You’re all done, build and run your app!

The cube still looks exactly like before, but now it’s running on MetalKit! Huzzah!

OK, you might feel a little disappointed because it looks like you ended up right where you started, and that nothing’s changed, right? But don’t fret. Just remember this ancient Chinese proverb: “The journey to Model I/O starts with the single step of porting to MetalKit.” Er, or something like that! :]

Where to Go From Here?

Here is the final example project from this iOS Metal tutorial.

Take a moment to review what you’ve done:

• You switched to using SIMD’s float4x4.
• You removed all Objective-C code from the project.
• You loaded a texture using MTKTextureLoader.
• You integrated MTKView into the project, removing lots of boilerplate code.
• And best of all, you didn’t break anything!

You made it! You totally deserve some rest. :]

Feel like you’re up for more Metal? We’re looking to create more Metal tutorials in the future, but in the meantime, be sure to check out some of the great resources below:

• Apple’s Metal for Developers page, which has tons of links to documentation, videos and sample code
• Apple’s Metal Programming Guide
• Apple’s Metal Shading Language Guide
• The Metal videos from WWDC 2014
• The Metal videos from WWDC 2015
• The Metal videos from WWDC 2016
• MetalByExample.com

Also, tune into the OpenGL ES video tutorials on this site and learn as Ray explains — in depth — how many of these similar concepts work in OpenGL ES.

Thank you for joining me on this tour through Metal. As you can see, it’s a powerful technology that’s relatively easy to implement once you understand how it works.

If you have any questions, comments or Metal discoveries to share, please leave them in the comments below!

The post iOS Metal Tutorial with Swift Part 5: Switching to MetalKit appeared first on Ray Wenderlich.