Apples to apples

When Craig Federighi arrived at his presentation slide about Objective-C during this year’s WWDC keynote everyone in the room seemed puzzled, curious, and maybe even a bit uneasy. What was happening? As he continued, he considered what Objective-C would be like without the C, and the room abruptly filled with rumblings and whispers ^[1] as developers in the audience confided in those around them. If you had been following the discussions in our community about the state of Objective-C (and why we need to replace it) during the previous months, you could only have imagined one thing: Objective-C was no more — at least not as we knew it.

And then Federighi said, let there be Swift; and there was Swift.

— WWDC 2014, 1:44:48

The third floor of Moscone West erupted with applause as if we had traveled back in time to Steve Jobs’ 2007 announcement of the iPhone: “An iPod, a phone, and an Internet communicator”.

As the keynote continued, we were assured safety, optimizations, clarity, modernity, and speed. But, as some have already investigated, Swift may not be as swift as promised. However, Swift is still in beta (along with Xcode 6, iOS 8, and OS X 10.10), so we will undoubtedly see many improvements and changes in the coming months.

As a fun and interesting code kata, I decided to port my objc-sorts project on GitHub to Swift. Behold, swift-sorts. These projects are collections of sorting algorithms implemented in Objective-C and Swift, respectively. I completed a rough version of the Swift project during the week of WWDC and have since refined both. I also shared the results below with Apple engineers in the Swift Labs during WWDC, but more on that later.

Setup

• Code: Swift Sorts and Objective-C Sorts
• Software: OS X Mavericks 10.9.3 10.9.4, Xcode6-beta4 beta2 WWDC seed
• Hardware: 2008 unibody MacBook Pro, 2.4 Ghz Intel Core 2 Duo, 8 GB 1067 MHz DDR3 memory ^[2]

Each project is a command line app with a debug, release, and unit-test scheme. Build and run, then watch the console for output.

The benchmarks consist of T trials, which are averaged at the end to obtain the average execution time for each algorithm. Each trial begins by generating an array of N random integers in the range [0, UINT32_MAX). Then, each sorting algorithm is passed a copy of this initial array to sort. The current time is logged before and after each sort and the difference between the two yields the execution time for the algorithm for the current trial.

These two programs were carefully crafted to be a true apples-to-apples comparison. All of the algorithms, as well as main.swift and main.m, are implemented as similarly as possible, bounded only by the confines and paradigms of the languages themselves. In Objective-C, NSArray and NSNumber are used intentionally as the counterparts to Swift’s Array and Int. The APIs are language-specific too, for example exchangeObjectAtIndex: withObjectAtIndex: versus swap().

The following were used for the standard library sorts:

// Swift
var arr: [Int] = // some array
let newArr = sorted(arr);

// Objective-C
NSMutableArray *arr = // some array
[arr sortUsingComparator:^NSComparisonResult(NSNumber *n1, NSNumber *n2) {
    return [n1 compare:n2];
}];

Previous Swift std lib sort implementation here.

Results

Below are the results of running each program over 10 trials with 10,000 integers. The build configuration settings are noted for each run and the execution times are displayed in seconds. The average case runtime complexity for each algorithm is also noted. I realize that 10,000 is relatively small, but you’ll see that Swift was taking quite a long time.

Table 1

Table 2

According to the Apple engineers that I spoke with, -O3 in Objective-C is essentially the equivalent to -O in Swift.

Table 3

Note that -O is the standard optimization level for Swift and -Ofast, though faster, removes all safety features (array bounds-checking, integer overflow checking, etc.) from Swift. In other words, do not ship an entire app compiled with -Ofast. More on that below.

There are a few notable discoveries here:

1. Rather shockingly, debug is incredibly slow in Swift but improves dramatically with compiler flags. The difference in performance between no optimizations and -Ofast in Swift is stark. On the other hand, Objective-C sees relatively minor benefits.

2. At the standard optimization level (see Table 2), the two languages begin to perform more similarly. Objective-C is still noticeably faster though. Std lib sort is 6.5x faster. Quick sort is 7.0x faster. Heap sort is 10.4x faster. Insertion sort is 16.0x faster. Selection sort is 2.47x faster.

3. Only with -Ofast do we begin to experience the swiftness of Swift, and even then the standard library sort in Objective-C is almost twice as fast (1.84x). However, when comparing Swift to Swift the discrepancies are enormous. Swift performs orders of magnitude better than it did without optimizations and puts Objective-C to shame with quick sort, heap sort, insertion sort, and selection sort (see Table 3).

4. We all know that selection sort and insertion sort are not particularly optimal algorithms, and Swift does a good job to emphasize this (when not using -Ofast, see Table 1 and Table 2). But why are these two so terrible in Swift? Especially insertion sort — in debug Objective-C is 308.0x faster. I’m still puzzled by this. These two sorting algorithms are not complex, but they stand apart from the other sorts in the following ways: selection sort has nested for-loops and insertion sort has a while-loop nested in a for-loop. Perhaps Swift is having trouble optimizing these? Is this a bug?

5. My mundane quick sort implementation is faster than the standard library sort for both languages. Typically, these library methods would utilize multiple sorting algorithms that are guided by a set of heuristics that help determine the best algorithm to use based on the dataset. I suspect that we would see the standard library sorts perform best with larger datasets and/or with complex objects.

According to the benchmarks presented during the keynote (1:45:30), we should (probably?) be seeing different results here. Federighi noted that for complex object sort, Objective-C performed at 2.8x and Swift performed at 3.9x, using Python as the baseline (1.0x). It is not clear at this time how these benchmarks were achieved. What were the build and optimization settings? What is a “complex object”? In any case, surely Swift should be able to sort integers just as well as “complex objects”, right?

Swift Labs at WWDC

The Apple engineers hanging out in the Swift Labs at WWDC were interested in these benchmarks and were somewhat surprised to see them. Unfortunately, the engineers that I spoke with did not have an explanation for why we were seeing these results. We filed Radar #17201160, noting most of the points above.

Additionally, I asked what the best practices are regarding using -Ofast. They recommended the following approach: (1) profile your app to find out where it is slow, (2) extract this slow code into a separate module/framework, (3) thoroughly test this module, and then (4) compile the module using -Ofast and link it to your app. Remember, this removes all safety features from Swift.

Moving forward

The results above seem to indicate that Apple has not (yet) followed through on their promises of speed and safety — at least in the sense that these features can be mutually inclusive. Again, it is still early. Hopefully these benchmarks will improve as Swift nears a 1.0 release. I plan on updating this post or writing follow-up posts as Apple releases updates for Swift and Xcode6-beta.

As Brent Simmons said, Objective-C used to be considered slow compared to plain C, but it is not slow compared to Java or Python. I am not sure if the reaction to these results should be we have faster hardware, so a slower language is fine, or nothing will ever be as fast as C, or somewhere in-between. But after completing these two projects, I do know this: Swift is a pleasure to write and read. Many things came easier and more naturally in Swift, and Playgrounds are pure gold. Swift has a lot of potential. Let’s hope this is the next step that we have all been waiting for, and not another Copland.

Futher reading

• The Official Apple Swift Blog
• Swift? from Splasm Software
• The Foundation Collection Classes by Peter Steinberger, objc.io issue #7