A common problem when developing UI tests for iOS (and tests in general) is controlling the state before running each test. Having a well known and fixed environment to run tests is paramount.

If a test fails sporadically due to an external event such as side-effects from other tests or networking issues, you’ll not only lose trust on the outcome of that test but also waste your time investigating the failure. Carefully composing and managing this state — called a fixture — is essential for a robust and effective test suite.

When developing UI tests for Secrets for iOS the difficulty in managing these fixtures became apparent. To run these tests, Xcode installs another app on your device/simulator to run your test code. This means your UI tests not only run in a separate process but also in another sandbox.

A quick web search will find you a couple of different solutions to this common problem:

1. Uninstalling the app from the simulator between tests by placing the code on the test tearDown() method that will go to the SpringBoard, long press on your app icon and uninstall it.
2. Resetting the application state by passing a —Reset launch argument and handling it on the app itself at startup.

Option 1 adds more running time to every test, which for large test suits can be really noticeable. Option 2 avoids this problem and achieves a similar effect, but both of these solutions only allow you to reset the application to a clean state. Similar to when the app is just installed. For a thorough test suite, running tests in many different states should be easy.

You could adapt option 2 and pass a launch argument that states which fixture the app should use, for example:-SetupFixture <fixture1|fixture2>. This works but implies that you’ll have to bundle those fixtures in your main app. But it’s hard to only include these fixtures when testing and making sure they don’t ever spill into the production build of your app.

Ideally, we should be able to include the fixture on the test bundle and pass it to the app being tested. But because of sandboxing restrictions, you can’t simply set up a folder and have both the app and the test runner read and write to it.

Passing fixtures in the Simulator

Having used the snapshot tool from Fastlane I remember they were actually writing to /Users/<username>/Library/Caches/ from the simulator when taking screenshots of the app. A quick test confirmed that both apps — the test runner and the tested app — could read and write to this folder.

A minor caveat is that when running tests simultaneous on different simulators — a feature that became available in Xcode 9 — you would have to be careful with concurrent access to that folder. A better solution is to write to the running simulator’s /Library/Caches folder. Fortunately, there’s an environment variable that allows us to do just that.

if let simulatorSharedDir = ProcessInfo().environment["SIMULATOR_SHARED_RESOURCES_DIRECTORY"] {
    // running on the simulator. We'll write to ~/Library/Caches
    let simulatorHomeDirURL = URL(fileURLWithPath: simulatorSharedDir)
    let cachesDirURL = simulatorHomeDirURL.appendingPathComponent("Library/Caches")
    XCTAssertTrue(FileManager.default.isWritableFile(atPath: cachesDirURL.path), "Cannot write to simulator Caches directory")
    let sharedFolderURL = cachesDirURL.appendingPathComponent("Secrets")
    XCTAssertNoThrow( try FileManager.default.createDirectory(at: sharedFolderURL, withIntermediateDirectories: true, attributes: nil), "Failed to create shared folder \(sharedFolderURL.lastPathComponent) in simulator Caches directory at \(cachesDirURL)")
    return sharedFolderURL
}

Now that we have a folder to write and read the fixture to, it’s just a matter of copying the fixture from the test bundle to that folder and passing the fixture URL to the main app via a launch argument. Because Secrets handles all user data with a subclass of UIDocument, our test fixtures are simply different documents. Here’s what our UI tests setup and tear down code look like:

class SecretsTouchUITests: XCTestCase {

    var testDirectoryURL: URL!
    var testDocumentURL: URL!
    
    func documentName() -> String? {
        return "testDoc1"
    }
    
    override func setUp() {
        super.setUp()
        
        self.continueAfterFailure = false
        
        let folderURL = sharedFolderURL()
        testDirectoryURL = temporaryDirectoryAtURL(folderURL)
        
        if let docName = documentName() {
            guard let docURL = Bundle(for: SecretsTouchUITests.self).url(forResource: docName, withExtension: "secrets") else {
                XCTFail("Failed to get URL for \(docName). Are you sure you included it in the test bundle?")
                fatalError()
            }
            
            testDocumentURL = testDirectoryURL.appendingPathComponent(docURL.lastPathComponent)
            XCTAssertNoThrow( try FileManager.default.copyItem(at: docURL, to: testDocumentURL), "Failed to copy document to shared folder")
        }
        else {
            self.testDocumentURL = testDirectoryURL.appendingPathComponent("NewDoc.secrets")
        }
        
        let app = XCUIApplication()
        var args: [String] = []
        args.append(contentsOf: ["-ResetDefaults", "YES"])
        args.append(contentsOf: ["-com.apple.CoreData.ConcurrencyDebug", "1"])
        
        if FileManager.default.fileExists(atPath: testDocumentURL.path) {
            args.append(contentsOf: ["-OpenDocumentAt", testDocumentURL.path])
        }
        else {
            args.append(contentsOf: ["-SetupNewDocumentAt", testDocumentURL.path])
        }
        app.launchArguments = args
        app.launch()
    }
    
    override func tearDown() {
        // Tidy up by removing the test directory
        try? FileManager.default.removeItem(at: testDirectoryURL)
        super.tearDown()
    }
}

Passing fixtures on the device

Unfortunately, on the device, you’re only able to write to folders inside your application container or to an application group folder. So the above solution wouldn’t work, but perhaps the application group folder would… Because your test bundle is run indirectly by means of a test runner app Xcode installs on your behalf, I knew it was a long shot but decided to try anyway.

On the developer portal, I proceeded to create an App ID for my UI test bundle, adding an application group to it (I simply reused one that Secrets already uses) and generating the respective provisioning profile. In Xcode, I set up my UI test bundle to run using that profile and added an entitlements file containing the application group. And…

…much to my surprise, it worked! I could pass my fixtures on the device via this app group folder 🎉.

So now we have a single method that gives us a shared folder to pass our fixtures that work both on the simulator and on the device.

private func sharedFolderURL() -> URL {
        
    if let simulatorSharedDir = ProcessInfo().environment["SIMULATOR_SHARED_RESOURCES_DIRECTORY"] {
        // running on the simulator. We'll write to ~/Library/Caches
        let simulatorHomeDirURL = URL(fileURLWithPath: simulatorSharedDir)
        let cachesDirURL = simulatorHomeDirURL.appendingPathComponent("Library/Caches")
        XCTAssertTrue(FileManager.default.isWritableFile(atPath: cachesDirURL.path), "Cannot write to simulator Caches directory")
        let sharedFolderURL = cachesDirURL.appendingPathComponent("Secrets")
        XCTAssertNoThrow( try FileManager.default.createDirectory(at: sharedFolderURL, withIntermediateDirectories: true, attributes: nil), "Failed to create shared folder \(sharedFolderURL.lastPathComponent) in simulator Caches directory at \(cachesDirURL)")
        return sharedFolderURL
    }
    else {
        // running on the device. We'll write to the AppGroup folder
        guard let appGroupURL = FileManager.default.containerURL(forSecurityApplicationGroupIdentifier: AppGroupName) else {
            XCTFail("Failed to get URL for app group \(AppGroupName). Check your entitlements.")
            fatalError()
        }
        return appGroupURL
    }
}

Finally, in case you’re wondering, the app group folder solution doesn’t work when running in the simulator… Xcode doesn’t actually sign the test runner app when targetting the simulator, so your profile and entitlements don’t seem to come into play.

Back to posts