Building a healthcare platform from the bottom up and one that is now ubiquitous in operating rooms around the world means that our development team has to be able to detect issues early, react to them, and make rapid changes to solve any problems. All in the blink of an eye. The adaptability, creativity and agility of the Proximie development team enables them to tackle challenges - seen and unforeseen - head on. Following a recent update from Apple to Xcode - an integrated development environment (IDE) from Apple designed to help developers build apps - and after a major iOS update, a host of unique challenges disrupted the team's CI (Continuous integration is the practice of automating the integration of code changes from multiple contributors into a single software project) infrastructure. In this case study which Proximie is making open-source, May Chehab and Hamza Jadid, from our development team, showcase how our team expertly negotiated the choppy waters post the transition to XCode 16.
As an iOS team, we’re no strangers to the challenges that come with new XCode and iOS releases. Apple regularly introduces updates to Xcode alongside major iOS versions, bringing exciting new features and improvements. However, these updates often come with changes that can disrupt existing workflows, particularly in CI pipelines.
In our case, the transition to XCode 16 and the upgrade of our device farm posed a unique challenge. While we expected some adjustments to our CI infrastructure, one critical issue caught us off guard: the deprecation of functionalities in xcresultparser, a tool we relied on for test result processing. We used that tool to extract test result data from XCode’s .xcresult format and convert it into an XML format. This XML format was crucial, as it was compatible with another tool in our CI pipeline. After extracting the data, we uploaded it to GitHub Actions, where we used a GitHub Action called dorny/test-reporter.This action would take the JUnit XML format and generate a visually clear report, summarizing the results of the tests. It provided an overview of failed and passed tests.
After the deprecation, we needed to switch to a new tool for extracting test result data from XCode’s .xcresult format. As part of this transition, we also had to select a new GitHub Action that was compatible with the new tool.
We began exploring other tools, but none of them quite fitted our needs. So during our rocket time (other companies call it innovation time), we started to build our own tool.
If we opened the .xcresult, we found an SQLite database containing test results to the very last detail. And after running a schema visualizer on the database, we were able to understand how the data was structured and write queries accordingly, and voila, we got our test results in our pipeline!
We ran into the "worked on my machine" issue, in which the .xcresult bundle had the test run database. But when we added the tool to our CI, it couldn't find the database.
We discovered that the database file only becomes accessible after manually opening the .xcresult bundle with XCode. Once opened, we were able to access the database and query it successfully.
To automate this process, we initially added a step in our CI to open the .xcresult bundle with XCode, run the SQL query to retrieve the data, and then close XCode after finishing. However, this approach required additional functions to manage XCode's process lifecycle (e.g., ensuring it closed properly after data retrieval), which introduced unnecessary complexity.
To streamline the process, we used the built in xcrun tool, which provided us with the database without requiring XCode to be opened manually. This eliminated the dependency on XCode's UI and significantly reduced the overhead involved in accessing the test results database.
Once we gained access to the database, understanding its structure and extracting meaningful data became a significant challenge.
The database was complex, with numerous tables and intricate relationships between them. It contained a large amount of data, much of which was not immediately relevant, making it difficult to pinpoint the information we needed.
This process required a deep understanding of the database schema, careful query crafting, and significant time to validate the relationships between tables to ensure accuracy.
The tool requires the following inputs:
For instance, the above command generates a file named result.md in your directory, containing a markdown report of the failed tests only. For each failed test, the report includes the reason for the failure, the file and code line where the issue occurred, duration of the test, and the number of times the test failed (if it was configured to run multiple times).
But why stop there? Since we had all the data we needed about the test results, we could enhance the feedback process even more.
We made the tool reliant on the Handlebars template engine to generate the output; that way, we would never be limited to a specific format.
Currently, for our use cases, we've created two templates; the first one is a Markdown template which we can use in our CI summary. We use GitHub Actions, so we could echo the output to $GITHUB_STEP_SUMMARY, and it appears in the summary tab.
The second one is a Slack template that uses the Slack Builder Kit.
You can install the tool either using cargo or brew.
The ability to read directly from the SQLite database provides customized insights and better understanding of the data. Also, using templates improves the visualization experience by allowing for adjustable data displays and easy integration with CI. For example, the results can be transmitted to platforms such as Discord or reported directly in GitLab. And the upcoming new features will make it even more unique!
We are looking at implementing these features:
After overcoming all these challenges, we have made it open-source!
x-result-analyzer was built to solve our problem, and we've seen online that people are facing similar problems. So, we have open-sourced it to give back to the community.
If you are facing an issue with the tool or you have a feature request, issues and PRs are welcomed!
.jpg)
.jpg)