iOS developers have been encountering this issue: bitcode app stack traces are showing
_hidden#123 instead of the symbols they expected. This change makes the stack traces unusable for the purposes of debugging.
While not new, this is a result of Apple’s decision to directly take bitcode from app development teams and re-finalize it on their own servers. This decision introduced an extra step where Apple chooses to obfuscate the binary programs so the symbols are protected from copyright issues. Going forward, we can expect this to be the way Apple operates with all bitcode programs.
In this post, we’ll cover the following:
- Why does Apple refinalize bitcode?
- Why does this process obfuscate my bitcode symbols?
- How can developers deobfuscate bitcode when debugging?
Why Does Apple Refinalize Bitcode?
iOS applications are really binary programs distributed in a container called an IPA. Because they are just binary programs, when they crash we see a stack trace. You use the symbols in that stack trace to help you map what happened at runtime back to the code you originally wrote.
A few years ago Apple introduced a new distribution method called bitcode. Bitcode just means that instead of uploading finalized binary code to Apple for distribution, you instead upload an intermediate representation called bitcode, or sometimes IR code.
Modern compilers use a 2-pass system to finalize the binary:
- A tokenizer runs the original source code (called the Clang frontend in iOS development) and turns it into bitcode.
- The LLVM compiler then turns this bitcode into binary code for iPhones to run.
Apple has continuously been improving the LLVM frontend to produce more efficient code as well as fixing security issues in the finalizer. However, Apple faced the problem of upgrade resistance — companies were not eager to upgrade their Xcode.
Ultimately, Apple decided to just accept that developers weren’t upgrading anytime soon so they began doing this process on their own terms. The result is that when you build with bitcode, you are giving Apple permission to re-finalize your binary on their servers whenever they fix a bug or security issue in LLVM.
Why Does This Process Obfuscate My Bitcode Symbols?
Normally, mobile teams ship their binaries with exposed symbols because the iPhone is a closed environment and users cannot view the symbols. “Symbols” refer to functions that other binary programs can call.
However, Apple’s process has a problem: binary programs expose all the symbols to them if they receive your bitcode. This opens Apple to the possibility of potential lawsuits over copyright issues. To avoid this, Apple only uploads obfuscated bitcode to their servers.
In other words: Apple will take a symbol like
_applicationDidFinishLaunching and obfuscate it to
_hidden#23456 before they see it.
This is why the stack trace just shows
_hidden#xyz for all the symbols in a bitcode app. Unfortunately, this means that developers cannot analyze the stack trace until it’s been deobfuscated.
How Can Developers Deobfuscate Bitcode When Debugging?
There are two options to deobfuscate these symbols: Manual and Automated.
The manual option is better if you only need to deobfuscate a few symbols and both the module and offset are already known.
Here’s an example line:
6 Embrace 0x5bf50 hidden#4051
In this example, the module is
Embrace and the offset is
0x5bf50. You must find the corresponding binary file or extract it from the IPA installed on the device.
You can then use a utility like
atos or even load the binary in a decompiler such as Ghidra to view the offsets. This allows you to look directly at the code that has crashed and the stack trace.
Sometimes you want to automate this process to improve workflow efficiency. Fortunately, Apple gives you everything you need to do this in the archive that you originally uploaded to them. Assuming you have that archive, right click on it and select the “Show in Finder” option, then select “Show Package Contents.”
Now you’ll see the normal dSYMs folder, but since you built your app with bitcode there will be an extra folder called “BCSymbolMaps”. This folder contains the mapping you need to deobfuscate all the hidden symbols.
Here’s an example image:
The name of each BCMap file should match the UUID of one of the dSYM files. So to automate this you write a script to loop through the dSYM folder, and output the UUID of each one:
dwarfdump -u Embrace.framework.dSYMUUID: D1B24FBA-C6F3-3418-803D-ED1452E54376 (armv7) Embrace.framework.dSYM/Contents/Resources/DWARF/EmbraceUUID: 40F947B3-2295-3344-A67E-D5D0BCCF9CBF (arm64) Embrace.framework.dSYM/Contents/Resources/DWARF/Embrace
Now you have the UUIDs of the Embrace module. This module contains two architectures, armv7 and arm64, so it has 2 UUIDs and 2 BCMaps that need to be applied. Using the UUID you can now find the matching BCMap file and apply it:
dsymutil --symbol-map ../BCSymbolMaps/D1B24FBA-C6F3-3418-803D-ED1452E54376.bcsymbolmap Embrace.framework.dSYM
If you run the above for every UUID/BCMap pair of all dSYMs in the archive, you will have a fully deobfuscated dSYM set. The stack trace for your bitcode app is now usable!
Summing It All Up
If you have been frustrated with obfuscated symbols and the dSYM map, that’s completely normal! As Apple shifts to obfuscating all bitcode, an additional step has simply been added for developers to deobfuscate their symbols when they receive a stack trace. We recommend setting up the automation of deobfuscation if the team expects to need to do it multiple times.
How Embrace Helps Mobile Teams
Embrace is an observability and developer analytics platform built for mobile teams. We are a one-stop shop for your mobile app’s needs, including error debugging and monitoring performance and feature releases.
Want to see how Embrace can help your team grow your non-game app with best-in-class tooling and world-class support? Request a customized demo and see how we help teams set and exceed the KPIs that matter for their business!