“Hunch” Study : Network Calls and their Effect on Mobile Apps

Every good mobile engineer has questions that they wish they could answer, if they only had the time. One of those questions, related to concurrent network calls, goes something like: “How many calls can my app do and not impact app performance?” Until recently, no one had actually measured to figure it out — then we did!

Hop in the time machine with us and we’ll explain how it happened . . .

Timehop is a nostalgic social networking app with tens of millions of downloads on iOS and Android. We worked with the Timehop team to understand how technical choices impact users’ experiences within their iOS app, and identified suboptimal areas to focus and improve upon. For purposes of this calculation, we focused on 3 days in October during which we collected data globally from over 7.5 million experiences, spread across over 2.5 million devices.

Let’s start with the common culprits.  SDKs are often the primary hunch for the cause of suboptimal experiences on mobile.  Knowing the hunch, most app developers still initialize *all* their SDKs on startup (somewhat of a standard practice).  Like all apps, Timehop embeds a lot of SDKs; 47 to be exact, ~16 from 3rd parties vendors.  (Please note that SDKs are not limited to 3rd parties, like analytics, ads, Google, FB, Twitter and the like, but are also many of the libraries that we use out of the box, like Bolts, Appirater, and SocketRocket). Starting the app with this many SDKs obviously impacts the quantity of network calls that occur during start-up.  

When initializing the app with all of these SDKs *plus* your own 1st party network calls, you’ll get a logjam, and many of these calls (3rd or 1st party) will block your app from progressing. In Timehop’s case, we focused on startup time and a very slow benchmark of 5 seconds.  We figured-out that the number of slow startups increased to 6% with ~6 concurrent network calls.  The number increased to 13% of startups with 10+ concurrent network calls.  Getting a bit more granular, we helped Timehop sort out that a user with any network call of over 2.5 seconds during startup correlated to an increase of +27% chance of that user leaving the app.

Further, we found that 3.3% of users were experiencing a slow startup (over 5 seconds), and 2.9% had a stall (e.g. what amounted to a freeze from the user’s perspective) at startup. We were also able to pinpoint that Mixpanel and StatHat were the primary sources of HTTP errors during startup. Timehop has been able to take this info, tweak their startup, and reduce user churn as a result.

The moral of the story? With a little time, good technology performance becomes good business.

You can’t change how many, or how fast, network calls occur, BUT you can control how well you have built your app. Embrace.io cures app blindness.  We make your app’s performance visible for each user and segment, leading to better user experiences, and improved business outcomes.

We are here to help. Let us know how we can for you.

–    Eric

 

Live Event Debugging, Memory Improvements, Feedback turned into Reality

We are excited for our latest release focused on taking the first step towards live feedback and continuing to make our SDK have as small a footprint as possible.  

  • Live event mode! 

One of the primary reasons we kicked-off Embrace.io (only) last year was to provide true visibility to the mobile team, especially the developers.  The ability to see technical details and reproduce sub-optimal experiences on any device is not limited to when then app is live.  Development has many similar pains, and we are happy to take the first step to provide 100% visibility when in that mode.

Live Event Debugging : For any app running with the debugger attached, watch “Live Events” roll by for you or any other user.  Test your integration to watch Embrace.io log messages and app moment events in real-time.  Select a Device via the dropdown for anyone attached – QA, developer or product.  As we iterate on this feature, our plan is to expose other device information so you can see in-app information fly by!

(Click to for a high-res view of the Live Event Debug Mode)

  • Run your app in the background?

The SDK now provides additional context around when your app is running or unexpectedly dies in the background.  Admittedly, we never thought of background as a primary use-case because users of an app are by definition not viewing the app.  Our minds were changed when customers explained that many of their use-cases relied on being in the background; for example, playing an audio/video file or tracking the movements of a driver.  

  • Memory management improvements

We will continue to improve our iOS and Android SDKs to reduce their footprint while bringing as many features to each customer as possible.  (Its a true balance!)  In this release of the iOS SDK (2.6.0,) we focused on reducing the SDK footprint around memory thereby making the app startup even faster

Please let any of us know whether you have any questions via Slack, Email or Twitter.

Thanks for all the support!

– Eric

There’s more to life than just crashes!

Mobile teams are singularly focused on crashes. Understandably so — they are the simplest to identify and diagnose. BUT crashes account for less than 1% of your issues! You still find errors and your users are still complaining.

With mobile apps, we are conditioned to think of all performance issues as ‘crashes’. That’s because a) there’s a tendency for users and business people alike to rush to the term ‘crash’ when it may not really apply, and b) there are limited tools on-par with their desktop counterparts for identifying and diagnosing mobile app performance issues.

So, even though the performance issue may deal with an inhibited experience (the app is taking so long time to load that it appears frozen), it’s treated like a crash. Ask yourself:
When you fix a crash, how often does it really impact the experience of any user?
How often is it in the background?
How often does it affect a small number of users who don’t contribute to the app or app community?
How often does a user just re-open the app anyway?

The truth is, most crashes are a fender-bender, but we treat them like 50 car pile-ups.

Instead of solving every issue, shouldn’t we be focused on solving the most important ones? If you are constantly distracted with minor performance issues that affect just a few low-value users, then you’re not spending time improving experience and heading off a serious error that could kill-off lots of high-value users.

The logic is the same whether you’re an engineer or a business person: focus on what matters and de-emphasize what doesn’t. This brings us right back to the need for mobile-first tools purpose-built to help engineers and their mobile teams quickly assess and resolve performance issues.

One of the unique things that Embrace does is let engineers stitch together user sessions. So, if the user experienced a crash, but then just immediately re-opened the app and all was fine, maybe it’s not worth solving that problem right now. We surface that data and visualize it so the engineer, product manager, or other member of the team can easily dig in and decide the appropriate course of action.

This ensures the most mission critical performance issues get the attention they deserve, relieves pain for engineers, and keeps users happy. And that’s what it’s all about.

We do what we do because we’re devs too

There are two categories of performance errors that engineers must worry about. The first are visceral issues: those that actually stop the experience altogether and abruptly. These include crashes, which are fairly easy to diagnose and stop, and things like memory issues, CPU pegging, and too many concurrent network calls, which are more difficult.

The second category deals with issues that inhibit the user and make them leave : a key moment taking too long, a spinner (of death) appearing, an error pop-up b/c of a broken purchase mechanism, a video not starting — things that impact an app’s ability to run fast or impede the user’s ability to interact with your app and perform the actions you intended. Outside of crashes, every one of these whether abrupt closes or frustrating experiences are tough to identify, diagnose and resolve quickly.

With desktop web, which is obviously a more mature ecosystem than mobile, engineers have a full complement of tools to manage every performance scenario from ‘The website crashed’ all the way to down to optimization (e.g. making a page load faster). So, when you do get a complaint for an experience-based performance issue that may or may not be code-based, like a page not loading quickly or a purchase mechanism failure, it’s fairly straightforward to identify, recreate and solve the problem. Unfortunately, that’s not the case with mobile.

The Embrace.io team cares deeply about this stuff because we’re developers too. We’ve felt your pain. That’s why we’re on a mission to eliminate the frustrations of building and running large-scale apps. No more duels with the Spinner of Death. No more time wasted trying to reproduce issues, or answer questions, that have nothing to do with your own code. We’re all about providing you the tools and data to move past the pain and get back to coding cool stuff. We’re rising to that challenge by creating a platform for mobile performance management as good, or even better, than what’s out there for desktop.

Why Embrace.io?

To put it simply, we love mobile.  Mobile is the center of our lives and will continue to be the center.  Three simple examples of why:

  • Apps: Over 80% of people prefer apps to mobile web experiences.  While web is here to stay, native is so much better for more utility, work, and social experiences.  We will have an app for everything and this won’t change;
  • IOT: How do you plan to control all those devices? Your phone is the definitive remote; and
  • VR: In 2 or 3 years, do you think you’ll be plugging that headset into a computer, or running it on its own? No, it’ll be plugged into your next gen phone.

While my team and I enjoyed building apps, we find it much more fun to work with a ton of developers.  Everyone loves giving advice and now it’s our full-time job. Not only that, but we back it with real data streaming from each mobile device.  

We know the specific user who had a spinner of death. We know when a marketing or advertising SDK is blowing up a segment of users. We know when a 3rd party network call for a purchase isn’t working . . .  we know a ton.  I wish I had this level of visibility when I built Dice With Buddies, instead of finding stuff out after the fact via complaints, lower retention curves, or banging on all the devices strewn across my desk.  

Nothing is more ‘fun’ than grabbing one of those phones, walking over to the lead mobile developer, and asking, “Why is the startup taking so damn long?!?”  Then they say, “Can you reproduce it?”, or “Can I plug your phone in?”, or “Wait a few days and I’ll see if I can dig into the log files or [MixPanel / Omniture / Flurry / <Insert ill-suited marketing analytics tool here>]”.  The answers to all three are far from ideal!

As a B2B company, our goal is to make customers love us.  While we’ve built an awesome platform and team to advise you, the mobile team and developer, we want you to ask us anything!  We’ve used every mobile tool on the market, we’ve mastered IAP and Ads, we know user acquisition, and LTV, and CPI calcs.   

We’re happy to be of service, make your lives easier, and your users happier! Just ask.