Supercharge Beta by Crashlytics with fastlane: automate app testing to save more time

By Hemal Shah, Product Manager

Supercharge beta by Crashlytics with fastlane

Feedback that comes early and often is the best way to improve your app experience. Your users are your greatest source of learning (what can I do better?) and inspiration (what should I build next?), which is why it’s such a shame that beta testing is a tedious, tiring, and hair-pulling process.

We built Beta by Crashlytics to fix this by making it easy and straightforward to distribute beta builds to your users. Simplified beta distribution is awesome, but imagine how much more time you’d save if you could also automate it in seconds. Now, all iOS apps using Beta by Crashlytics can harness the power of fastlane to automate beta deployment with a simple, guided setup process!

 

Less tedious work, more time to innovate

A successful beta release involves a lot of steps – you have to bump version numbers, compile the build, append the release notes, distribute it to your testers, alert your team that a build went out, and the list goes on and on. And if that wasn’t enough, sometimes only one person knows how to distribute a beta build, which means you’re tied to their schedule and there is a bottleneck for delivering value to your app users.

Beta by Crashlytics takes the pain out of this process and makes it intuitive and efficient. By installing fastlane, an award-winning mobile deployment toolset, you can automate the tedious tasks that slow down your app development. Say “goodbye” to manual work and spend more time innovating, while still getting a healthy stream of user feedback.

Beta distribution takes only seconds using fastlane:

Beta distribution with fastlane

 

Instant access to all the power and benefits of fastlane

As an added bonus, with fastlane, you’ll also unlock new tools to help you customize your build process and effortlessly release your app to the App Store. Plus, fastlane is fully extensible so you can create and consume actions that automate all aspects of your mobile development – whether it’s working with Git, posting to Slack, etc. – there are 170 built-in actions and 50 third-party plugins to choose from!

Here’s an example of fastlane’s extensibility: We noticed that developers are increasingly forced to use multiple beta distribution services to address different needs. With fastlane, you can use the same process to automate beta deployment to Beta by Crashlytics and stage your build on iTunes Connect. fastlane integrates with the services you already use and love, so everything will still work exactly as you designed it – it’ll just be light years faster and you’ll have more flexibility and control.


lane :beta do
 increment_build_number
 gym                  # Build your app
 testflight           # Upload to TestFlight
end

lane :appstore do
 snapshot             # Generate screenshots for the App Store
 gym                  # Build your app
 deliver              # Upload the screenshots and the binary to iTunes
 slack                # Let your team-mates know the new version is live
end

 

The fastest set-up process, ever

We know that one of the biggest hesitations to adopting a new tool, no matter how incredible it looks, is the fear of a terrible and time-consuming onboarding experience. Nobody wants to spend days connecting everything and making sure it’s set up properly. That’s why we worked hard to make it easy as pie to enhance Beta by Crashlytics with fastlane. This will be the smoothest onboarding process you’ve ever seen.

Want to see for yourself? All you need to do is open up the Fabric Mac app and click on the “fastlane” tab in the top menu. Then, click the “Automate your beta” link and we’ll use existing information about your app to auto-generate your single instruction file (called a Fastfile). From there, just build your app and you’re good to go.

You can even use the same Fastfile to automate taking screenshots and deploying to app stores. And once fastlane is setup, everyone on your team can use it out of the box (no extra knowledge transfer needed!) and it seamlessly works with your CI server.

fastlane + beta onboarding process

 

A better beta experience awaits

User feedback can reveal app issues and opportunities that even the most astute development team may miss – it is the key to meaningful iteration. We care about your success, which is why we invested time and resources to drastically improve the beta testing process. With the combined power of Beta by Crashlytics and fastlane, you can get all the feedback you need in a fraction of the time and effort. And it only takes a few seconds to save precious hours. Upgrade to fastlane today and let us know what you’re building with all of your extra time!

For more information about doing beta deployment through fastlane, check out our quick setup guide.

Fabric September Update

by Brian Lynn, Sr. Product Marketing Manager

Now that fall is here and school’s back in session, we’re studying up on how to make life easier for you and your app development team. This month, we released three new features to help you understand your users and strengthen your app quality:

Your key performance indicators — now available on the go

Many of you are already tracking Answers Events and your key performance indicators (KPIs) via your Fabric dashboard. Now, with the latest version of our Fabric iOS app, you can easily track those KPIs and monitor user actions even when you’re away from your desk. By combining these additional insights with your adoption and stability metrics (e.g., DAU, MAU), you’ll know exactly where to focus your app improvement efforts.

Get the latest update:

(Android coming soon!
Follow us on Twitter so you don’t miss it)



See the most critical issues across builds or by device/OS

To help you fix crashes even faster, we added a new build selector as well as device and OS filters to your Crashlytics dashboard. Now, you can easily find and prioritize issues that are affecting multiple builds and increase stability across the spectrum.

On top of the new build selector, you can even filter crashes by device or OS. Want to focus in on crashes happening on iPhone 7 or Android 7, or compare crashes on iOS 10 vs. iOS 9? These new filters make it easy to search, compare and address your app's stability.

See these in action on your Crashlytics dashboard!

New fastlane docs: automate your app releases

Earlier this month, we launched docs.fastlane.tools, our new docs website for fastlane. This new site will walk you through fastlane’s set-up process and show you how to streamline tedious work when releasing your app, like taking screenshots, beta distribution, code signing, and more. Check it out!

Sidenote: we’re also happy to share that fastlane has now surpassed 500 contributors on GitHub. We continue to be humbled by your support, and we can’t wait to keep moving fastlane forward!

Here’s our internal changelog:

Fabric

  • iOS

    • Update upload-symbols for macOS Sierra compatibility

    • Warn on projects built for ≤ iOS 6 and ≤ macOS 10.7 that Fabric compatibility with those version is deprecated

  • Android

    • Update the Fabric dependency to update Crashlytics Core

Crashlytics

  • Android

    • Improved support for Android M & N

    • Facilitated improved NDK support on Android M & N

    • Updated Crashlytics Core dependency

    • Fixed issue which prevented sending crash reports in the rare case battery level info is not available

Answers

  • Android

    • Fixed a bug that caused Answers to undercount the number of Daily New Users Daily New Users count may be temporarily higher when you first launch a version of your app with this SDK

    • Updated Answers dependency

Digits

  • Android

    • Fixed error inflating StateButton when requesting email

    • Introduced new logger events to obtain errors while submitting phone number or confirmation code

Twitter Kit

  • Android

    • Bump dependencies

    • Added translations

    • Removed pseudo locales from translations

    • Updated proguard rules for Okhttp3 and Retrofit2

    • Removed pseudo locales from translations

    • Moved TwitterCollection from internal package to models

    • Minor bug fixes

    • Removed pseudo locales from translations

MoPub

How Crashlytics symbolicates 1000 crashes/second

by Matt Massicotte, Software Engineer

One of the most complex and involved processes in the Crashlytics crash processing system is symbolication. The needs of our symbolication system have changed dramatically over the years. We now support NDK, and the requirements for correctness on iOS change on a regular basis. As the service has grown, our symbolication system has undergone significant architectural changes to improve performance and correctness. We thought it would be interesting to write something up on how the system works today.

First things first – let’s go over what symbolication actually is. Apple has a good breakdown of the process for their platform, but the general idea is similar for any compiled environment: memory addresses go in, and functions, files, and line numbers come out.

Symbolication is essential for understanding thread stack traces. Without at least filling in function names, it’s impossible to understand what a thread was doing at the time. And without that, meaningful analysis is impossible, whether by a human or an automated system. In fact, Crashlytics’ ability to organize crashes into groups typically relies heavily on function names. This makes symbolication a critical piece of our crash processing system, so let's take a closer look at how we do it.

It starts with debug information

Symbolication needs a few key pieces of information to do its work. First, we need an address to some executable code. Next, we need to know which binary that code came from. Finally, we need some way of mapping that address to the symbol names in that binary. This mapping comes from the debug information generated during compilation. On Apple platforms, this information is stored in a dSYM. For Android NDK builds, this info is embedded into the executable itself.

These mappings actually hold much more than needed just for symbolication, presenting some opportunities for optimization. They have everything required for a generalized symbolic debugger to step through and inspect your program, which may be a huge amount of information. On iOS, we have seen dSYMs greater than 1GB in size! This is a real opportunity for optimization, and we take advantage of this in two ways. First, we extract just the mapping info we need into a lightweight, platform-agnostic format. This results in a typical space-saving of 20x when compared to an iOS dSYM. The second optimization has to do with something called symbol mangling.

Dealing with mangled symbols

In addition to throwing away data we don't need, we also perform an operation called “demangling” upfront. Many languages, C++ and Swift in particular, encode extra data into symbol names. This makes them significantly harder for humans to read. For instance, the mangled symbol:

_TFC9SwiftTest11AppDelegate10myFunctionfS0_FGSqCSo7NSArray_T_

encodes the information needed by the compiler to describe the following code structure:

SwiftTest.AppDelegate.myFunction (SwiftTest.AppDelegate) -> (__ObjC.NSArray?) -> ()

For both C++ and Swift, we make use of the language's standard library to demangle symbols. While this has worked well for C++, the fast pace of language changes in Swift has proven more challenging to support.

We took an interesting approach to address this. We attempt to dynamically load the same Swift libraries that the developer used to build their code, and then use them to demangle their symbols on their machine before uploading anything to our server. This helps to keep the demangler in sync with the mangling the compiler actually performed. We still have work to do to stay on top of Swift demangling, but once its ABI stabilizes it will hopefully present much less of a problem.

Minimizing server-side I/O

At this point, we have lightweight, pre-demangled mapping files. Producing the same files for both iOS and NDK means our backend can work without worrying about a platform’s details or quirks. But, we still have another performance issue to overcome. The typical iOS app loads about 300 binaries during execution. Luckily, we only need the mappings for the active libraries in the threads, around 20 on average. But, even with only 20, and even with our optimized file format, the amount of I/O our backend system needs to do is still incredibly high. We need caching to keep up with the load.

The first level of cache we have in place is pretty straightforward. Each frame in a stack can be thought of as an address-library pair. If you are symbolicating the same address-library pair, the result will always be the same. There are an almost infinite number of these pairs, but in practice, a relatively small number of them dominate the workload. This kind of caching is highly efficient in our system – it has about a 75% hit rate. This means that only 25% of the frames we need to symbolicate actually require us finding a matching mapping and doing a lookup. That's good, but we went even further.

If you take all of the address-library pairs for an entire thread, you can produce a unique signature for the thread itself. If you match on this signature, not only can you cache all the symbolication information for the entire thread, but you can also cache any analysis work done later on. In our case, this cache is about 60% efficient. This is really awesome, because you can potentially save tons of work in many downstream subsystems. This affords us a great deal of flexibility for our stack trace analysis. Because our caching is so efficient, we can experiment with complex, slow implementations that would never be able to keep up with the full stream of crash events.

Keeping the symbols flowing

Of course, all of these systems have evolved over time. We started off using hosted Macs and dSYMs directly to symbolicate every frame they saw. After many scaling issues, the introduction of NDK support, and Swift, we've ended up in a pretty different place. Our system now makes use of Twitter's Heron stream-processing system to handle close to 1000 iOS/tvOS/macOS and Android NDK crashes per second. Our custom mapping file solution uses an order of magnitude less bandwidth from developer's machine than it once did. We have a cross-platform symbolication system. On-client demangling produces more correct results than ever, especially as Swift goes through rapid iterations.

We’ve come a long way, and we're always working to improve the process. A few months ago, we released a new tool for uploading debug symbols. Not only does it incorporate all of our dSYM translation optimizations, but is also really easy to use in scripts or automation.

If I had to pick one big take-away from this work, I would say always be willing to consider custom file formats. Translating dSYMs right on the developer’s machine allowed for a dramatic savings in size, much improved privacy of the app’s internal structures, and more correct results. For NDK, the same approach also made for a simpler developer workflow, and cross-platform compatibility. These are very substantial wins for our customers and for our operational costs and maintenance of our systems.

For those of you that would like to hear even more technical details about crash reporting on iOS, you can check out my Flight presentation. Let us know how you enjoyed this look at our symbolication internals, and if there are other areas you might want to learn more about.

Get Crashlytics

 

We're building a mobile platform to help teams create bold new app experiences. Want to join us? Check out our open positions!

Fabric August Update

By Annum Munir, Product Marketing Manager

Summer’s almost over but we’re not taking any breaks from shipping new features. In August, we focused on expanding your view into app stability, updated a few of our Android SDKs, and added new functionality to our mobile app. We also celebrated a major milestone for fastlane! Read on for more details:

Introducing OOM reporting: a new dimension to app quality

We extended our crash coverage to include out-of-memory (OOM) reporting on iOS. An OOM event is an unexpected app termination that occurs when a mobile device runs out of memory. However to your users, OOM events look just like crashes, which makes them detrimental to your stability and also extremely difficult to detect. This month, we used intelligent heuristics to bring OOM reporting to Crashlytics. Now, you can monitor your OOM-free sessions, immediately see when they become a problem, and get valuable direction on where to start your troubleshooting.

Don’t let OOMs disrupt your user experience! Learn how on the Crashlytics blog.

Launched Twitter Kit 2.0 and Digits 2.0 for Android

We upgraded our Twitter Kit and Digits SDKs for Android to keep them stable, predictable, and reliable (we already did this for Twitter Kit for iOS a few months ago). In version 2.0, we refined our libraries based on your feedback, updated major underlying dependencies, and also enhanced performance and aligned with modern Android tools. To get these latest versions, simply click the “Update” buttons within your Android IDE plugin - we’ll take care of the rest.

For more specifics, check out the Twitter Developer blog.

Fabric mobile app updates: Crashlytics-only mode & account switching

The Fabric mobile app helps you keep tabs on your app when you’re on the go. While our app gives you a wealth of real-time analytics data, if you want to focus solely on stability you can now switch to a “Crashlytics-only” mode. This way, you have visibility into all of your crashes even if you don’t have Answers enabled!

On top of the new mode, you can also easily switch between your Fabric accounts within our mobile app. In just a few clicks, you’ll be able to monitor all of your releases across all accounts — even if you’re away from your desk. 

Update the app today to get these upgrades:



fastlane surpasses 10,000 stars on GitHub

fastlane automates the tedious, repetitive tasks of mobile deployment so you can spend more time doing what you love: creating amazing user experiences. Over the past months, we’ve worked with our community to make fastlane even better. Today, we’re excited to share that fastlane has surpassed 10,000 stars on GitHub. In fact, fastlane now has more GitHub stars than the language it was written in. We’re humbled by your support and we can’t wait to keep moving fastlane forward!

Here’s our internal changelog:

Fabric

  • iOS

    • Added logging of a warning if Fabric +with: is incorrectly invoked multiple times

    • Improved beta support when Fabric is embedded in a dynamic library

  • Android

    • Fixed issue causing the Crashlytics privacy prompt to not be shown in rare cases

Crashlytics

  • iOS

    • Improved defensiveness when handling Custom Keys and Logs data

    • Added support for Answers 1.3.0

  • Android

    • Updated Crashlytics Core dependency

    • Improved crash reporting efficiency when handling stack overflow errors

Answers

  • iOS

    • CPU/networking are now reduced when in Low Power mode on iOS, or under thermal pressure on macOS

    • Adopted NSURLSession background uploads, making for much more efficient and reliable networking

    • Adopted NSBackgroundActivityScheduler, which results in improved background behavior on macOS

    • Improved compatibility for macOS apps that use Automatic Termination and Sudden Termination

    • Improved visibility of Answers background operations by adopting NSActivity APIs

    • Improved on-disk event storage, reducing I/O and CPU overhead

    • Fixed a bug that could cause Answers to send a report with no events

  • Android

    • Updated Crashlytics Core dependency

Digits

  • Android

    • Clarified external api by defining "internal" package

    • Clarified events generated by defining "events" package

    • Enabled unique user counts per custom attribute by updating the sample application's logger to use custom events

Twitter Kit

  • Android

    • Removed Digits dependency

    • Dropped support for API versions before API 14 (ICS)

    • Updated Twitter Core dependency

    • Removed previously deprecated methods and classes

    • Added contentDescription for media based on altText field

    • Migrated to Retrofit 2.0 and OkHttp 3.2

    • TwitterApiClient now automatically refreshes expired guest tokens

    • Removed all public reference to Application Authentication

    • Fixed issue parsing withheldInCountries field in User objec.

    • Added altText field to MediaEntity object

    • Added Quote Tweet to Tweet object

MoPub

Introducing OOM reporting: a new dimension to app quality

by Sean Curran, Software Engineer

Stability issues can derail the success of even the best apps – glitchy software repels people. We know that app quality is one of your top priorities and crashes are your worst nightmare, which is why Crashlytics will always alert you when issues arise. We’ll even help you pinpoint their root cause so you can fix issues fast – we’ve got your back!

Today, we’re extending our crash coverage to include out-of-memory (OOM) reporting on iOS. Now, you can see stability from a whole new angle by understanding the impact OOM events have on your app experience.

What’s an OOM event and when does it occur?

Unexpected app terminations degrade your app experience and interrupt your user’s session. One type of app termination you’re probably familiar with is crashes, but there’s another unexpected termination that warrants your attention called an OOM event.

An OOM event is an app termination that occurs when a mobile device runs out of memory. All apps need memory to work, but there is only a finite amount available on each device. When an app needs more memory and there isn’t any available, the operating system terminates the app session. To your users, this looks like any other crash, however in reality, this is an OOM event.

Our approach to solving a hard problem: intelligent heuristics

OOM events are difficult to report because iOS doesn’t provide any direct mechanism to detect them and they can be caused by factors beyond your app environment and control, such as your app’s memory usage. But because of how important it is to understand your rate of OOM events, we took on this challenge and added OOM reporting to Crashlytics for iOS devices.

Here’s how it works: When you enable the Crashlytics and Answers Kits available on Fabric, we get a stream of live data about your app’s performance. Then, we apply a server-side process of elimination to this data stream to detect OOM events in your app. This detection is based on an intelligent heuristic, inspired by the work of two talented engineers. In other words, we analyze your app’s event stream to come up with an explanation of why it terminated and if we can’t match it up it to a known reason for termination, we count it as an OOM event. And, since no changes to the Answers SDK were required, you will automatically get OOM reporting without needing to do any work (as long as you have both Crashlytics and Answers installed).

This is an example of how your app analytics and stability kits can work together to unlock powerful new insight – something neither kit could do alone.

At Pinterest, we're shipping iOS app updates every two weeks to millions of people. Using the new OOM insights, we've been able to track our memory optimizations and have confidence in the stability of each release.

- Scott Goodson, Head of Core Experience


New line of sight into the impact of OOM events on app quality

Our new OOM reporting dashboard will help you understand if OOM events are a problem for your app. Now, you’ll be able to see the overall percentage of app sessions that were unaffected by OOM events across your builds. You can even drill down and see the percentage of OOM-free sessions for individual builds. This will help you answer important questions like, “Is my app being terminated more on one build than another?” and “Is my app seeing more or less OOM-free sessions over the last week?”

We’ll also give you a sortable, daily device breakdown so you can compare the raw OOM counts and the percent of OOM-free sessions across different iOS devices. Once you know which iOS device experiences the most OOM events, you can better triage and spend your time investigating problems on your most used devices. After you roll out a fix, you can monitor the number of OOM events on that device to see if your solution helped reduce them. You may even learn that there are some low-end devices that you simply can’t support.

This additional information paints a clearer picture of your app stability across devices and provides clues for troubleshooting.

Finally, we’ll show you the total number (i.e., the raw counts) of OOM events across all of your builds versus your top three builds. Use this graph to understand the prevalence and magnitude of OOM events.

Don’t let OOM events crash your app party 🎉

Our mission is to ensure that there are no more sad, unstable apps. By adding OOM reporting into Crashlytics, we’re giving you even more insight into the quality of your app. Don’t be blindsided by OOM issues that disrupt your users’ app experiences (and maybe even cause them to flee!). Monitor your OOM-free sessions, promptly identify when OOM events become a problem, and get valuable direction on where to start your troubleshooting by going to the OOMs page from your Crashlytics dashboard. Check it out and let us know what you think!

Get Crashlytics Now