Introduction
The State Farm Android team fiercely defends a 99.9+% crash-free rate for our flagship State Farm® App. A team culture that focuses on availability and resiliency for our customers and a set of coding best practices that ensure a seamless experience when using the app.
In this post, we’ll take you through our journey of achieving and maintaining a 99.9+% crash-free rate for the State Farm Mobile App through an insider view of our strategies for monitoring and preventing crashes through every day engineering and testing practices.
A new start: Measuring success
A great starting point to our crash journey was in 2017 when the Android team faced a huge engineering endeavor to rebuild our flagship app “Pocket Agent” (which originally launched on Google Play in 2010) with a new app identity. Our new app, the State Farm App, featured a new design system, architectural patterns and features. Around the time of our efforts to build the app, Google Firebase Crashlytics emerged as an essential tool on our tool belt, getting virtually 100% coverage on most crashes. Now that we had a true measure of our crash-free rating, it was game on.
When you can measure it, you can improve it.
As we prepared for the big launch of our app redesign, we discovered several new ways to discover crashes and work towards a more resilient app. When we finally went live at the end of 2017, our app’s key features like paying a bill, filing a claim, viewing policy info were all going strong and highly available. We aimed for 99%, and we hit 99.1% crash-free. While we hit our first goal, what we actually had was a wealth of new data through Crashlytics.
We didn’t stop at 99.1% though; we learned, we modernized and became experts at building an incredibly reliable mobile app… eventually reaching a remarkable, sustained 99.99% crash free rating! At times, we even peaked at an incredible 100% crash-free, with Crashlytics reporting fewer than one crash per 10,000 users. This is no small feat, considering our app has nearly 5 million installs on active devices. According to the Google Play Console, the State Farm Mobile App ranks significantly higher in stability compared to most apps within the Insurance category.
Today, the State Farm App has more than 400 unique screens, calling more than 200 unique endpoints. Every release, we implement new features, enhance existing ones, and increase stability through bug fixes and modernizing our tech frameworks. The engineering team is composed of 15 native Android engineers, and works side-by-side with a dedicated team of manual and automation testers who help ensure every new app release is stable for our customers.
Mobile chaos engineering (destructive testing)
To fix things, you need to know how your users can break them, before they break them. While we can code to happy golden path scenarios at our desks, we needed new tools in our tool belts to simulate the real world, and the myriad conditions our app is used in. We doubled down on testing efforts and made a sport out of finding novel ways to find crashes before our customers did.
Recovering from Android process termination
We also discovered that combining two Android system Developer Options, 1) “Do not keep activities” (DKA) and 2) limiting the background process limit to zero (0PL), were a reliable way of simulating Android process termination. These are settings users can find in their device’s Android OS system settings after enabling a developer mode. It’s not uncommon for users “in the wild” to have these settings enabled thinking they may enhance device performance, when actually increasing the likelihood of unexpected app behavior for many of their apps.
Enabling these settings revealed to us a more direct way of simulating Android process termination (when the system ends your app’s process due to current resource constraints). For us, this meant that when a user resumed on the same screen days later, but without the data the screen assumed was there, our app would crash. These conditions are also similar to what happens when a user downgrades an app permission via Settings while our app is in the background. There are also other techniques to simulate and/or cause process termination within Android Studio via Logcat (i.e., Force stop application, Kill process and Crash application).
When process death occurs, the Android OS will still resume the user on the last screen they saw, even if it was last seen days, weeks, or months ago. So a great rule of thumb is to test each screen and how it recovers when a user backgrounds and restores the app with DKA and 0PL enabled. This often uncovers issues when screens don’t independently load their own screen data (e.g., Screen B assuming Screen A called required APIs, then crashing when Screen B gets restored after process death) as well as testing the consequences of long-running async processing finishing after a view has been destroyed.
Simulating unpredictable user behavior
This became an endless well of new problems to solve before our big release. In the real world, our users didn’t have the strong connections we had at our desks, they answer phone calls while using the app, they tap around and navigate rapidly, and they run on thousands of different Android devices, all of which reveal crash risks through detached Fragments, localization quirks and device specific nuances.
Through DKA + 0PL, we started finding that the crashes and the stack traces we found in the Developer Console and Crashlytics were finally replicable, locally and en-masse by simulating what customers often did: open our app from the background after hours, days or months since the last time they used it. What it revealed were state and session management issues, e.g., assuming any given screen had its required data in-memory, when in fact, the Android OS had started it from a clean slate.
This was the beginning of our destructive testing efforts. We started asking ourselves:
- What happens if I background the screen or navigate away while a service is running?
- What happens if I rotate?
- What happens if I double-tap or tap three buttons at the same time?
- What happens when users have non-US English locales?
What we found were new problems to solve and more techniques for earning higher crash-free rates.
API chaos testing through stubbing responses (test every scenario)
Another essential aspect of destructive testing is to test how our app responds to sporadic API failures, so that if things go wrong retrieving and submitting data, our app is ready to gracefully handle anything thrown our way. A key part of this is an in-house “Stub” framework where we can mock the behavior of all 200+ API calls in a highly detailed way, through specifying HTTP status, response time, payload data, and even simulating what happens on subsequent calls to the same service to test recovery from errors. Our stub test suite has more than 2,400 unique scenarios and that number grows each sprint.
"sample1" : {
"description" : "Sample API fails on first load, then recovers",
"expectation" : "User sees an error message. When the error is tapped, or user revisits the screen, then the API is successful and data is loaded.",
"map" : [
{"status":500, "matcher":".*/endpoint/sample", "file":"error", "isVariablePayload":true, "sleep":5000},
{"status":200, "matcher":".*/endpoint/sample", "file":"samplePayload", "isVariablePayload":true, "sleep":500}
]
}
Monkeying around
One last wily tool we discovered was Android’s “monkey testing” ADB commands, which allowed us to send tens of thousands of randomly triggered UI events (taps, swipes, system interactions). While coding at our desks, we’re a sample size of one, but we can never predict how our millions of users will use the app, and how the thousands of Android devices, each with their own performance specs and limitations, will perform in the real world.
How we monitor releases and what we look for
The Android team releases a new version of the app to the Play Store every three weeks. Every release typically has a combination of either new features or feature enhancements, bug fixes and technical upgrades.
After rounds of dedicated iterative manual testing from engineers and testers as well as our automation team running extensive regression test suites, how do we know that our release is stable and performing well in the public once it goes live?
Phased rollouts and early monitoring
We start with a phased rollout for a week on Google Play, which allows us to get the latest version out to the public, but at a controlled pace, so that if an early issue arises, we can halt the release, fix it, and continue forward.
When a rollout begins, Google Firebase Crashlytics alert emails for new crashes and velocity spikes let us know if something requires urgent action within the first hour. Typically, we look for multiple users impacted once, and especially multiple users impacted multiple times (crash looping). This allows us to estimate potential impact and decide what has potential to be an outbreak crash (e.g., will 10, 100, 1,000 users crash in the full release cycle? Does the crash have potential to self-resolve?). The context of a feature matters too: Is it a feature used a thousand times per month or a thousand times per minute? When we clearly see a potential major issue, we’ll chat, collaborate, talk about impacts and decide if it’s best to halt the release and fix now.
Two sources of truth: Crashlytics and the Developer Console
It was also surprising to learn that you need both Google Firebase Crashlytics and the Google Play Developer Console to see the fullest picture of crashes. The Developer Console often catches crashes that Crashlytics cannot, such as crashes in native code as well as crashes that occur prior to the initialization of the Crashlytics SDK. So checking both places regularly are key to staying on top of stability. For both platforms, prior to release, we also run a “Crashlytics health check” that helps us verify that crash reporting is working as expected and that our obfuscation mapping has successfully been uploaded (so we can easily decipher crashes in the consoles when they occur).
Pre-release readiness review with all Android engineers
Even before releases, we monitor crashes in a separate Crashlytics test project and verify any major issues introduced in development during our sprint have been properly addressed in our pre-release “Android Readiness Review”. We make sure any open crashes are fixed or closed if necessary. The Android Readiness Review is also a great chance for the team to spend dedicated time reviewing each others features on a release build, create follow-up issues, and make sure our release is ready for our customers in the coming days. We started this Review in response to a few rough patches with crashes a few years back, but have kept it going and continued evolving as it is always fruitful for discussion and resiliency.
Signals from even more channels
In addition to crash-oriented metrics, we heavily monitor analytics through multiple dedicated channels, including Adobe Analytics, Splunk and a customer-feedback platform, in addition to Play Store reviews and customer support channels. So even if our crash data suggests app health, we have multiple signals that help give us a full picture.
Fix now or fix later
When we see lower volume edge case crashes that don’t necessarily require an immediate fix, we still try to prioritize quick fixes in our next sprint to avoid having crashes pile up. Even if just a single user crashed once, if it’s an easy fix (they usually are), then we fix it while working on other sprint feature work. While we always try to replicate all crashes we attempt to fix, it’s not always possible, so we’ll make sure the issue is addressed. Closing crashes in the Crashlytics console and then getting subsequent alert emails about a resurfacing crash is a great way to know when we need to dig further.
Crash out loud in dev, log in prod
Our golden rule: Users should not experience a crash. The “should never happens” may indeed happen, and if it does, handle it gracefully. Engineers and testers are equally keyed in on covering our happy paths thoroughly and then bulletproofing through destructive testing. For the “should never happen” edge cases, we want to know as soon as possible in sprint development when they do occur. A technique is to explicitly throw crashes in BuildConfig.DEBUG while logging troubleshooting info to Crashlytics using non-fatal event logs for BuildConfig.RELEASE builds.
if (theImpossibleHappened) {
if (BuildConfig.DEBUG) throw IllegalStateException(…)
CrashlyticsNonFatalEventLogger.log("theImpossibleHappened ${moreContextDetailsAboutWhatHappened}")
}
Kotlin non-null assertions (!!) and nullability annotations in Java
As we migrated more Java to Kotlin, we had some hidden problems: un-annotated @Nullable fields, particularly data deserialized from network calls. So while our Kotlin code was equipped with null-safety, it can't handle nulls safely if it doesn't know about them. So as we saw crashes happen as our mobile API layer change data optionality, one thing was true: avoid !! like the plague. It was key that while we moved hundreds of thousands of lines of code from Java to Kotlin, in between, we had to make sure to mark @Nullable on our Java code.
One thing we learned is never assume a field will be returned. What this means to us: assume all API field data could potentially be null.
class SampleSaferVehicleJavaModel {
@Nullable
final String year;
@Nullable
final String make;
@Nullable
final String model;
}
data class SampleSaferVehicleKotlinModel(
val year: String?,
val make: String?,
val model: String?,
)
Along these lines, while we were all learning Kotlin, we needed to become smarter about smart casts, never assuming data is 100% guaranteed or cautiously handling expected data types.
fun processResponse(responseData: Any?) {
val isSuccessful = responseData as Boolean // crashes if responseData is null
val isSuccessful = responseData as Boolean? // crashes if responseData is not a boolean
val isSuccessful = responseData as? Boolean ?: false // handles null and non-Booleans safely, with a default value of false.
}
Feature flags and Feature Blocks
Our apps fully embrace feature flagging as well as the idea of Feature Blocks. If we see a new problem emerge, we can quickly toggle any related feature flags through Google Firebase Remote Config at a granular level (app version specific). Our Feature Block framework lets us block more than 60 unique features within our app with a high level of customization. At a high-level, Feature Blocks are a way to tell customers, “This feature is still here, but we’re working on fixing an issue”. This allows custom messaging, offering alternatives to customers, such as navigating to the equivalent feature on statefarm.com or through a call-in channel. So in a crash outbreak scenario, our app could temporarily turn off a feature, while letting non-impacted features to continue working as usual. This framework also gives us an opportunity to ask users to update to the newest version of the app on the Play Store to use features that have been repaired.
Even when things remain stable for long periods of time, feature flags and feature blocks provide us with flexibility, safety, and confidence that when something goes wrong, we can impact the fewest number of users possible.
// Feature flagging sample for quickly toggling features remotely
if (FeatureFlags.SAMPLE_LOCAL_FLAG.isEnabled && FirebaseRemoteConfigFeatureFlag.RELATED_SAMPLE_REMOTE_FLAG.isEnabled()) {
handleAutoClaimSelected()
} else {
handleAutoClaimSelectedLegacy()
}
Lifecycle safety
Before we moved to more modern patterns like MVVM and Jetpack Compose, our MVP and XML-based architecture, detached Fragments were a leading source of crashes.
A simple example:
- A user submits a preference update (that took 4 seconds on a poor connection).
- After 1 second, the user decides to navigate back.
- The Activity and Fragment are popped from the back stack.
- After 4 seconds passed, the Presenter called back to the View, and the View incorrectly assumed the Activity was present.
With detached Fragments, calls to Fragment#getString, Fragment#getContext, Fragment#startActivity as well as showing error AlertDialogs would all cause a crash. Ultimately, this revealed flaws where our Presenters were living longer than they should have while also leaking Views. So we got better at testing and removing View callbacks, while leveraging WeakReferences when it made sense.
In more modern architecture, Compose helps us better manage state and how our data layer triggers updates to the UI, particularly with StateFlow#collectAsStateWithLifecycle.
Even lifecycle-safe coding practices can be vulnerable to edge cases, either in the form of crashes or unexpected behavior. So a few useful extensions we built:
Extension: fun NavController.navigateSafely(…)
NavControllers can be prone to edge case crashes, particularly when users double or triple-tap buttons when device resources are low. Each of our NavController#navigate invocations flow throw an exception catching extension. Our extensive manual and automation testing efforts will catch any happy path issues, while the extension covers us on the edge cases.
fun NavController.navigateSafely(…) {
try {
this.navigate(…)
} catch (navigationException: Exception) {
Logger.e(TAG, Log.getStackTraceString(navigationException))
}
}
Extension: fun LifecycleOwner.isAtLeastStarted()
When navigating from Compose UIs, we perform a check to verify that the UI is in an interactive lifecycle state that is ready to navigate. This also prevents multiple destinations from stacking up in the event a user multi-taps different actions on a screen simultaneously.
fun LifecycleOwner.isAtLeastStarted(): Boolean {
return lifecycle.currentState.isAtLeast(Lifecycle.State.STARTED)
}
Remote crash absorbing through UncaughtExceptionHandler
In rare cases, background crashes emerge in the wild that aren’t user-facing, but nonetheless still impact crash reporting metrics. App crashes that occur in the background are imperceivable to users, but may interrupt any ongoing background processing work. These types of crashes can happen with new OS releases or mid-release changes in dynamic libraries. While we never want to fail silently, sometimes we choose to absorb 100% background crashes that would have no user impacts.
We do this through setting a custom UncaughtExceptionHandler. Through Google Firebase Remote Config, we can configure different crash signatures to match on based on their stack trace. In some cases, we may absorb a targeted crash and log the instance to our analytics suite. In other cases, where there’s potential for user recovery, we’ll intercept the crash to show resolution advice in a quick Toast message, before allowing the Android OS to continue with standard crash handling behavior.
[
{
"checkForCrashMessageContaining": "can't deliver broadcast",
"toastMessage": "",
"absorbCrash": true,
"developerNotes": "Handle Google IssueTracker bug /245258072 for Android 13"
}
]
Trust but verify 3rd party dependency safety and external integrations
Once our team nearly eliminated crashes from our own business logic, lifecycle, or implementation issues, crashes from third party libraries became a top source of crashes. While we vet the technical quality of dependencies we bring in, the magnitude of users, devices, and device conditions means vendors can’t always guarantee things will be 100% stable. So a general approach on the team is to wrap interactions to third party SDK functions in try/catch blocks and log caught exceptions to Google Firebase Crashlytics via non-fatal logging for observability.
When things do go wrong, it’s important for our team to establish a tight, rapid feedback loop of reporting crashes to our vendor partners so they have visibility on issues, create tickets and prioritize fixes in their products. The State Farm Android team is actually often one of the first companies to surface crashes to vendors, which has a nice benefit of improving stability in the larger Android ecosystem. Reporting SDK crashes helps keep the provider/consumer relationship strong and creates a two-way value proposition.
Aside from library dependencies, it’s also important to double-check and safely handle launches to third party apps for browser destinations, maps, contacts, calendars and more, since Android users are free to disable and/or uninstall any given app.
Ongoing challenges and conclusion
Our app continues to grow in features and users and our technology continues to evolve. The latest evolution of our app was the merger of the Drive Safe & Safe app with the State Farm Mobile App, which brought telematics capabilities and accident detection to our users. New challenges include working with sensors, background data processing and additional partner integrations.
Despite our evolving engineering and testing best practices, it’s always unpredictable what can surface through dependency updates, updates in the Android ecosystem (new OS versions, updates to WebView, Chrome, Maps and more). What remains constant is our team’s ability to adapt and carry a team culture of providing the best possible user experience for our customers, keeping an everyday engineering focus on code safety and stability, and recovering from the unexpected.
To learn more about technology careers at State Farm, or to join our team visit, https://www.statefarm.com/careers
Information contained in this article may not be representative of actual use cases. The views expressed in the article are personal views of the author and are not necessarily those of State Farm Mutual Automobile Insurance Company, its subsidiaries and affiliates (collectively “State Farm”). Nothing in the article should be construed as an endorsement by State Farm of any non-State Farm product or service.
<hr /><p>The State Farm® Android Team’s Journey to 99.9+% Crash-Free Sessions was originally published in State Farm Engineering Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>