How Verizon Uses Instabug to Securely Achieve 99.9% Crash-Free Sessions

Verizon, the largest telecommunications company in the US, offers consumers the ability to sync and back up their content across multiple devices with the Verizon Cloud app.

For Mohammad Hariri, DMTS for Verizon’s Communications and Cloud division, maintaining the highest standards of application health is crucial — he knows that consumers will quickly abandon an app that crashes. As an enterprise company, Verizon also requires the highest standards of data security and consumer privacy.

So he turned to Instabug for crash reporting and application health monitoring and we provided dedicated, single-tenant hosting in a private cloud to meet Verizon’s security needs.

Below, Hariri shares the outstanding outcomes of integrating the Instabug SDK into the Verizon Cloud app.

Objective

Because we are a telecommunications company we are under a lot of regulation controls in terms of subscriber privacy, which makes this a bit tricky.

Initially, we started looking at Crashlytics, which I’ve used in the past with other companies, and once I started going down that path it became clear that Verizon’s Chief Information Security Office had a lot of concerns about using a free, public SDK since we did not have control over the data and how it’s being used and managed.

From a security perspective, we needed to make sure that there are no customer data exposed to anybody at the end of the day. That is the biggest concern, and the example that our security uses over and over and over is back in 2011, there were some customers’ passwords that were stored in a log file in Crashlytics, and so you could just look at a log and see their password, which is a big no no.

Now with respect to monitoring, we always wanted to get a better understanding of what is happening with the health of our application and over time. There is a limited set of reporting from Google Play Store and Apple App Store, which you can use, but it’s not easily shared or managed, and we were not able to compare these two app stores’ data because they were in different domains.

Challenge

Prior to Instabug, we had to go through the different app stores to get information, but Google Play and App Store data is not in a user-friendly format and not shareable.

A lot of our development is not done in-house, it’s done with outside vendors, and that makes it even more complex, because you cannot provide a third-party vendor with access to Verizon’s app store accounts.

So what we were doing prior to this was copy and pasting, downloading things, and sharing, which was not really flexible. It wasn’t something that we could easily manage — from day to day it was different, from version to version it was different, and the process was not repeatable.

So being able to limit access only to relevant data and provide a tool to track app issues through Instabug, that really helped us, plus the ability to see iOS, Android, and all that in one place is a great benefit.

If you look at our crash rate over the last two to four years, the trend was down, we were making headway, but not as fast as we wanted to, and Instabug provided us the ability to make a major impact in a rather short time.

We were continuously looking at crashes and the management of those is just a small part of what we do. There are a lot of efforts in UI/UX, new features, there’s a lot of different activities that goes on with respect to Verizon Cloud. Yes, we were making headway, but how much effort goes into that? And then how reliable is that or how repeatable is that process?

So I would go to some engineer and say, ‘Every day I want you to go and copy and paste this and then send it to this other guy, our vendor’ — that may not happen on a regular basis, that may not be taken seriously, there are a number of reasons why that process wasn’t optimal.

So we were like, hey, let’s do something about it.

We need to make this product as good as it can be and so that’s how we came across Instabug.

Solution

Because I was pushing hard for some sort of monitoring tool, Verizon’s security office asked me to identify a product to use for application health monitoring. So I did some work and I came across Instabug. The other product we considered is a Verizon product, but once we tried them side by side, it was obvious that the other tool was not really meant for crash reporting and it did not have all of the features and abilities of Instabug such as communication with the customer, integration with Jira, so on and so forth.

So after building a little app and trying it out, we decided to go with Instabug. I made that recommendation to the security office and they were very thorough in validating Instabug. Meanwhile, we started getting our vendor and developers involved and see what it takes to integrate the SDK into our product. And the integration was really easy. There was really no hiccups to the best of my recollection.

For us, the ability to have our own standalone backend services that’s not shared with anybody else was a big plus. Plus the ability to easily share data with our vendor and developers.

When you look at our configuration here with the current setup with Instabug, we have our own private instance with Instabug, we have a lot of features, we have the ability to turn off and handle it privately, and it’s not exposed to Google indexing and all that stuff.

The privacy and the ability to manage and control the data is what we were looking for, which Instabug provided.

A lot of times with vendors, and established vendors especially, there’s a lot of back and forth and you go through a lot of hoops to get what you need. In this case, part of the success was caused by the fact that we had open dialogue and communication between our developers and the Instabug team, there was no middle man and they could just talk and Instabug was responsive. They got back to us quickly with fixes and anything we were looking for and that was a great advantage.

The key here was that the Instabug team really were responsive to our requests.

Results

We have various metrics that we report up and down the chain, in terms of the health of the application, performance, revenue, subscriber counts, we have about 267 metrics, and one of them is app crashes. We use this data and share it. This last result was such a big, major accomplishment that it went all the way up the chain in our weekly report and we had a lot of positive comments all the way up to the top as far as our results.

Once we instrumented the app with Instabug and made it available to developers to have access to it, they were able to quickly fix issues.

Prior to Instabug, we had about an average of 98.5% crash-free sessions. Once we instrumented the application in 19.4 version of Verizon Cloud, we were able to have a clear view of the app’s issues. We saw a number of minor, but quite a few issues, and we found that most of the crashes were happening on the backend, so it wasn’t something that was visible to the customer. We were able to get the developers to quickly work on fixes and we released 19.5 version a month ago.

Because of Instabug, we are now 99.9% crash free on Android. And on iOS we dropped by a factor of five. And this was the first go around. The application became very stable and quite robust because it has Instabug to track and measure and manage crash data.

‍

With every release, developers go in, check out the results, and take action. Bug fixes are part of our standup process, so you talk about which bug you’re going to fix and how you’re going to decide what to fix.

In Instabug, we can look at it by instances of crash or number of users impacted, so it helps us with the targeting of bugs to fix.

Developers are looking at this data on a regular basis and they address them consistently, so I think we’re going to get better over time. With 19.6, there were a couple of other things that they found and fixed.

In our next version we’re starting to integrate user feedback in the 19.7 release of iOS and 19.8 on Android, where we’re going to use the Instabug feedback mechanism to collect customer feedback. We believe by doing that, we can improve our App Store rating. Our App Store rating is really good, 4.67, but I believe by providing a means for customers to provide feedback, that will help with getting a direct response from customers and allowing them to feel that someone is listening to them rather than just dumping something on the App Store.

‍