WHAT WE LEARNED
FROM ANALYZING
INTRODUCTION
Why did we create this report?

Since Instabug was founded in 2012, we have captured 100 million bugs from mobile apps, along with volumes of comprehensive metadata about 15 billion sessions and 1 billion devices where these bugs and crashes were reported.

This wealth of information about mobile bugs doesn’t exist anywhere else, so we wanted to dig into our data and share these valuable insights with the mobile community. This comes from our mission to empower app developers and mobile PMs to iterate faster, drive five-star reviews with error-free apps, and build products that their users will love. With this report, devs and product managers can measure the performance of their apps against our benchmarks.

SUMMARY | TL;DR

We’re proud to have users who are the best in the business, so we wanted to understand how our customers resolve issues.

  • Errors discovered through Instabug are most likely to be resolved within 24 hours of being reported.
  • More bugs are caught through Instabug in continual builds, reflecting current industry trends that favor more frequent version releases for improved workflow efficiency, greater app stability, and higher app quality.

In addition, we verify when app errors occur across various dimensions. Under which conditions are bugs reported most often?

  • Most bugs are reported from iPhones, while more bugs/user are reported from LG devices.
  • Most bugs are reported from the English language local, while most bugs/user are reported from the Danish language locale.
  • Most bugs are reported from unplugged phones, while most bugs/user are reported from fully charged devices.
  • Most bugs are reported in portrait mode, while most bugs/user are reported in landscape mode.
PRIVACY
Did we analyze any personal information?

No personally identifiable information of any kind was used in this report, which means that all of the data we analyzed is anonymous and cannot be tied to any individual person. These are aggregate trends across 15 billion sessions, 1 billion devices, and 30,000 apps.

We care deeply about our users and we affirm our responsibility to uphold their trust, consent, and confidentiality. We respect the privacy of our users as well as the privacy of their end users, and we take the security of all personal data seriously. Instabug, Inc. is fully compliant with the terms of the EU General Data Protection Regulation (GDPR), effective May 25, 2018.

DATA
What data did we analyze?

This report covers six years of our data, from when Instabug was founded in 2012 and through our journey to reach one billion devices. No personal data was analyzed.

In the charts that follow, you will see "bugs", which represents the total count of bugs in pre-production (beta) and production (live) apps, and "bugs/user", which represents those bugs normalized by the number of app testers and end users who reported those bugs. We show you both so that you can understand the volume of absolute bugs reported through Instabug, as well as their weight relative to the amount of users who experienced those bugs.

In the line charts, you will also see our raw data, represented by dark blue lines, and smoothed data, represented by blue trend lines. Smoothing the data filters out noise, anomalies, and fine-scale differences, showing you the important patterns in the data.

What does this data represent?

With this wealth of data that represents the vibrant mobile app ecosystem, we can draw insights about market trends.

  • Every locale appears in our data, from Afrikaans to Zulu.
  • A range of app publishers appear in our data, including indie developers, side projects, startups, software houses, unicorns, and public companies.
  • Every app category appears in our data, including games, business, education, and lifestyle.
15B
sessions
1B
devices1
100M
bugs
36M
attachments2
50K
Instabug accounts3
30K
mobile apps
2,190
days of data
756
cups of coffee
1 Anonymized data of testers and end users of mobile applications that have the Instabug SDK integrated.
2 Screenshots, audio, and video.
3 Team members invited by companies subscribed to Instabug.
Total Bugs vs. Time

Data based on 15 billion sessions and 100 million bugs in 30,000 apps on 1 billion devices. We take privacy seriously. All data is anonymous.

TIME TO CLOSE
How long does it take to resolve a bug?

Our data shows that issues discovered through Instabug are resolved within the first few days of being reported (Fig. 1). Developers often tackle easier bugs first, which can help explain why more issues are closed when they first appear. In addition, major bugs must be fixed within sprint times, which are often bi-weekly for critical releases, which helps to explain why numbers level off after 14 days.

Within the first 24 hours of an issue being reported, we see the largest proportion of tickets resolved within the first hour (Fig. 2). These are most likely not programmatic bugs, but could be support issues or spam.

    KEY TAKEAWAYS
  • Bugs discovered through Instabug are most likely to be resolved within 24 hours of being reported.
Fig. 1:

Bugs vs. Time to Close (Days)

Data based on 15 billion sessions and 100 million bugs in 30,000 apps on 1 billion devices. We take privacy seriously. All data is anonymous.

Fig. 2:

Bugs vs. Time to Close (Hours)

Data based on 15 billion sessions and 100 million bugs in 30,000 apps on 1 billion devices. We take privacy seriously. All data is anonymous.

APP BUILDS
How often do apps release builds?

Our data shows that the largest share of apps release builds up to a weekly basis, followed by bi-weekly, then monthly (Fig. 3). This reflects the industry’s move toward Continuous Integration and Continuous Deployment, with more frequent version releases, monthly releases occurring less often over time, and shorter app store review times.

We also see more bugs/user reported from apps that release builds up to weekly (Fig. 4). There are a few explanations for this. One is more code, more bugs. Another is that these are often beta builds, which are buggier by nature given quick turnaround times. In addition, apps that iterate weekly often include new features, which likely contain bugs, or bug fixes, which are constant.

    KEY TAKEAWAYS
  • Most apps release builds up to a weekly basis.
  • More bugs/user are reported from apps that release builds up to weekly.
Fig. 3:

App Build Frequency

Fig. 4:

Bugs vs. Build Frequency

Data based on 15 billion sessions and 100 million bugs in 30,000 apps on 1 billion devices. We take privacy seriously. All data is anonymous.

INTEGRATIONS
What are the most common tools used with Instabug?

Our data shows that 70% of our plan subscribers integrate third-party tools with Instabug in order to forward issues to their other workspaces. When we look at the top 15 integration combinations, we can gather insights about our users’ workflows (Fig. 5). The most common integrations with Instabug are Jira, Slack, Trello, GitHub, and Zendesk.

Through discussions with our users, we know that multiple teams in one company often collaborate through Instabug. For example, the customer support team can use the Instabug + Zendesk integration for feedback, the product team can use Instabug + Trello, and the engineering team could forward crashes with Instabug + GitHub to track in the code repo, while Instabug + Jira could be used for bugs. Two different development teams in the same company could also have different backlogs on different project management platforms, with both integrated in the same Instabug account.

    KEY TAKEAWAYS
  • The most common integrations are with Jira, Slack, Trello, GitHub, and Zendesk.
Fig. 5:

Most Common Integrations Used With Instabug

Data based on 15 billion sessions and 100 million bugs in 30,000 apps on 1 billion devices. We take privacy seriously. All data is anonymous.

BETA PROGRAMS
How responsive are beta programs?

We took a sample of our customers who use Instabug in beta versions of their apps based on their use of our “beta” token and build names. We wanted to know how responsive they were to errors. We found that crashes reported through Instabug are most often fixed when they occur (Fig. 6). If they aren’t fixed right away, these issues are less likely to be closed.This reflects developers being very proactive when it comes to resolving crashes and being more tolerant when it comes to business logic bugs.

We also wanted to know how responsive beta programs were to in-app chats sent from testers, and we found that messages are most often replied to within the first 24 hours (Fig. 7). This reflects beta testing best practices, which recommend actively engaging with testers in order to maximize their participation in the beta program and keep them invested in the product.

    KEY TAKEAWAYS
  • Crashes reported through Instabug are most likely to be resolved within 24 hours.
  • In-app chats sent through Instabug are most often replied to within 24 hours.
Fig. 6:

Beta Programs: Average Time to Resolve a Crash (Hours)

Data based on a representative sample of beta apps in February 2018. We take privacy seriously. All data is anonymous.

Fig. 7:

Beta Programs: Average Time to Respond to Chats (Hours)

Data based on a representative sample of beta apps in February 2018. We take privacy seriously. All data is anonymous.

DEVICES
Which manufacturers have the most bugs?

Our data reflects the industry-wide fragmentation of Android devices across multiple manufacturers and OS versions. Meanwhile, all versions of the iOS platform are concentrated in iPhone devices and more iOS apps have the Instabug SDK integrated than Android apps, which is why we see the highest count of bugs reported from iPhones (Fig. 8) .

When we normalize our data by the number of app testers and end users who reported bugs from those devices, we might expect Samsung to hold first place according to global market share. However, LG takes the top spot in our data (Fig. 9) . This could be explained by our technical user base and the popularity of Nexus devices (some versions were manufactured by LG) with Android developers. The high proportion of bugs/user reported from LG phones is mostlikely due to internal beta testing by developers.

    KEY TAKEAWAYS
  • Most bugs are reported from iPhones.
  • Most bugs/user are reported from LG devices.
Fig. 8:

Bugs vs. Device Manufacturer

Data based on 15 billion sessions and 100 million bugs in 30,000 apps on 1 billion devices. We take privacy seriously. All data is anonymous.

Fig. 9:

Bugs/User vs. Device Manufacturer

Data based on 15 billion sessions and 100 million bugs in 30,000 apps on 1 billion devices. We take privacy seriously. All data is anonymous.

LOCALES
Which locale has the most bugs?

When we look at the count of bugs reported across different locales, the vast majority were reported from English-language app versions (Fig. 10) .

However, when we normalize our data by the number of app testers and end users who reported bugs from those locales, Danish edges out English to take first place by a small margin (Fig. 11) .

    KEY TAKEAWAYS
  • Most bugs are reported from the English language locale.
  • Most bugs/user are reported from the Danish language locale.
Fig. 10:

Bugs vs. Locale

Data based on 15 billion sessions and 100 million bugs in 30,000 apps on 1 billion devices. We take privacy seriously. All data is anonymous.

Fig. 11:

Bugs/User vs. Locale

Data based on 15 billion sessions and 100 million bugs in 30,000 apps on 1 billion devices. We take privacy seriously. All data is anonymous.

BATTERY
How does battery affect app stability?

While most bugs were reported from devices that were unplugged (Fig. 12) , representing the typical use case, when normalized, more bugs/user were reported from devices with full battery (Fig. 13) . This is most likely due to our customers with large tester and user bases whose devices are always plugged in and fully charged.

That explanation also applies to the large spike of reported bugs we see at 100% battery level (Fig. 14) . When normalized, however, we see that users report bugs at about the same rate across all battery levels (Fig. 15) . We also see an anomalous spike at 5% battery level. Many apps restrict resource use at low battery levels in order to reduce battery consumption, so it’s not surprising to see more reported bugs/user at low levels.

    KEY TAKEAWAYS
  • Most bugs are reported from unplugged phones.
  • Most bugs/user are reported from fully charged devices.
Fig. 12:

% Bugs vs. Battery Charging State

Fig. 13:

% Bugs/User vs. Battery Charging State

Data based on 15 billion sessions and 100 million bugs in 30,000 apps on 1 billion devices. We take privacy seriously. All data is anonymous.

Fig. 14:

Bugs vs. Battery Level

Data based on 15 billion sessions and 100 million bugs in 30,000 apps on 1 billion devices. We take privacy seriously. All data is anonymous.

Fig. 15:

Bugs/User vs. Battery Level

Data based on 15 billion sessions and 100 million bugs in 30,000 apps on 1 billion devices. We take privacy seriously. All data is anonymous.

ORIENTATION
Which UI orientation has more issues?

When it comes to UI bugs, we see a far greater count of bugs reported from portrait mode than landscape (Fig. 16). Most people use apps in portrait mode and many apps don’t support landscape mode.

During development, portrait mode is typically prioritized and landscape mode often isn’t optimized. This is reflected in our normalized data. We see a greater share of bugs/user reported from landscape mode (Fig. 16).

    KEY TAKEAWAYS
  • Most bugs are reported in portrait mode.
  • Most bugs/user are reported in landscape mode.
Fig. 16:

% Bugs vs. Orientation

Fig. 17:

% Bugs/User vs. Orientation

Data based on 15 billion sessions and 100 million bugs in 30,000 apps on 1 billion devices. We take privacy seriously. All data is anonymous.

IOS VS. ANDROID
Which OS has buggier apps?

In the global mobile market, far more devices run Google’s Android operating system than Apple’s iOS. Meanwhile, most users adopt the latest version of iOS, while Android users are fragmented across older OS versions and different devices. Thus, it would be reasonable to expect more Android bugs reported than iOS bugs.

However, our data shows a greater count of iOS bugs than Android bugs because the Instabug SDK is integrated in more iOS apps than Android apps (Fig. 18) . This reflects our customer base, most of whom are located in markets where iOS dominates. When normalized by the number of app testers and end users who reported those bugs, we see that the proportion of reported bugs/user is about the same for both platforms (Fig. 19) . Although more iOS users reported bugs through Instabug than Android users, iOS users reported bugs about as often as Android users.

    KEY TAKEAWAYS
  • Most bugs are reported from iOS.
  • Reported bugs/user are about the same between iOS and Android.
Total Bugs vs. Time
Fig. 18:

% Bugs vs. OS

Fig. 19:

% Bugs/User vs. OS

Data based on 15 billion sessions and 100 million bugs in 30,000 apps on 1 billion devices. We take privacy seriously. All data is anonymous.

MEMORY
How does memory affect app stability?

Our data shows that more bugs are reported at higher levels of memory usage (Fig. 20) . Between 99-100% memory usage, we see a steep drop in the number of bugs reported, which reflects apps being terminated when devices run out of memory.

When normalized by the number of app testers and end users who reported those bugs, we see that bugs are reported more often at high levels of memory usage (Fig. 21) . Meanwhile, bugs reported at low levels of memory usage are probably due to testing.

    KEY TAKEAWAYS
  • More bugs are reported at higher levels of memory usage.
Fig. 20:

Bugs vs. Memory Usage

Data based on 15 billion sessions and 100 million bugs in 30,000 apps on 1 billion devices. We take privacy seriously. All data is anonymous.

Fig. 21:

Bugs/User vs. Memory Usage

Data based on 15 billion sessions and 100 million bugs in 30,000 apps on 1 billion devices. We take privacy seriously. All data is anonymous.

DISK STORAGE
How does disk storage affect app stability?

At higher levels of disk storage usage, we expect to see more errors, which is reflected in our data (Fig. 22). Many apps don’t handle when they attempt to do operations and they fail because of low or full disk space, so they either crash or engage in unexpected behavior that results in bugs.

When normalized by app testers and end users who reported issues, we see that bugs are reported at about the same rate across all levels of disk storage usage, with a slightly downward trend (Fig. 23). Bugs reported at low levels of disk storage or high disk space are likely due to testing.

    KEY TAKEAWAYS
  • More bugs are reported from devices with low disk space.
Fig. 22:

Bugs vs. Disk Storage Usage

Data based on 15 billion sessions and 100 million bugs in 30,000 apps on 1 billion devices. We take privacy seriously. All data is anonymous.

Fig. 23:

Bugs/User vs. Disk Storage Usage

Data based on 15 billion sessions and 100 million bugs in 30,000 apps on 1 billion devices. We take privacy seriously. All data is anonymous.

SESSION DURATION
When do crashes occur?

Our data shows that most crashes happen at app launch (Fig. 24). The large spike of crashes that occur within the first five seconds of start up includes launch crashes that occur during development and in production apps. These crashes at launch affect more users than crashes that occur at other times.

    KEY TAKEAWAYS
  • Most crashes occur at app launch.
Fig. 24:

Crashes vs. Session Duration (Seconds)

Data based on 15 billion sessions and 100 million bugs in 30,000 apps on 1 billion devices. We take privacy seriously. All data is anonymous.

THANK YOU

The creation of this report was made possible by a handful of talented people across different teams at Instabug. However, the data itself was made possible through the outstanding work of the entire BugSquad over the course of six incredible years. Most importantly, none of it would have been possible without the thousands of amazing companies that we're proud to call our users. Thanks to all of you. 🙏

ABOUT INSTABUG

Instabug empowers mobile-first companies to iterate faster and enhance their app quality by providing intuitive user feedback channels, and comprehensive bug and crash insights.

Join the world's top companies who rely on Instabug for bug reporting, crash reporting, and in-app feedback.

Thanks Logos