Beta Testing in Big vs. Medium vs. Small Companies

One of the biggest challenges of beta testing is that there are no established standard processes. The main reason for this lack of standard procedure is that optimal beta testing processes tend to vary wildly with differing scenarios.

At Instabug, we have helped hundreds of companies shape their beta testing programs. Today we will take a look at examples of a successful beta test process and the best practices in use by big, medium, and small companies.

‍

‍

Big companies are usually at an advantage when it comes to beta testing. Many of these companies have a well-established beta program that has been running for several years. They've had the resources and opportunity to iterate and adapt their programs to their needs.

‍

Team

At their size, it makes sense for big companies to hire a dedicated team for their beta program. Big companies also tend to have multiple products in development that need beta testing. The beta team can sustain an ongoing beta program that consists of several beta testing projects.

The size of the beta program team is, of course, directly proportional to the size of the beta program. The team is headed by a beta program manager who owns the beta program and is responsible for setting the goals, strategy, and process of the beta program and overseeing the progress of the individual beta projects that might be running.

The beta program manager leads a few beta test managers/engineers who are responsible for setting up the individual beta tests. They are responsible for planning individual beta tests and oversee the handling of all the test details, like tester selection, communication, issue verification, etc. If the company usually has only one running beta test, these responsibilities are consolidated under the beta program manager's role.

The rest of the team consists of one or more members to handle QA, support, community management, and marketing. The total size of the team varies according to the number of testers participating in it.

‍

Tools

To handle the complexities of a sophisticated beta test process, big companies use specialized tools for the beta test's needs. The beta program manager chooses the appropriate tools that fit the needs and process of the beta test.

Their arsenal usually includes tools for app analytics, in-app feedback and reporting, issue tracking, customer relations management (CRM), and app distribution. Some of these tools like bug trackers and CRM platforms might already be in use by different teams, but they can opt for using a different tool if it is more suitable. In any case, the tools chosen for the job should be able to seamlessly integrate with the rest of the tools used by the company.

At the scale found in big companies, tools are essential to successfully manage the beta test and maximize its benefits. In-house tools are sometimes used by the mega size companies or those who have very specific and specialized needs. However, third-party tools are almost always preferred due to the resources and time needed to develop and maintain an in-house tool that can satisfy their needs.

‍

Recruitment and community building

Arguably, the biggest advantage working for big companies is their big user base from which they can recruit testers. The availability of a huge pool of users to recruit from eliminates the step of finding a representative sample. The beta team examines the app analytics available to them to sort their users into several segments from which they can recruit a representative sample.

Generally, "power users" with heavy, regular usage, who have shown readiness to communicate are invited first. However, power users alone do not make a representative sample and the beta team will include testers from different segments to ensure they represent all their personas. To make sure the sample represents all possible use cases, the number of testers recruited runs into tens of thousands.

The beta testing team's job does not end with the recruitment of the testers. They maintain constant communication with the testers and update them regularly to ensure they're engaged. Moreover, they provide incentives to active participants in the test and organize physical meetings and events to personalize their relationship and increase the sense of ownership. On the other hand, they keep an eye on testers and their participation to monitor their activity. They identify testers who have burned out and are no longer active and remove them from the program.

Building and maintaining an active community is essential for big companies to interact with testers and keep them interested. These companies have an established community of interested users with which they are constantly communicating even if there are no ongoing beta tests. As a result, they foster a relationship that makes it easier to recruit testers who will provide high-quality feedback. Moreover, the community provides a pool of testers from which they can recruit for future iterations or newly developed apps. The number of beta testers involved at this scale is usually in the thousands of testers because of their massive user bases.

‍

Beta test process

The beta test starts with a meeting of all the stakeholders as soon as a sufficiently stable beta build is ready. The purpose of this meeting is to ensure that everyone is aligned with the beta test's process and goals. These meetings are held regularly throughout the beta test to update all stakeholders on the progress of the beta test and gather feedback and suggestions from the rest of the teams. Moreover, the beta team holds frequent meetings to proactively monitor the beta test and take corrective actions if needed.

Big companies tend to have shorter, time-boxed beta test cycles with multiple beta products being tested. To achieve this they use several techniques and tools that maximize their ability to iterate quickly like test automation, continuous integration and deployment, and feature flags. Moreover, they will have an internal beta test or dogfooding program to whom they release nightly builds to be tested before they are made available to external testers.

A typical beta test process at large companies consists of releasing nightly builds to internal testers or dogfooders. When the beta team deems the nightly builds stable, they release weekly beta builds to external testers and a bi-weekly or monthly production build to the app users. This schedule is shorter than the usually recommended six-week minimum, but is not set in stone and differs from one company to the other.

‍

Release and rollback

To sustain the rapid iterations on the beta build, big companies need a sophisticated and flexible release process. This makes the use of feature flags and canary releases a sensible addition to the process.

When a beta build is ready for testing, it is released to the developing team to ensure stability and catch obvious bugs. Next up, the build is released to internal beta testers for thorough testing before pushing it to external beta testers. Once external testing is done, the app will be released to users in a staggered fashion.

The production build is first pushed to a small percentage of users to make sure that any issues that might have slipped do not affect a large user base. Following the build's success, the percentage is increased until all users are on the build, usually within one week.

The use of feature flags makes these staggered releases possible through enabling or disabling the beta features for certain users. More importantly, if a release encounters any major issues, rollback is as simple as toggling off the respective feature flag. This is especially important for apps since app stores' approval processes for updates aren't immediate, essentially preventing the app from instantly dealing with the issues.

‍

Reporting and post-mortems

Throughout the beta test, the team keeps an eye on several metrics to measure the health and progress of the test. This allows them to detect issues even before they happen and handle them accordingly. They also prepare several reports for each team to update them on the progress of the test and highlight the issues that need to be prioritized.

Tracking metrics like fixed vs. reported bugs, the average time to fix, build adoption, and tester participation sheds light on the quality and success of the beta build. If the metrics indicate any problems, they can proactively investigate the issue and handle it before it causes any harm.

The stakeholders hold a meeting to dissect and discuss the results of the test after the beta test is complete. These post-mortems investigate issues that might have happened during the test and identify successful processes and techniques already in use. As a result, the company can always identify issues with its process and is able to continuously improve its techniques.

‍

‍

Beta testing programs in medium companies are often young or newly started projects. They might have not had the same opportunity to improve and iterate on their beta program as much as bigger companies have, but they are better off than small companies when it comes to available resources, while still retaining much of their flexibility and speed.

‍

Team

More often than not, medium companies do not have a dedicated beta test team but divide the tasks of the beta test between a few teams. A growing number hire a beta program manager as the owner of the beta program to design and manage the tests and coordinate between the teams. Otherwise, it will usually fall under the product, QA, release, or support manager.

Most of the time, the presence of a beta program manager ties to the maturity of the beta testing program. As the beta program grows, the benefits of having a dedicated person to own the beta test become clear. In any case, the owner of the beta test coordinates between QA, engineering, support, and marketing teams and makes sure everyone has good visibility over the progress of the beta tests.

‍

Tools

To deal with the disadvantages of not having a dedicated beta team, medium companies need to leverage tools that will facilitate and streamline their beta test process. Choosing the right tools for their needs and leveraging their use can make or break a medium company's beta test.

CRM and communication tools are essential to keep constant communication with testers and keep them engaged. Tools for in-app crash reporting and bug reporting are used to make bug reporting easier for testers and more structured for developers. A bug tracking tool is then used to track these bugs through their lifecycle and analyze their trends. The company's project or issue management tools can be used for bug tracking if they support this function.

‍

Recruitment and community building

While they might have a sizeable user base, medium-sized companies have a fledgling or young beta testing community. This makes it more difficult to find a representative sample of all their personas. As they continue to see the value in the beta testing program, the benefits of having an engaged beta community become clear.

Building a beta community is a priority and they are constantly reaching out to their most active users to involve them in the beta program. The team tries to establish a rapport with the beta community through frequent communication and being responsive to their feedback. On the other hand, they try to keep the number of testers below the limit that they can handle. The number of beta testers involved at this scale is usually between 200 and 1,000 testers, but the ideal number of testers needed will depend on the app's total active users.

‍

Beta test process

At this scale, companies typically reserve beta tests for significant updates and don't usually hold one for versions that have mostly bug fixes and no important changes to the code.

When a beta candidate is identified, the owner of the beta test will hold a meeting with the teams involved to set the goals of the test and the responsibilities of each member. Once everyone is aligned, they start by releasing the build to internal employees to dogfood it for a few weeks. When the issues found by dogfooding are resolved, the build is pushed to external testers.

The beta test usually takes between six to twelve weeks, throughout which the team will be releasing weekly or bi-weekly beta builds. These iterations are time-boxed to control the duration of the beta. Regular meetings between stakeholders are held in that period to provide updates on the beta's progress and monitor its success.

‍

Release and rollback

While the use of feature flags among medium-sized companies is growing, many of them do not use this technique yet. This makes the release process a little trickier since quick rollback is not possible. Since the app store approval process makes instant rollback impossible, an updated version that fixes the issue becomes a better option than rollback. This is one of the reasons for using a longer beta testing cycle to make sure that no issues slip through to production.

Staggered releases can still be used without feature flags through the Google Play console or iTunes Connect through the recently introduced features of "staged rollout" and "phased releases" respectively.

After the initial release of the production version, the team prepares for the issues that come up to quickly release a fix within a few days. According to the rate of issues being reported, the team may need to follow a weekly update schedule until the rate is under control. When the rate and severity of issues are down to an acceptable level, they revert back to bi-weekly or monthly releases.

‍

Reporting and post-mortems

As with big companies, the owner of the beta program tracks key metrics for the beta test and prepares regular progress reports for the stakeholders to keep them updated. After the beta test is completed, the beta owner prepares another detailed report of the overall test and a post-mortem meeting is held to evaluate it.

Medium-sized companies might not have accumulated the same amount of data as big companies, but they can still leverage the data they have to improve their beta test process. In fact, this post-mortem analysis is doubly important for them because their beta testing program has not matured yet. It allows them to quickly identify issues with their beta test process and improve and streamline their workflow.

‍

‍

The biggest issue facing small companies is usually the limited resources available to them. Therefore, adopting a simple, flexible beta testing process is key to the success of small companies.

‍

Team

Small companies typically don't have a dedicated beta team or even a dedicated beta manager. The beta test's tasks are distributed among the teams according to their scope. While there sometimes is no clear owner for the beta test, it usually falls under QA or product management.

‍

Tools

For small companies with limited resources and headcount, tools add great value and make a beta test easier and more efficient. The tools in use must be restricted to the goals and focus of the beta test and priority is given to simple flexible tools that can be adapted for different uses.

Regardless of the beta test's focus, tools that facilitate communication and feedback must be utilized. Without excellent communication with the testers, it is very difficult to get the quality feedback needed. In-app tools for automatic crash reporting and bug reporting increase the number and quality of received reports drastically.

To track the bugs crashes and feature requests, small companies can get away with a simple spreadsheet. If the team uses a project management tool or any other tool that can be adapted for issue tracking, the team gets better results from tracking their bugs and issues.

‍

Recruitment and community building

This is one of the biggest challenges facing small companies because they do not have a large enough user base or community to recruit from. Additionally, community building and management need a lot of effort, dedication, and continuous nurturing.

With some effort, getting a suitable number of testers is not very difficult, but you will rarely get enough to segment and choose a representative sample. Testers are first recruited from the app's active users if there are any, then through reaching out to communities of beta testers or early adopters on sites like Reddit and Twitter. There are also sites that connect makers to early adopters and testers, where they can find testers.

For small companies with small user bases, the number of beta testers is usually somewhere between 50 and 200 testers.

Building and maintaining a community might not be possible for small companies, but they try to maintain a good relationship with their testers by responding to their feedback and updating them on the progress of the issues they reported. After the beta test ends, occasional contact with updates on the apps progress and new features helps the app stick in their minds and keeps the relationship from fizzling out.

‍

Beta test process

This is where it gets a little fuzzy. The beta test process in small companies varies a lot depending on the team size, user base, and app complexity. Generally speaking, their process is essentially the same process as a medium-sized company, but carried out over a longer time period.

As usual, the stakeholders hold a meeting before the beginning of the beta to determine the goals and focus of the beta test. The beta build is first pushed to friends and family for preliminary testing, then released to the external testers. Regular meetings are held to monitor the progress of the test, but these meetings are very short due to the relatively small number of testers and size of the team. The pros of having a small team are flexibility, which allows them to pivot easily, and ease of syncing between members, which helps them to stay aligned.

The duration of the beta test will usually be around the upper bounds of the six to twelve-week range, but will frequently exceed it. In small companies, the duration of the beta test is not time-boxed and is determined by the time it takes the team to work through all the issues generated by the beta test. With this approach, beta tests take somewhere between three to six months but can be much shorter with an efficient process.

‍

Release and rollback

The final release of the production version is a critical stage for small companies. The absence of a mechanism for quick rollback and the time it takes for the team to fix and release any issues makes it crucial for the release to be free of hitches.

This is why they do not time-box their beta tests with a time limit. They try to make sure that they have discovered every possible issue and are confident in the app's quality.

Since the use of feature flags at this scale doesn't make sense for most cases, quick rollback is practically impossible. Any issue encountered in production might have to wait for up to 48 hours for the fix or rollback to go through the app store's approval process.

‍

Reporting and post-mortems

As with the rest of the cases, the owner of the beta test will prepare a report on the beta's results and metrics. However, since small companies usually limit the number and scope of tools they use, they do not have access to extensive metrics and analytics.

Post-mortems help improve the process and the team will use the available metrics to identify inefficient workflows and areas for improvement. Since a big chunk of their beta process is ad-hoc, these metrics are not as helpful as when there is an established process and historical data to compare it to.