Where can I learn more about A/B testing?
Posted: Wed Jan 22, 2025 10:10 am
Product teams strive to create a product that the user will definitely like and find convenient. This requires making dozens of decisions. For example, where to place the button, what kind of interface to make, and what phrases will influence the purchase decision.
You shouldn't rely on subjective taste; you need to test options. A/B tests will help with this, with which you can check the effectiveness of solutions when developing a product.
Difficulty: easy
Reading time: 7 minutes
What will be in the article:
What are A/B tests and how to conduct them?
Who needs to understand A/B testing?
When should you not use A/B testing?
A collection of tools for setting up A/B tests and resources for self-study
What is A/B testing?
A/B testing is a tool that helps teams test hypotheses and make decisions based on data rather than gut instinct. The idea is to split an audience into parts and show them different versions of something to see which version is better.
You can test the landing page design, the color and location of the call to action button, the registration form, the design of the email newsletter, the text of the ad on the website, and other changes.
A/B testing of newsletter subject lines
A/B test results show which solution will provide a higher conversion rate to the desired target action. For example, in which case more users will click on a link, register on a website or in an application, subscribe to a newsletter, fill out a feedback form. There are also more complex tests that are aimed at studying long-term metrics, such as the average check or the impact of changes in the product on profit.
To be sure of the reliability of the results, it is better to conduct the test with a control sample. To do this, you need to divide the audience on which the variants will be tested not into two groups (A and B), but into three (A, A and B). Show one version to two groups, and another to the third. This will reveal whether external factors influenced the test result and whether there were any errors in collecting metrics.
Imagine you are testing a subscription form on a store website.
At this time, another department launched an advertising campaign to promote a certain product in the catalog, and many users came to the site. They want to buy now and are not interested in subscribing at all, so they can distort the results of the experiment.
The control sample will show whether the A/B test participants were distributed evenly across groups and whether the advertising campaign affected the results.but whether the A/B test participants were distributed into groups, and whether the advertising campaign influenced its results.
External factors did not affect the result, and the data can be trusted if the indicators of groups A and A do not differ. If they do, the numbers are distorted, which means that you should not make a decision based on the test.at, and the data can be trusted if the indicators of groups A and A do not differ. If they do, the figures are distorted, which means that you should not make a decision based on the test.whether they differ - the numbers are distorted, which means you shouldn’t make a decision based on the test.
The principle of A/B testing with a control sample
What if you need to test more than two options? For example, test four application formsneed to check moree two options? For example, test four application formsname, test hfour application forms
If you have more than two options, you can conduct multivariate testing.
The principle is the same as in A/B testing, only more than two versions of one change are compared simultaneously. A part of the audience is allocated for each option for display, at the end of the test their results are compared. The version that showed the best metrics wins.
Multivariate testing
A multivariate test is best used to test several versions with minor changes. For example, you can test four call to action phrases for one button.
This method is also useful for testing the metrics of different combinations. Let's say you have four button text options and two colors. Using a multivariate test, you can compare eight possible versions and see which combination performed best.
We discussed in more detail how and why to test creatives and their variations in the article about visuals in SMM and paid traffic .
As with A/B testing, the multivariate method can be used to analyze not only the conversion of the first target action, but also to evaluate the subsequent behavior of users.
Let's say a red button with the text "Buy at a discount" will collect the most clicks. But it may turn out that the conversion to purchase for this combination is two times lower than for a less clickable green button with the text "View catalog". The winner is not always the most obvious combination.
Is it possible to conduct several experiments in parallel?
Yes, you can. But with restrictions.
The samples of two parallel tests should not be mixed. Otherwise, one change may affect the perception of the second.
How to and how not to conduct a parallel test
Let's practice determining in which cases parallel A/B tests can be conducted and in which they cannot.
Exercise
Here are solutions for improving the online lecture website. Select from the list those solutions that can be tested in parallel.
The changes that are proposed to be tested in parallel are within one answer option.
Please select multiple options
1. The "Enable subtitles" button in the video lecture and the call to action text on external traffic sources
2. The location of the menu buttons and visual design on the main page of the site
3. The location of the button for going to your personal account and the visual design of creatives for the advertising campaign
4. Filters of lecture catalog sections and its visual design about visual design
5. Subscription widget on the main page and the color of the "Place an order" button in the cart color of the "Place an order" button in the shopping cart
6. Form for questions to the lecturer and the button "Turn on subtitles" Turn on subtitles»
Parallel tests can be set up on the frontend side by tagging events with different tags for different versions of the interface. Or you can use special tools the frontend side, marking events with different tags for different versions of the interface. Or you can use special toolsthat simplify this process,forgive this process, such as Google Optimize .
Guide to the best skillsetter articlestatiam skillsetter
Our blog already has over 150 articles about product growth and careers in IT. For easy navigation, we have combined them into thematic collections.over 150 articles about product growth and careers in IT. For easy navigation, we have combined them into thematic collections.
Select a collection
Who needs to understand A/B testing?
At a basic level, everyone involved in working on the product needs to understand it. This will help choose the best research methods for different tasks and at different stages of work.
A deeper knowledge of A/B testing is needed by a product manager, marketer, and analyst.
The analyst participates in test setup: forms requirements for launch, defines metrics, checks the correctness of the operation, and analyzes the results after completion.
Product and marketers need to understand what they can test, how to conduct an experiment, and analyze its results.
Marketers use A/B testing to attract more people and better convert them into a purchase. Most often, they conduct experiments when working with email newsletters, push notifications, landing pages, and advertising creatives.
Product managers primarily use A/B testing to improve product metrics. For example, they might compare two versions of onboarding, different versions of product features, and their usability for users.
When working on a product, the best option is teamwork at all stages, including during testing of ideas.
Let's see how this looks in practice. A product manager conducted a survey of users of an online lecture hall. He found out that lectures in English are difficult to perceive by ear because of the special terms. The team gathered for a brainstorming session, during which the idea of adding subtitles to video lectures appeared.
After that, the designer drew two versions of the subtitle button. The analyst prepared the parameters for collecting metrics for the A/B test and the technical task for the developer. The developer wrote the code for the interface. The analyst set up the testing system, launched the test, checked the quality of the data, assessed the statistical significance of changes in the metrics, and made a report. The report was discussed at a team meeting, and based on the test results, they decided together which button to add to the site.
What to do if there is no analyst on the team?there is no analyst in the team?
You can prepare and conduct an A/B test without an analyst.without analyst.
Built-in A/B testing features are available in advertising campaign setup services, email and push notification tools, and some website builders, such as I have services for setting up advertising campaigns, tools for email newsletters and push messages, as well as some website builders,For example,Wix .
To set up an A/B test, you can also use special tools:use special tools:
Google Optimize . The tool has an intuitive interface. You can calculate the main metrics, and after the A/B test, upload the data to Google Analytics to analyze the results.o clear interface. You can calculate the main metrics, and based on the results of the A/B test, upload the data to Google Analytics to analyze the results.
VWO : This tool supports multivariate testing and URL split testing. You can create visual versions without changing the code via the built-in editor.Variant testing and split testing of URLs. You can create visual versions without changing the code via the built-in editor.
Unbounce . The tool is suitable for basic A/B testing. There is a visual designer and a visual editor.A/B testing. There is a visual designer and a visual editor.
Optimizely . The tool supports multi-page tests. There is also a visual editor to create versions without changing the code, optimization for mobile applications is available.rcii without changing the code, optimization for mobile applications is available.ization for mobile applications.
Optimizely Interface
We recommend reading other articles from this series:
10 Embarrassing Questions About Promotion in Google Play and App Store
10 Embarrassing Questions About Unit Economics
10 Embarrassing Questions About No-Code
10 Embarrassing Questions About SEO
10 Embarrassing Questions About Paid Traffic
Free course on product management
In 3 days you will learn to think like a product and cope more successfully with tasks at work and in life
Start learning for free
When should you not use A/B testing?
A/B testing should not be used in several cases.
When Not to Use A/B Testing
Let's take a closer look.
The product is at the start and has little traffic. In such a situation, you risk not getting results in the required time. For example, when a landing page with startup services is being tested or a new product is being launched, there is not much traffic yet. You will have to wait a long time to get a statistically significant result from the experiment. During this time, the hypothesis being tested may become irrelevant.
Special calculators will help you calculate how justified it is to conduct an A/B test for a product. You can determine the sample size with Evan Miller's calculator , and the test duration with the VWO calculator .
A product for B2B, B2G or premium segment. These areas have a high value of each client. During an A/B test, part of the audience may see a "raw" version of the solution and refuse to cooperate. This can happen if you show users an inconvenient interface or unclear tariffs.
In the B2B and B2G product market, there is a possibility that customers communicate with each other. Part of the audience may learn that the “chosen ones” are offered special conditions that are not available to them.
Let's look at an example. A real estate ad placement service is testing a new payment model. The service's corporate clients are developers and real estate agencies, and some of them saw the new rates during the A/B test. Many market representatives communicate with each other. Soon, puzzled users call the corporate client department to find out about the new rates and why they don't see them on the site. Such a situation can reduce customer confidence in the product, and also distort the results of the test itself.real estate ads is testing a new payment model. The service's corporate clients are developers and real estate agencies,and some of them saw the new rates during the A/B test. Many market representatives communicate with each other. Soon, puzzled users call the corporate customer service department to find out about the new rates and why they don’t see them on the site. This situation can reduce customer confidence in the product, and also distort the results of the test itself.corporate client work to find out about new rates and why they don't see them on the website. This situation can reduce the trust of the clientients to the product, and also distort the results of the test itself.
The vacancies say that A/B testing skills are needed. How can I get them if I don’t work with the product yet?ut that A/B testing skills are needed. How can I get them if I don't work with the product yet?o need A/B testing skillsovanija. How can I get it if I don’t work with the product yet?I. How to get it if INot working with the product yet?
Try to master it yourself.
If the skill is needed for a junior position, you probably won't be assessed by the number of A/B tests you've run. It's important for the employer that you understand how the tool works and how to set it up. That way, they won't have to teach you from scratch.
How to Get A/B Testing Skills
Practice on Pet Project. You can start your own project — a personal blog or a habit tracker — and A/B test some feature. This is a good way to practice setting up testing.
If you have no experience in launching your own project, we recommend reading our article on this topic. In it, we analyzed how to make your own Pet Project using a real example and gave a detailed guide.
Limitations of the method: It is unlikely that you will be able to analyze the results. At the start, you will have little traffic, which means a small sample for the experiment.
Work as a freelancer. You can add real cases to your portfolio if you find small orders on the exchange.
Limitations of the method: Customers care not only about the result itself, but also about trust in this result. Therefore, you need to try to find an order that will be given to a beginner without experience. To do this, at first you will have to set the price for your services below the market price. It can be increased when you gain enough experience, get reviews and a positive rating on exchanges.
Reach out to experts to help. You can find an A/B testing expert in your circle or outside of it and ask them to delegate some of the tasks to you.
Limitations of the method: Not everyone will respond to your offer, and those who do will most likely offer a symbolic payment for your services or not pay at all. Do not be upset and remember that your goal is to gain experience, not to earn money.
Take courses. You don’t have to look for a course just on A/B testing. You can take a more comprehensive topic and understand several related issues at once.
Limitations of the method: You can master the theory on your own, so choose courses where training is based on cases.
If you're looking to change careers, read our article on how to get an interview with no experience.Read our article aboutRead our article about It'll give you even more ways to gain skills in a new field.re.
As a rule, A/B testing skills are rarely found in entry-level positions. If you are attracted to this type of job, don’t get lost. Try one of the options we have suggested.ko is found in vacancies for entry-level specialists. If you pattracted by this one - don't get lost. Try one of the options we offer.
Let's say I want to do A/B testing. What should I do?
We've broken down the A/B testing process from planning to analyzing results into eight steps.planning to analyzing the results in eight steps.
How to Conduct A/B Testing
Let's take a closer look at each step.
Step 1. Formulate hypotheses
The hypothesis will help you define the goals of the experiment, select metrics, and interpret the results. ment, select metrics and interpret the results.
In another article, we described in detail how to formulate a data-driven hypothesis in 5 steps . In this one, we offer a template that is suitable for A/B testing:
A/B Testing Hypothesis Template
Let's look at an example of a template. Imagine that you are working on an interface for an online store. There is an idea - to place a newsletter subscription widget in a pop-up that appears as soon as the user enters the site. The text of the widget is planned to include information about a 10% discount for a subscription. Currently, this information and the subscription widget are placed at the bottom of the page.
The hypothesis can be formulated using the framework:
IF the subscription pop-up with information about the discount will appear immediately as soon as the user enters the site,
THEN users will subscribe more often and place an order immediately,
BECAUSE the message about the discount will encourage them not only to sign up for the newsletter, but also to make a purchase.
The last part of the hypothesis may contain guesses - this is normal. The experiment is conducted to confirm or disprove them.
There may be so many hypotheses that it will not be possible to test them all with one test. Therefore, we recommend reading our article on how to prioritize product research hypotheses .
Step 2: Define Metrics
An A/B test metric is an indicator used to evaluate whether a hypothesis has been confirmed or not. In the example above, where we formulated the hypothesis, the metrics for testing it will be conversion rates to target actions — subscription and order placement.
Let's practice defining experimental metrics.
Exercise
You are working on a feature — a button to go to the product catalog. You need to make it more noticeable to users to attract their attention to the catalog. There are two possible solutions — a text button format and an animated one. What metrics will you choose to find out which option is better?
Please select one of the options
Percentage of button clicks
Bounce rate
Button click rate and bounce rate ov
Number of items in the cart
Read the best articles on product launches and growth product launch and growth
Your email →
Subscribe
Once a week we will send a fresh digest to your email. We are read by 25,000 people
and a digest to your email. 25,000 people read us Us. 25000 people read us
It is important to consider all the metrics that an experiment may affect. Otherwise, there is a risk of choosing an option that improves one metric but worsens the product as a whole.
Step 3: Confirm success criteria and actions based on test results
A success criterion is an expected result that can be used to decide whether a test is successful.
For example, the conversion to a subscription is 10%. You add a pop-up with information about a promo code for a discount for a subscription and expect the conversion to grow to 11%. 11% is the success criterion.
If the conversion to subscription for the pop-up version really increases, then the hypothesis has been confirmed - the experiment was a success, and the change can be introduced.
If the metrics have worsened, then other solutions need to be worked out.
If the metrics remain at the same level or the changes are statistically insignificant, there may be different options for further action, such as:
change test conditions
select another feature
test on a different audience segment
The action plan for different final results is determined at the stage of experiment preparation. Otherwise, questions like: "What to do next? Why did we conduct an A/B test in the first place?" may arise later.
Step 4. Submit to colleagues for cross-review
Cross-review is worth conducting when preparing large and important experiments. Colleagues will help evaluate all controversial points. They will check how the hypothesis is formulated, whether all metrics that can be affected by the experiment are taken into account, and how correct are the decisions that are planned to be made based on the results.
The template for preparing an experiment , which is published in our Telegram channel, will help you organize the cross-review process . The template systematizes the information needed to set up an A/B test and analyze the results.
A/B test preparation template
Step 5. Prepare the experiment
At this stage, a version with the changes being tested is created and the following parameters are defined:
A/B test parameters
Control and experimental sample — users who will participate in the experiment. You can check both on all users and on groups that are united by a certain indicator: location, gender, age, and so on.
Minimum sample size is the minimum number of users who must see the versions being tested for the results to be statistically significant. This can be calculated using calculators such as Evan Miller's calculator .
The statistical significance indicator is the difference between the control and experimental sample metrics at which it is unlikely that this result is due to chance. Calculators for calculating this indicator are usually integrated into the test setup system.— the difference between the control and experimental sample metrics that makes it unlikely that this is a random result. Calculators for calculating this indicator are usually integrated into the test setup system.ki, which is unlikely to be a random result. Calculators for calculating this indicator are usually integrated into the test setup system.V.
Test duration — the number of days the experiment will run. The indicator depends on the total sample size and current traffic. You can calculate the test duration using the formula: sample / traffic = number of days. in days when the experiment will be carried out. The indicator depends on the total size of yousampling and current traffic. The duration of testing can be calculated using the formula: sampling / traffic = number of days.
Let's say the minimum sample size for statistically significant test results is 10,000 unique visitors. The site has an average of 1,000 visitors per day. So the test duration is 10 days.for statistically significant test results - 10,000 unique visitors. The site has an average of 1,000 visitors per day. So the test duration is 10 days.test - 10 days.
Step 6. Run the experiment
The main thing to do when launching is to check that the A/B test is running correctly. For example, the button being tested works, and users are randomly selected. If there is a big difference in the conversion of versions A and B right away, or vice versa, there is no difference at all - this is a signal that something is wrong.— check that the A/B test is running correctly. For example, a button that testsis being tested, it works, and users are randomly included in the samples. If a big difference in the conversion of versions A and B immediately appears, or vice versa, there is no difference at all - this is a signal that something is going wrong.in a special way. If a big difference in the conversion of versions A and B or vice versa immediately appeared, pthere is no difference at all - this is a signal that something is going wrong.
If the test is working correctly, do not try to analyze the results before it is finished or make changes to the test settings during the process. One option may be the winner on the first day, and another on the next day. You need to wait until the test is finished to get reliable results.
Step 7. Analyze the results and draw conclusions
Once you have the metrics data, compare them with the success criteria you defined during the test preparation stage. Now you can draw conclusions about the results of the experiment and refer to the action plan that was drawn up before the testing began.
Step 8. Create a report and share it with colleagues
Share the results of the A/B test with your colleagues — it will help them understand user behavior, even if, at first glance, the hypothesis is not relevant to their tasks. You can share the results in the form of a presentation for the team.
It makes sense to store reports in a way that all employees of your company can easily find them. For example, in Confluence or another system where you can create a single company knowledge base. This will help save time and resources of the team and avoid repeated A/B tests.
What can replace A/B testing?
Sometimes it is easier to test a hypothesis using other methods. Let's look at what these methods are and in what situations they can replace an A/B test.
Methods that can replace A/B testing
Usability testing. This method checks how convenient the interface is for users.
For research, you don’t need to involve developers, as in the case of A/B testing. You need to create a new interface at the level of layouts, assemble an interactive prototype and observe how users interact with it. Then identify possible problems and find a solution. We talked about how prototypes are tested and the results are analyzed in the article about UX.
Fake door test. When developing a feature is difficult and time-consuming, this method can be used to check whether users need it. difficult and time-consuming, this method can be used to check whether users need it.
To do this, a button with nothing behind it is added to the interface — a fake door — and the percentage of users who click it is tracked. A message is usually placed behind the fake door stating that the section is under development. You can also add a link to a survey and thus collect additional data for the future product.a, behind which there is nothing, - a fake door - and the percentage of users is trackedtel will click it. A message is usually placed behind the fake door stating that the section is under development. You can also add a link to a survey and thus collect additional data for the future product.additional data for the future product.
Release of a new product to a limited audience. audience.If there is enough time, then instead of testing, you can launch the product to one city, region or other selected part of users.
The method is suitable when the product is local and it is necessary to test major changes in the business model, or try a completely new product. For example, the unmanned taxi that Yandex is testing in one of the districts of Moscow. If the results are positive, the product can be scaled to the rest of the audience.and it is necessary to test major changes to the business model, or try a completely new product. For example, a driverless taxi, which Yandex is testing in one of the districts of Moscow. If the results are positive, the product can be scaled to the rest of the audience.es-models, or try a completely new product. For example, a driverless taxi, whichYandex is operating in one of Moscow's districts. If the results are positive, the product can be scaled up to the rest of the audience.the results will be positive, the product can be scaled up throughout the worldsteel audience.
Let's practice choosing a hypothesis testing method for a specific situation.
Exercise
You want to increase the number of orders through a service that delivers ready-made food from restaurants. One way to do this is to add the ability to order products from stores in the delivery area. How will you check whether the new service will be popular with users?
Please select one of the options
Conduct an A/B test
Add the ability to use the service only in one area of the city
Add a new service section to the fake door service application
Organize usability testing
We have collected a selection of useful resources and publications on the topic:
The Complete A/B Testing Kit is a free guide to A/B testing, you can download it after registration.
How to Do A/B Testing: 15 Steps for the Perfect Split Test is a comprehensive checklist of what to do before, during, and after an A/B test.
A/B testing of a website or landing page — a step-by-step guide to launching an israel email list experiment through Google Optimize from TILDA EDUCATION.
How to Increase Sales with A/B Testing (Includes Examples) — an article that shows, using cases, how to increase sales using A/B testing.
How to make a billion - a speech by Head of Product Marketing at Skyeng Denis Pushkin, in which he talks about how to choose product hypotheses, test them and correctly calculate the results of experiments.
The best experiment in Booking.com history is a post on the channel of the article's co-author, and this experiment is not an A/B test at all.
Let's sum it up
A/B testing allows teams to quickly test multiple hypotheses and continuously evolve an online product. In real life, more resources are needed to test which solution is best for the business.
If you're working on a digital product with a large audience, try A/B testing. This way, you can learn the value of product changes based on the actions of real users.
You shouldn't rely on subjective taste; you need to test options. A/B tests will help with this, with which you can check the effectiveness of solutions when developing a product.
Difficulty: easy
Reading time: 7 minutes
What will be in the article:
What are A/B tests and how to conduct them?
Who needs to understand A/B testing?
When should you not use A/B testing?
A collection of tools for setting up A/B tests and resources for self-study
What is A/B testing?
A/B testing is a tool that helps teams test hypotheses and make decisions based on data rather than gut instinct. The idea is to split an audience into parts and show them different versions of something to see which version is better.
You can test the landing page design, the color and location of the call to action button, the registration form, the design of the email newsletter, the text of the ad on the website, and other changes.
A/B testing of newsletter subject lines
A/B test results show which solution will provide a higher conversion rate to the desired target action. For example, in which case more users will click on a link, register on a website or in an application, subscribe to a newsletter, fill out a feedback form. There are also more complex tests that are aimed at studying long-term metrics, such as the average check or the impact of changes in the product on profit.
To be sure of the reliability of the results, it is better to conduct the test with a control sample. To do this, you need to divide the audience on which the variants will be tested not into two groups (A and B), but into three (A, A and B). Show one version to two groups, and another to the third. This will reveal whether external factors influenced the test result and whether there were any errors in collecting metrics.
Imagine you are testing a subscription form on a store website.
At this time, another department launched an advertising campaign to promote a certain product in the catalog, and many users came to the site. They want to buy now and are not interested in subscribing at all, so they can distort the results of the experiment.
The control sample will show whether the A/B test participants were distributed evenly across groups and whether the advertising campaign affected the results.but whether the A/B test participants were distributed into groups, and whether the advertising campaign influenced its results.
External factors did not affect the result, and the data can be trusted if the indicators of groups A and A do not differ. If they do, the numbers are distorted, which means that you should not make a decision based on the test.at, and the data can be trusted if the indicators of groups A and A do not differ. If they do, the figures are distorted, which means that you should not make a decision based on the test.whether they differ - the numbers are distorted, which means you shouldn’t make a decision based on the test.
The principle of A/B testing with a control sample
What if you need to test more than two options? For example, test four application formsneed to check moree two options? For example, test four application formsname, test hfour application forms
If you have more than two options, you can conduct multivariate testing.
The principle is the same as in A/B testing, only more than two versions of one change are compared simultaneously. A part of the audience is allocated for each option for display, at the end of the test their results are compared. The version that showed the best metrics wins.
Multivariate testing
A multivariate test is best used to test several versions with minor changes. For example, you can test four call to action phrases for one button.
This method is also useful for testing the metrics of different combinations. Let's say you have four button text options and two colors. Using a multivariate test, you can compare eight possible versions and see which combination performed best.
We discussed in more detail how and why to test creatives and their variations in the article about visuals in SMM and paid traffic .
As with A/B testing, the multivariate method can be used to analyze not only the conversion of the first target action, but also to evaluate the subsequent behavior of users.
Let's say a red button with the text "Buy at a discount" will collect the most clicks. But it may turn out that the conversion to purchase for this combination is two times lower than for a less clickable green button with the text "View catalog". The winner is not always the most obvious combination.
Is it possible to conduct several experiments in parallel?
Yes, you can. But with restrictions.
The samples of two parallel tests should not be mixed. Otherwise, one change may affect the perception of the second.
How to and how not to conduct a parallel test
Let's practice determining in which cases parallel A/B tests can be conducted and in which they cannot.
Exercise
Here are solutions for improving the online lecture website. Select from the list those solutions that can be tested in parallel.
The changes that are proposed to be tested in parallel are within one answer option.
Please select multiple options
1. The "Enable subtitles" button in the video lecture and the call to action text on external traffic sources
2. The location of the menu buttons and visual design on the main page of the site
3. The location of the button for going to your personal account and the visual design of creatives for the advertising campaign
4. Filters of lecture catalog sections and its visual design about visual design
5. Subscription widget on the main page and the color of the "Place an order" button in the cart color of the "Place an order" button in the shopping cart
6. Form for questions to the lecturer and the button "Turn on subtitles" Turn on subtitles»
Parallel tests can be set up on the frontend side by tagging events with different tags for different versions of the interface. Or you can use special tools the frontend side, marking events with different tags for different versions of the interface. Or you can use special toolsthat simplify this process,forgive this process, such as Google Optimize .
Guide to the best skillsetter articlestatiam skillsetter
Our blog already has over 150 articles about product growth and careers in IT. For easy navigation, we have combined them into thematic collections.over 150 articles about product growth and careers in IT. For easy navigation, we have combined them into thematic collections.
Select a collection
Who needs to understand A/B testing?
At a basic level, everyone involved in working on the product needs to understand it. This will help choose the best research methods for different tasks and at different stages of work.
A deeper knowledge of A/B testing is needed by a product manager, marketer, and analyst.
The analyst participates in test setup: forms requirements for launch, defines metrics, checks the correctness of the operation, and analyzes the results after completion.
Product and marketers need to understand what they can test, how to conduct an experiment, and analyze its results.
Marketers use A/B testing to attract more people and better convert them into a purchase. Most often, they conduct experiments when working with email newsletters, push notifications, landing pages, and advertising creatives.
Product managers primarily use A/B testing to improve product metrics. For example, they might compare two versions of onboarding, different versions of product features, and their usability for users.
When working on a product, the best option is teamwork at all stages, including during testing of ideas.
Let's see how this looks in practice. A product manager conducted a survey of users of an online lecture hall. He found out that lectures in English are difficult to perceive by ear because of the special terms. The team gathered for a brainstorming session, during which the idea of adding subtitles to video lectures appeared.
After that, the designer drew two versions of the subtitle button. The analyst prepared the parameters for collecting metrics for the A/B test and the technical task for the developer. The developer wrote the code for the interface. The analyst set up the testing system, launched the test, checked the quality of the data, assessed the statistical significance of changes in the metrics, and made a report. The report was discussed at a team meeting, and based on the test results, they decided together which button to add to the site.
What to do if there is no analyst on the team?there is no analyst in the team?
You can prepare and conduct an A/B test without an analyst.without analyst.
Built-in A/B testing features are available in advertising campaign setup services, email and push notification tools, and some website builders, such as I have services for setting up advertising campaigns, tools for email newsletters and push messages, as well as some website builders,For example,Wix .
To set up an A/B test, you can also use special tools:use special tools:
Google Optimize . The tool has an intuitive interface. You can calculate the main metrics, and after the A/B test, upload the data to Google Analytics to analyze the results.o clear interface. You can calculate the main metrics, and based on the results of the A/B test, upload the data to Google Analytics to analyze the results.
VWO : This tool supports multivariate testing and URL split testing. You can create visual versions without changing the code via the built-in editor.Variant testing and split testing of URLs. You can create visual versions without changing the code via the built-in editor.
Unbounce . The tool is suitable for basic A/B testing. There is a visual designer and a visual editor.A/B testing. There is a visual designer and a visual editor.
Optimizely . The tool supports multi-page tests. There is also a visual editor to create versions without changing the code, optimization for mobile applications is available.rcii without changing the code, optimization for mobile applications is available.ization for mobile applications.
Optimizely Interface
We recommend reading other articles from this series:
10 Embarrassing Questions About Promotion in Google Play and App Store
10 Embarrassing Questions About Unit Economics
10 Embarrassing Questions About No-Code
10 Embarrassing Questions About SEO
10 Embarrassing Questions About Paid Traffic
Free course on product management
In 3 days you will learn to think like a product and cope more successfully with tasks at work and in life
Start learning for free
When should you not use A/B testing?
A/B testing should not be used in several cases.
When Not to Use A/B Testing
Let's take a closer look.
The product is at the start and has little traffic. In such a situation, you risk not getting results in the required time. For example, when a landing page with startup services is being tested or a new product is being launched, there is not much traffic yet. You will have to wait a long time to get a statistically significant result from the experiment. During this time, the hypothesis being tested may become irrelevant.
Special calculators will help you calculate how justified it is to conduct an A/B test for a product. You can determine the sample size with Evan Miller's calculator , and the test duration with the VWO calculator .
A product for B2B, B2G or premium segment. These areas have a high value of each client. During an A/B test, part of the audience may see a "raw" version of the solution and refuse to cooperate. This can happen if you show users an inconvenient interface or unclear tariffs.
In the B2B and B2G product market, there is a possibility that customers communicate with each other. Part of the audience may learn that the “chosen ones” are offered special conditions that are not available to them.
Let's look at an example. A real estate ad placement service is testing a new payment model. The service's corporate clients are developers and real estate agencies, and some of them saw the new rates during the A/B test. Many market representatives communicate with each other. Soon, puzzled users call the corporate client department to find out about the new rates and why they don't see them on the site. Such a situation can reduce customer confidence in the product, and also distort the results of the test itself.real estate ads is testing a new payment model. The service's corporate clients are developers and real estate agencies,and some of them saw the new rates during the A/B test. Many market representatives communicate with each other. Soon, puzzled users call the corporate customer service department to find out about the new rates and why they don’t see them on the site. This situation can reduce customer confidence in the product, and also distort the results of the test itself.corporate client work to find out about new rates and why they don't see them on the website. This situation can reduce the trust of the clientients to the product, and also distort the results of the test itself.
The vacancies say that A/B testing skills are needed. How can I get them if I don’t work with the product yet?ut that A/B testing skills are needed. How can I get them if I don't work with the product yet?o need A/B testing skillsovanija. How can I get it if I don’t work with the product yet?I. How to get it if INot working with the product yet?
Try to master it yourself.
If the skill is needed for a junior position, you probably won't be assessed by the number of A/B tests you've run. It's important for the employer that you understand how the tool works and how to set it up. That way, they won't have to teach you from scratch.
How to Get A/B Testing Skills
Practice on Pet Project. You can start your own project — a personal blog or a habit tracker — and A/B test some feature. This is a good way to practice setting up testing.
If you have no experience in launching your own project, we recommend reading our article on this topic. In it, we analyzed how to make your own Pet Project using a real example and gave a detailed guide.
Limitations of the method: It is unlikely that you will be able to analyze the results. At the start, you will have little traffic, which means a small sample for the experiment.
Work as a freelancer. You can add real cases to your portfolio if you find small orders on the exchange.
Limitations of the method: Customers care not only about the result itself, but also about trust in this result. Therefore, you need to try to find an order that will be given to a beginner without experience. To do this, at first you will have to set the price for your services below the market price. It can be increased when you gain enough experience, get reviews and a positive rating on exchanges.
Reach out to experts to help. You can find an A/B testing expert in your circle or outside of it and ask them to delegate some of the tasks to you.
Limitations of the method: Not everyone will respond to your offer, and those who do will most likely offer a symbolic payment for your services or not pay at all. Do not be upset and remember that your goal is to gain experience, not to earn money.
Take courses. You don’t have to look for a course just on A/B testing. You can take a more comprehensive topic and understand several related issues at once.
Limitations of the method: You can master the theory on your own, so choose courses where training is based on cases.
If you're looking to change careers, read our article on how to get an interview with no experience.Read our article aboutRead our article about It'll give you even more ways to gain skills in a new field.re.
As a rule, A/B testing skills are rarely found in entry-level positions. If you are attracted to this type of job, don’t get lost. Try one of the options we have suggested.ko is found in vacancies for entry-level specialists. If you pattracted by this one - don't get lost. Try one of the options we offer.
Let's say I want to do A/B testing. What should I do?
We've broken down the A/B testing process from planning to analyzing results into eight steps.planning to analyzing the results in eight steps.
How to Conduct A/B Testing
Let's take a closer look at each step.
Step 1. Formulate hypotheses
The hypothesis will help you define the goals of the experiment, select metrics, and interpret the results. ment, select metrics and interpret the results.
In another article, we described in detail how to formulate a data-driven hypothesis in 5 steps . In this one, we offer a template that is suitable for A/B testing:
A/B Testing Hypothesis Template
Let's look at an example of a template. Imagine that you are working on an interface for an online store. There is an idea - to place a newsletter subscription widget in a pop-up that appears as soon as the user enters the site. The text of the widget is planned to include information about a 10% discount for a subscription. Currently, this information and the subscription widget are placed at the bottom of the page.
The hypothesis can be formulated using the framework:
IF the subscription pop-up with information about the discount will appear immediately as soon as the user enters the site,
THEN users will subscribe more often and place an order immediately,
BECAUSE the message about the discount will encourage them not only to sign up for the newsletter, but also to make a purchase.
The last part of the hypothesis may contain guesses - this is normal. The experiment is conducted to confirm or disprove them.
There may be so many hypotheses that it will not be possible to test them all with one test. Therefore, we recommend reading our article on how to prioritize product research hypotheses .
Step 2: Define Metrics
An A/B test metric is an indicator used to evaluate whether a hypothesis has been confirmed or not. In the example above, where we formulated the hypothesis, the metrics for testing it will be conversion rates to target actions — subscription and order placement.
Let's practice defining experimental metrics.
Exercise
You are working on a feature — a button to go to the product catalog. You need to make it more noticeable to users to attract their attention to the catalog. There are two possible solutions — a text button format and an animated one. What metrics will you choose to find out which option is better?
Please select one of the options
Percentage of button clicks
Bounce rate
Button click rate and bounce rate ov
Number of items in the cart
Read the best articles on product launches and growth product launch and growth
Your email →
Subscribe
Once a week we will send a fresh digest to your email. We are read by 25,000 people
and a digest to your email. 25,000 people read us Us. 25000 people read us It is important to consider all the metrics that an experiment may affect. Otherwise, there is a risk of choosing an option that improves one metric but worsens the product as a whole.
Step 3: Confirm success criteria and actions based on test results
A success criterion is an expected result that can be used to decide whether a test is successful.
For example, the conversion to a subscription is 10%. You add a pop-up with information about a promo code for a discount for a subscription and expect the conversion to grow to 11%. 11% is the success criterion.
If the conversion to subscription for the pop-up version really increases, then the hypothesis has been confirmed - the experiment was a success, and the change can be introduced.
If the metrics have worsened, then other solutions need to be worked out.
If the metrics remain at the same level or the changes are statistically insignificant, there may be different options for further action, such as:
change test conditions
select another feature
test on a different audience segment
The action plan for different final results is determined at the stage of experiment preparation. Otherwise, questions like: "What to do next? Why did we conduct an A/B test in the first place?" may arise later.
Step 4. Submit to colleagues for cross-review
Cross-review is worth conducting when preparing large and important experiments. Colleagues will help evaluate all controversial points. They will check how the hypothesis is formulated, whether all metrics that can be affected by the experiment are taken into account, and how correct are the decisions that are planned to be made based on the results.
The template for preparing an experiment , which is published in our Telegram channel, will help you organize the cross-review process . The template systematizes the information needed to set up an A/B test and analyze the results.
A/B test preparation template
Step 5. Prepare the experiment
At this stage, a version with the changes being tested is created and the following parameters are defined:
A/B test parameters
Control and experimental sample — users who will participate in the experiment. You can check both on all users and on groups that are united by a certain indicator: location, gender, age, and so on.
Minimum sample size is the minimum number of users who must see the versions being tested for the results to be statistically significant. This can be calculated using calculators such as Evan Miller's calculator .
The statistical significance indicator is the difference between the control and experimental sample metrics at which it is unlikely that this result is due to chance. Calculators for calculating this indicator are usually integrated into the test setup system.— the difference between the control and experimental sample metrics that makes it unlikely that this is a random result. Calculators for calculating this indicator are usually integrated into the test setup system.ki, which is unlikely to be a random result. Calculators for calculating this indicator are usually integrated into the test setup system.V.
Test duration — the number of days the experiment will run. The indicator depends on the total sample size and current traffic. You can calculate the test duration using the formula: sample / traffic = number of days. in days when the experiment will be carried out. The indicator depends on the total size of yousampling and current traffic. The duration of testing can be calculated using the formula: sampling / traffic = number of days.
Let's say the minimum sample size for statistically significant test results is 10,000 unique visitors. The site has an average of 1,000 visitors per day. So the test duration is 10 days.for statistically significant test results - 10,000 unique visitors. The site has an average of 1,000 visitors per day. So the test duration is 10 days.test - 10 days.
Step 6. Run the experiment
The main thing to do when launching is to check that the A/B test is running correctly. For example, the button being tested works, and users are randomly selected. If there is a big difference in the conversion of versions A and B right away, or vice versa, there is no difference at all - this is a signal that something is wrong.— check that the A/B test is running correctly. For example, a button that testsis being tested, it works, and users are randomly included in the samples. If a big difference in the conversion of versions A and B immediately appears, or vice versa, there is no difference at all - this is a signal that something is going wrong.in a special way. If a big difference in the conversion of versions A and B or vice versa immediately appeared, pthere is no difference at all - this is a signal that something is going wrong.
If the test is working correctly, do not try to analyze the results before it is finished or make changes to the test settings during the process. One option may be the winner on the first day, and another on the next day. You need to wait until the test is finished to get reliable results.
Step 7. Analyze the results and draw conclusions
Once you have the metrics data, compare them with the success criteria you defined during the test preparation stage. Now you can draw conclusions about the results of the experiment and refer to the action plan that was drawn up before the testing began.
Step 8. Create a report and share it with colleagues
Share the results of the A/B test with your colleagues — it will help them understand user behavior, even if, at first glance, the hypothesis is not relevant to their tasks. You can share the results in the form of a presentation for the team.
It makes sense to store reports in a way that all employees of your company can easily find them. For example, in Confluence or another system where you can create a single company knowledge base. This will help save time and resources of the team and avoid repeated A/B tests.
What can replace A/B testing?
Sometimes it is easier to test a hypothesis using other methods. Let's look at what these methods are and in what situations they can replace an A/B test.
Methods that can replace A/B testing
Usability testing. This method checks how convenient the interface is for users.
For research, you don’t need to involve developers, as in the case of A/B testing. You need to create a new interface at the level of layouts, assemble an interactive prototype and observe how users interact with it. Then identify possible problems and find a solution. We talked about how prototypes are tested and the results are analyzed in the article about UX.
Fake door test. When developing a feature is difficult and time-consuming, this method can be used to check whether users need it. difficult and time-consuming, this method can be used to check whether users need it.
To do this, a button with nothing behind it is added to the interface — a fake door — and the percentage of users who click it is tracked. A message is usually placed behind the fake door stating that the section is under development. You can also add a link to a survey and thus collect additional data for the future product.a, behind which there is nothing, - a fake door - and the percentage of users is trackedtel will click it. A message is usually placed behind the fake door stating that the section is under development. You can also add a link to a survey and thus collect additional data for the future product.additional data for the future product.
Release of a new product to a limited audience. audience.If there is enough time, then instead of testing, you can launch the product to one city, region or other selected part of users.
The method is suitable when the product is local and it is necessary to test major changes in the business model, or try a completely new product. For example, the unmanned taxi that Yandex is testing in one of the districts of Moscow. If the results are positive, the product can be scaled to the rest of the audience.and it is necessary to test major changes to the business model, or try a completely new product. For example, a driverless taxi, which Yandex is testing in one of the districts of Moscow. If the results are positive, the product can be scaled to the rest of the audience.es-models, or try a completely new product. For example, a driverless taxi, whichYandex is operating in one of Moscow's districts. If the results are positive, the product can be scaled up to the rest of the audience.the results will be positive, the product can be scaled up throughout the worldsteel audience.
Let's practice choosing a hypothesis testing method for a specific situation.
Exercise
You want to increase the number of orders through a service that delivers ready-made food from restaurants. One way to do this is to add the ability to order products from stores in the delivery area. How will you check whether the new service will be popular with users?
Please select one of the options
Conduct an A/B test
Add the ability to use the service only in one area of the city
Add a new service section to the fake door service application
Organize usability testing
We have collected a selection of useful resources and publications on the topic:
The Complete A/B Testing Kit is a free guide to A/B testing, you can download it after registration.
How to Do A/B Testing: 15 Steps for the Perfect Split Test is a comprehensive checklist of what to do before, during, and after an A/B test.
A/B testing of a website or landing page — a step-by-step guide to launching an israel email list experiment through Google Optimize from TILDA EDUCATION.
How to Increase Sales with A/B Testing (Includes Examples) — an article that shows, using cases, how to increase sales using A/B testing.
How to make a billion - a speech by Head of Product Marketing at Skyeng Denis Pushkin, in which he talks about how to choose product hypotheses, test them and correctly calculate the results of experiments.
The best experiment in Booking.com history is a post on the channel of the article's co-author, and this experiment is not an A/B test at all.
Let's sum it up
A/B testing allows teams to quickly test multiple hypotheses and continuously evolve an online product. In real life, more resources are needed to test which solution is best for the business.
If you're working on a digital product with a large audience, try A/B testing. This way, you can learn the value of product changes based on the actions of real users.