About this video
A quick walkthrough of A/B testing a hero button with Sanity Studio and PostHog. Set up control and variant pages in Sanity, ship them to users, and read exposures and clicks per user in PostHog.
Full transcript
Edited transcript from the original Roboto page
AB testing with Sanity Studio and PostHog. I'm going to show you a very cool example of a button hero that we built internally and how easily you can set your own test and check the results on PostHog.
As you can see on my screen, I've set up my test on Sanity. This is the AB testing space where I've added my control and my other one, called it blue. Essentially it's a black button. And the other thing is I have pages for black and white as well. You can set as many pages in your particular case as you want.
Now I'll show you exactly how this looks. The user will see this and what they can do is click on this button, and you can see the total clicks changing on the screen. If I refresh it, hopefully I get a different variation to show you.
There we go. So we've got the black one. Click a bunch of times. Anyway, you can do this for any use case with different variations. You can do it for form clicks, different pages, or whatever suits your needs. I'll show you how this looks on the PostHog side.
This is my particular demo. As you can see, total exposure is 125. That's 125 times this was exposed to people. I have my control and I have my blue. I called the other one blue but it's essentially a black button. I know it's a little bit confusing but bear with me.
As you can see here, on average there's eight clicks per person for the control, and on average there's nine clicks per user for blue. But the interesting thing is 30 versus 23 exposures to look at. So 23 times the blue button was exposed to the user as compared to 30 times for the control variation. But on average, nine clicks per user was recorded for the blue one. So the blue one, or the black button, is winning in our case.
You can set your metrics here, look at them, look at all the different charts. In our case, as I mentioned, the blue one is doing much better than the white one. But this is a very simple example. The whole point of this video is to show you how easy it is to record this and use it in your own use case.
The tool is completely open source. Let us know if you'd like to try it for yourself and how you found it. We love making these kinds of videos.
Read timestamped source captions
0:04Hey, how's it going?
0:04Hey, how's it going? In today's video, I'm going to show you
0:06In today's video, I'm going to show you
0:06In today's video, I'm going to show you AB testing with Sanity Studio and
0:09AB testing with Sanity Studio and
0:09AB testing with Sanity Studio and PostHog.
0:10PostHog.
0:10PostHog. So, in this video, I'm going to show you
0:12So, in this video, I'm going to show you
0:12So, in this video, I'm going to show you a very cool example of a button hero
0:16a very cool example of a button hero
0:16a very cool example of a button hero that we built internally.
0:17that we built internally.
0:17that we built internally. And I will show you how easily you can
0:20And I will show you how easily you can
0:20And I will show you how easily you can set your own test and check the results
0:23set your own test and check the results
0:23set your own test and check the results on PostHog.
0:25on PostHog.
0:25on PostHog. So, as you can see on my screen, I've
0:27So, as you can see on my screen, I've
0:27So, as you can see on my screen, I've set up my test on Sanity, and this is
0:29set up my test on Sanity, and this is
0:29set up my test on Sanity, and this is the AB testing space where I've added my
0:32the AB testing space where I've added my
0:32the AB testing space where I've added my control and my other one. I called it
0:34control and my other one. I called it
0:34control and my other one. I called it blue. Essentially, it's a black button.
0:37blue. Essentially, it's a black button.
0:37blue. Essentially, it's a black button. And the other thing is I have pages for
0:40And the other thing is I have pages for
0:40And the other thing is I have pages for black and white as well. And you can set
0:42black and white as well. And you can set
0:42black and white as well. And you can set as many pages in in your particular case
0:45as many pages in in your particular case
0:45as many pages in in your particular case as you want.
0:46as you want.
0:46as you want. Now, I'll show you exactly how this
0:48Now, I'll show you exactly how this
0:48Now, I'll show you exactly how this looks. So, the user will see this, and
0:51looks. So, the user will see this, and
0:51looks. So, the user will see this, and what they can do is they can click on
0:52what they can do is they can click on
0:52what they can do is they can click on this button. And you can see the total
0:56this button. And you can see the total
0:56this button. And you can see the total clicks changing on the screen.
0:59clicks changing on the screen.
0:59clicks changing on the screen. And if I refresh it, hopefully, I get a
1:02And if I refresh it, hopefully, I get a
1:02And if I refresh it, hopefully, I get a different variation to show you guys.
1:05different variation to show you guys.
1:05different variation to show you guys. There we go. So, we've got the black
1:07There we go. So, we've got the black
1:07There we go. So, we've got the black one.
1:08one.
1:08one. Click a bunch of time.
1:10Click a bunch of time.
1:10Click a bunch of time. Anyway, so you can do this for any use
1:13Anyway, so you can do this for any use
1:13Anyway, so you can do this for any use case for different variations. You can
1:15case for different variations. You can
1:15case for different variations. You can do it for form clicks. You can do it for
1:17do it for form clicks. You can do it for
1:17do it for form clicks. You can do it for different pages. Whatever suits your
1:19different pages. Whatever suits your
1:19different pages. Whatever suits your needs.
1:21needs.
1:21needs. And I'll show you how this looks on the
1:23And I'll show you how this looks on the
1:23And I'll show you how this looks on the PostHog side. So,
1:26PostHog side. So,
1:26PostHog side. So, this is my particular demo.
1:28this is my particular demo.
1:28this is my particular demo. As you can see, total exposure is 125.
1:32As you can see, total exposure is 125.
1:32As you can see, total exposure is 125. This is pretty simple stuff. Like, you
1:34This is pretty simple stuff. Like, you
1:34This is pretty simple stuff. Like, you know, 125 times this was exposed to
1:37know, 125 times this was exposed to
1:37know, 125 times this was exposed to people.
1:38people.
1:38people. And I have my control, and I have my
1:40And I have my control, and I have my
1:40And I have my control, and I have my blue. I called the other one blue, but
1:43blue. I called the other one blue, but
1:43blue. I called the other one blue, but it's essentially is a black button. I
1:44it's essentially is a black button. I
1:44it's essentially is a black button. I know it's a little bit confusing, but
1:46know it's a little bit confusing, but
1:46know it's a little bit confusing, but bear with me.
1:47bear with me.
1:47bear with me. So, as you can see here, on average,
1:50So, as you can see here, on average,
1:50So, as you can see here, on average, there is eight click per person for the
1:53there is eight click per person for the
1:53there is eight click per person for the control.
1:54control.
1:54control. And on average, there's like nine click
1:56And on average, there's like nine click
1:56And on average, there's like nine click per user for blue. But the interesting
1:59per user for blue. But the interesting
1:59per user for blue. But the interesting thing is 30
2:01thing is 30
2:01thing is 30 30 instances or 30 exposure versus 23
2:0530 instances or 30 exposure versus 23
2:0530 instances or 30 exposure versus 23 exposure is to look at. So, 23 times
2:08exposure is to look at. So, 23 times
2:08exposure is to look at. So, 23 times blue button was exposed to the user as
2:10blue button was exposed to the user as
2:10blue button was exposed to the user as compared to 30 times for the control
2:12compared to 30 times for the control
2:12compared to 30 times for the control variation. But on average, nine per nine
2:17variation. But on average, nine per nine
2:17variation. But on average, nine per nine click per user was recorded for the blue
2:20click per user was recorded for the blue
2:20click per user was recorded for the blue one. So, the blue one or the black
2:22one. So, the blue one or the black
2:22one. So, the blue one or the black button is winning in our case.
2:24button is winning in our case.
2:24button is winning in our case. And you can set
2:26And you can set
2:26And you can set your matrices here. You can look at
2:29your matrices here. You can look at
2:29your matrices here. You can look at them. Look at all the different graphs.
2:31them. Look at all the different graphs.
2:32them. Look at all the different graphs. In our case, as I mentioned, the blue
2:34In our case, as I mentioned, the blue
2:34In our case, as I mentioned, the blue one is doing much better than the white
2:35one is doing much better than the white
2:35one is doing much better than the white one.
2:36one.
2:37one. But this is a very simple example. But
2:39But this is a very simple example. But
2:39But this is a very simple example. But the whole point of this video is to show
2:41the whole point of this video is to show
2:41the whole point of this video is to show you how easy it is to record this and
2:43you how easy it is to record this and
2:43you how easy it is to record this and use it in your own use case. And yeah,
2:46use it in your own use case. And yeah,
2:46use it in your own use case. And yeah, the tuner again completely open source.
2:48the tuner again completely open source.
2:48the tuner again completely open source. So, let us know if you know, if you'd
2:50So, let us know if you know, if you'd
2:51So, let us know if you know, if you'd like to try it for yourself and how you
2:52like to try it for yourself and how you
2:52like to try it for yourself and how you found it. And yeah, we love making these
2:55found it. And yeah, we love making these
2:55found it. And yeah, we love making these kind of videos. So, give us a thumbs up
2:57kind of videos. So, give us a thumbs up
2:57kind of videos. So, give us a thumbs up for this one. And yeah, see you next
2:59for this one. And yeah, see you next
2:59for this one. And yeah, see you next time. Take care. Bye.
Questions answered
Editorial answers from the original Roboto page.
What is AB testing in Sanity Studio?
AB testing in Sanity Studio lets you create different variations of content, like buttons with different colors, and serve them to users to see which performs better. You set up a control version and one or more test variations, then track how users interact with each one to determine which wins.
Read the original ↗How do you set up an AB test in Sanity?
In Sanity's AB testing space, you add your control variation and name your other variations, like we called ours blue. You can also create multiple pages for each variation if you need them. The setup is straightforward and you can customize it for whatever you want to test.
Read the original ↗What metrics does PostHog show for AB tests?
PostHog displays total exposures, which is how many times a variation was shown to users, and average clicks per user for each variation. You can see these metrics side by side to compare performance between your control and test versions, along with additional charts and data points.
Read the original ↗How can you tell which variation is winning?
Compare the average clicks per user between your variations. In our example, the blue button had nine clicks per user on average while the control had eight, making the blue variation the winner. PostHog makes it easy to see these numbers and determine which version performs better.
Read the original ↗What kinds of things can you AB test?
You can test button variations, form clicks, different pages, or really any use case where you want to measure user interaction. The system is flexible enough to accommodate whatever variations you want to compare across your product or site.
Read the original ↗Is this tool free to use?
Yes, the tool is completely open source, so you can use it without paying for a license. You can set it up for yourself and start running tests right away with no cost barriers.
Read the original ↗How easy is it to track test results?
Once you set up your variations in Sanity, they automatically feed into PostHog and you can see all your metrics and charts immediately. The whole process from setup to viewing results is designed to be straightforward so you can focus on what the data tells you about user behavior.
Read the original ↗