How Often Should You Run Experiments? | Ep. #650

In episode #650, Eric and Neil discuss how often you should be running experiments. Tune in to hear when, why, and how you should be running experiments.


  • [00:27] Today’s Topic: How Often Should You Run Experiments?
  • [00:39] Eric thinks you should be running experiments every single week.
  • [00:55] For Eric’s site, they run experiments on the site itself and the business.
  • [01:05] They use North Star to run these experiments and Eric loves the Leaderboard function.
  • [01:45] The more experiments you run, the better off you are.
  • [01:52] However, it’s hard to run experiments when you don’t have enough traffic.
  • [02:04] Once you get enough conversion points (500 per month), you can run 1-2 experiments per month because you will be able to see micro and macro conversions.
  • [02:28] A micro conversion could be someone going to your checkout page.
  • [02:36] A macro conversion could be more sales in general.
  • [03:00] If you’re a small business, it’s ok to track micro conversions only.
  • [03:22] In the short run, if you can’t optimize for conversions and sales, your experimentation should be around traffic generation. Use Similar Web, SEMRush, and AHREFS.
  • [04:15] Managing Oneself is a short booklet that talks about the importance of feedback analysis.
  • [05:14] There isn’t one experiment that will make you a major success, but rather a lot of little things that add up.
  • [05:36] Make sure to break down the experiments to focus on tiny aspects of your business.
  • [06:28] Running experiments over time lets you see what your strengths and weaknesses are.
  • [06:44] That’s all for today!
  • [06:46] Go to for a special marketing tool giveaway!

Leave some feedback:

  • What should we talk about next? Please let us know in the comments below.
  • Did you enjoy this episode? If so, please leave a short review.

Connect with us:


The post How Often Should You Run Experiments? | Ep. #650 appeared first on Marketing School Podcast.

Full Transcript of The Episode

Announcer: Get ready for your daily dose of marketing strategies and tactics from entrepreneurs with the guile and experience to help you find success in any marketing capacity. You're listening to Marketing School with your instructors, Neil Patel and Eric Siu.

Eric Siu: Welcome to another episode of Marketing School. I'm Eric Siu.

Neil Patel: And I'm Neil Patel.

Eric Siu: And today we're gonna talk about how often you should be running experiments. So, Neil, what are your thoughts around, I mean, I guess I can go first because we kind of just talked about this recently. My thoughts are you should be running experiments - assuming you have, to Neil's point, you have a good marketer on your team - you should be running experiments every single week, assuming you have some kind of maturity with your product or your service already.
So, I'll use as an example. Like, for our website, even though the main business is a services business, we're running experiments every single week on the site. We're running experiments on the business, as well, and then we're tracking these. You can track them on a spreadsheet - we use a tool called NorthStar, this is from Growth Hackers - and from there, we're just looking over time and we're recording our learning to see, okay, what percent of the time are we winning and how often are people right? And there's also a leaderboard in NorthStar, so it sounds like maybe I'm affiliated. Definitely not affiliated with the tool. I just like how it's set up. It's built for, you know, somebody that's starting out and then, eventually, somebody that is looking to build on a more kind of mature growth or marketing team.
So, my take on it is you have experiments every single week, you're tracking it, and you're trying to improve by one to three percent every single week. Of course, down the road, you're gonna have diminishing returns, but aim for that in the beginning and you're gonna be off to a good start.

Neil Patel: The more experiments you run, generally, the better off you are. There's a few problems, though.
One, if you don't get enough traffic, it's harder to run experiments, so you're gonna have to figure out how to do things like ramp up paid advertising, tweet things out, push things out on the social web. What you'll find is, once you get enough users or conversion points - and I'm talking about conversion points in the 500+ per month, that could be check-outs, it could be a lead opt-in - and if you're getting that, you can start running a good amount of experiments. You're roughly looking at one to two a month and the reason you can start running one to two a month is, A, you can look at micro-conversions and, B, you can look at macro-conversions.
So a good example of this is a micro-conversion could be someone going to your check-out page and getting more people there. A macro-conversion could be more sales in general, right? If you don't have enough traffic, you want to start with micro-conversion to get more people down your funnel. If you have enough traffic, you always wanna be optimizing for the end solution. The end solution or the end result, more so, not really solution, is the total number of conversions: total revenue, total leads, total sales, whatever it may be.
So try to run an experiment every week. If you're very small, track micro-conversions. Don't worry about macro. Once you start growing in size and you're getting 500+ conversions or a 1000+ conversions and real conversions - sales, credit card entries, you're getting leads, opt-ins, qualified leads, whatever you want to classify as a conversion - then you could be running more experiments each and every single week.
Now, in the short-run, if you can't really optimize for conversions and sales, your experimentations should be around traffic generation and you should do your best job guessing at what would be relevant traffic. You can use tools like SimilarWeb, SEMrush, [inaudible 00:03:38] to see what's working for your competitors and start looking at similar traffic channels.

Eric Siu: Yeah and, by the way, when we talk about experiments, we're talking about this in the context of marketing but, keep in mind, you can apply this framework to your business in general. So, for example, let's say you've been hiring on gut feeling but it's been very kind of subjective, right? You know, "Oh, I have a good gut feeling about this guy." Well, you know that that hasn't worked for you long-term. What you could do afterwards is you could put together a recruiting scorecard, so that, in itself, is an experiment.
And there's also this one book written by Peter Drucker - great executive, just executive in general - wrote a bunch of books around that. He wrote a book called Managing Oneself. It's literally only 40 to 50 pages. I highly encourage you all to read that, but that in itself, this guy - before A/B testing became really popular, before software as a service became really popular - he talked about the importance of feedback analysis.
So, for me, I have a spreadsheet that shows me over time all the big experiments I've ran. So, for example, maybe hiring someone that didn't have experience in a certain area, or moving my office somewhere, or you're traveling somewhere. These are all kind of big experiments I've ran for myself and you can see, over time, I've marked the ones that went well in green. I can see, for the most part, for me personally, I'm about 70, 75% or so on the big experiments. Now, this is going to show you over time for yourself - this applies for everyone - how you're doing, how you really do when it comes to running these big experiments for yourself.
If you can start with yourself, then roll it out to your marketing team and then your business as well, that's gonna help you across the board.

Neil Patel: If you wanna grow, what you'll find is it's all about experimentation and it's not one experiment that'll turn you into a Dropbox or a Facebook. Even with those companies, it's not one thing that's caused them to be worth ten billion or four hundred billion; it's a lot of little things that added up.
The trick that I use when I run experiments, each and every single week, I break them down so small that not only could I get the results within a week, but I can implement the experiment within less than a day, ideally within an hour or two.
The reason I try to keep it short is because that allows me to do a lot of them instead of combining a ton of them, running a huge experiment, taking forever for it to get out, and it may not work. When you do things little by little, what you'll find is, oh cool, this is working. Let me do more of it. Oh, this is not working. All right, that's great. I was gonna run three other experiments that are very similar. Now I know not to do them 'cause they just don't seem to work. This will allow you to move much quicker versus if you try to run really huge experiments at once.

Eric Siu: Yeah, and again, just to really emphasize, when Neil was speaking at [inaudible 00:06:17] we were at about a week ago, he was kind of lightly joking at himself saying, "Well, you know, I'm not the best at paid ads. You guys are much better than me. I'm really good at content marketing." And I think running experiments over time when you're tracking the learnings - again, you're being very methodical about it - you can see what your strengths are and what you're weak at. What you're weak at, you have other people come in and help you fill up or plug up those holes. You don't necessarily need to try to be strong at everything and Neil's very self-aware. I'm self-aware, as well, at what I'm good at and what I'm not good at.
Anyway, go to to check out our marketing tools to grow your business and we'll see you tomorrow.

Announcer: This session of Marketing School has come to a close. Be sure to subscribe for more daily marketing strategies and tactics to help you find the success you've always dreamed of.
And don't forget to rate and review, so we can continue to bring you the best daily content possible. We'll see you in class tomorrow, right here on Marketing School.

We help great companies grow their revenues

Get Your Free Marketing Consultation