Ryan Lucht on LinkedIn: CMS Integrations: A Better Way for Marketing Teams to Run A/B Tests - Eppo… | 12 comments (2024)

Finally played a "full" run-through of Lukas Vermeer's thoroughly fun simulation game "So You Think You Can Test?" Pic below shows how I did. (This was on the default settings - instant feedback, 91 days) Link below if you want to play it for yourself - also starting a public petition for a new leaderboard sans Twitter integration 😉 But wanted to share a couple of self-observations from the playthrough: ❗ I made most of my decisions to ship *without* reaching a p < 0.05 threshold. Since I only had a 90-day window to maximize sales in the game, I aggressively looked to make ship/no-ship decisions by around 3 weeks into each test. 🎲 Despite the risk-taking, I only shipped one test with an actual impact below 0. Worth noting that this happened late in the game too when I could feel the decision fatigue setting in and I started to get more reckless 😅 I'm sure that playing with a 365-day window (and not when I'm rushing through since I'm distracting myself from my workday) would incentivize more cautious behavior. 📊 I also assumed that trying to prioritize tests based on their category was more of a distraction than a help. Most of the idea categories ended up with a wide distribution of outcomes in my game, and it's almost certainly better (if we want to maximize potential upside) to take as many swings at bat as possible... hence I prioritized almost solely by how many days it would take to get each test launched. 💸 Even though I "know better", I found myself making instinctual decisions that were largely based on point estimates. I was very likely to pause a test showing a p-value of something like 0.3 with a negative point estimate after 14 days compared to a test with a largely overlapping confidence interval but with a positive point estimate (which I often let run). A fascinating tool for studying your own biases in decision-making under uncertainty, pressure, and incentives! Give the game a run yourself, let me know how much you beat me by 😆 https://lnkd.in/eevd7xPX #experimentation #statistics #decisiontheory #gaming

62

30 Comments

Ask any company that puts their ideas to the test, and they'll tell you that most ideas fail. Put another way - most code that is written for new features, improvements, or initiatives doesn't stay in the production codebase. It just doesn't prove business impact. That means you can unlock tons of velocity and growth if you orient engineering efforts around impact and not just code craft. It's why Alexey Komissarouk is launching a new Reforge course on building Growth Engineering teams. If executed properly, these teams can consistently drive target metrics to increase by 10% or more every quarter.I highly recommend checking it out - Alexey is a brilliant operator and instructor with some great, applicable frameworks built from his experience at companies like Masterclass and Opendoor. https://lnkd.in/eJ8WyjVJ

16

Last week we launched a super exciting product at Eppo: contextual bandits. We've been hiding a major secret about it though. Everybody thinks it was built by all of our whipsmart statisticians and engineers and design teammates. It was actually built by my 10lb rescue shih tzu Captain. Pic 1 is him passed out one late night while reviewing the literature on bandit algorithms, Pic 2 is celebrating launch day. Check out his handiwork: https://lnkd.in/eDVegBx6Happy National Pet Day 😉

66

6 Comments

Congratulations to the entire Eppo team today! Incredible to see all these new roles finally announced, couldn't have a better day to do it.

10

I’m happy to share that I’m starting a new position as Head of Compliance and Risk Management, P-Values > 0.05 at Eppo!

This content isn’t available here

Access this content and more in the LinkedIn app

100

12 Comments

The marketing or product team needs to make their user experience personalized. Should you reach for a recommendation system or a contextual bandit algorithm? I wasn't sure of the distinction either! So I bothered Sven Schmit, repeatedly, to help me learn about the ideal use cases for each and the key considerations that can help you pick the right tool for the job. Then I wrote it all up in a handy blog post for everyone else :) The tl;dr - here's what you'll need to consider: -How big is the problem at hand? Does it warrant the heft of a recommendation system, or are there fewer arms and decisions to make? -Do you have enough historical data to overcome a cold start problem? If not, you'd be best served by a contextual bandit-How many times do we need to make this decision for each user? You can read through all these distinctions (and a bunch more color) on the Eppo blog:https://lnkd.in/dUvKpdAy

26

6 Comments

Most experimentation tools on the market today boast feature lists that are probably >80% similar. Yet picking the right tool for your context (company, team, culture, use cases) is still obviously a make-or-break proposition. So what should you be looking at beyond the features list?How do you evaluate if a tool is strategically aligned with your needs? I came up with 6 suggestions to consider, based on my experience advising dozens of companies on their decisions (both at Eppo and in my years of consulting experience prior). https://lnkd.in/eZJjaEhi

27

1 Comment

Devastating but true. Don't fall prey to Big Math.

233

17 Comments

This was my pitch to promote our live demo of Eppo hosted by Lukas Goetz-Weiss in a few weeks. It was flatly rejected as "too cheeky" for our company page. If you're curious to see what a state of the art experimentation platform looks like, or interested in checking out Eppo for your company, join us on March 21st:https://lnkd.in/eX29cTS9 Showtimes also available on Moviefone

21

4 Comments