Evaluating Test Outcomes in E-commerce

If an A/B test is 'inconclusive', it does not necessarily mean that the change does not work. It rather just means that you have not been able to prove whether it works or not. It is entirely possible that the change does have an impact (positive or negative), but that it is just too subtle for you to detect with the volumes of traffic you have. Mostly though, subtle (if you could detect it) would still be meaningful in terms of revenue. If you discard everything which is inconclusive, how do you know you are not throwing away things which would be worth implementing? So what to do? Well, experimentation is really about degrees of risk management. If you cannot prove the positive benefit of a change, then the first thing is to accept that the risk surrounding that decision is greater. BUT, you can understand the parameters of that risk. The image is from the awesome sequential testing calculator in Analytics Toolkit, created by Georgi Georgiev. This is the analysis of an inconclusive test, which is nevertheless able to show, based on what was determined by the observation, that there is a 70% likelihood of the effect falling between around -8.5% and +5%. This particular case is vague, but at least you know the boundaries of the risk you're playing with. In some cases the picture is more heavily skewed in one direction. An A/B test is a way of making a decision, and the outcome of that test is always simply an expression of the degrees of confidence you can have in making that decision. How you make the decision is always still up to you. #cro #experimentation #ecommerce #digitalmarketing #ux #userexperience

A few years back, I was working with an e-commerce client who was struggling with low conversion rates. We decided to take a deep dive into user behavior to identify pain points. Using Hotjar, we were able to see exactly how users were interacting with their website. We noticed that many users were dropping off during the checkout process. By analyzing heatmaps and user recordings, we identified areas where the checkout flow could be simplified. We used Google Optimize to test different checkout variations, such as reducing form fields and streamlining the payment process. These small UX improvements led to an 17% increase in conversions. Have you ever used user testing tools to identify and fix conversion bottlenecks on your website?

How I Helped My E-Commerce Client Measure Postcard Mailing ROI with A/B Analysis A client approached me with a common challenge—do postcard mailings actually increase revenue, or are they just an unnecessary expense? They wanted data-backed proof before deciding whether to continue the campaign. Using A/B testing, I conducted an in-depth analysis to compare the revenue impact between customers who received a postcard vs. those who didn’t. The Approach: 🔹 Data Preparation – Cleaned and structured customer transaction data from multiple months. 🔹 Segmentation – Categorized customers into two groups: Group 1: Received a postcard. Group X: Did not receive a postcard. 🔹 T-Test for Statistical Significance – Used statistical analysis to determine if there was a real impact on revenue. Key Findings: → Customers who received postcards (Group 1) had higher revenue per customer compared to those who didn’t (Group X). → Despite Group X having more customers, their revenue contribution per customer was lower. → T-Test results confirmed a statistically significant difference—proving the postcard campaign had a measurable impact. Final Insights & Recommendation: → The postcard campaign positively influenced revenue. → It’s worth continuing and optimizing for better targeting. → Future tests should explore personalized postcards or different frequency strategies. What This Means for Businesses By using data-driven A/B testing, businesses can move away from assumptions and make decisions with real evidence. This method isn’t just for postcards—it applies to ads, email campaigns, pricing strategies, and customer retention efforts. When you track what works and what doesn’t, you’re not just spending on marketing—you’re investing in profitable growth. By applying data analytics and A/B testing, I provided my client with clear insights to make an informed decision—turning what seemed like a guessing game into a data-driven strategy. ------------------ Are you tracking your marketing ROI the right way? Let’s connect and analyze your campaigns! #Ecommerce #MarketingAnalytics #DataDriven #ABTesting #CustomerInsights #ROI

The data in the table below 👇, from a recent journal paper, is mathematically impossible! If 95.6% of sessions have no purchase (a typical number in e-commerce), then to average the reported $3.49 per session, sessions with purchase must have a mean of $3.49/(1-0.956)=$79.32. With such a mass at $0, the SD (Standard Deviation) must be high: the minimum SD happens when the purchasing sessions spend exactly the mean, that is, $79.32, and the minimum SD is therefore 16.3. An SD of 2.86 is impossible! I asked the authors, and they confirmed it’s a typo. The SD should be 20.86. Why is this important? Because of statistical power. - With an SD of 2.86, the power formula for an MDE (minimum detectable effect) of 5% with 80% power implies that you need 16*2.86^2/(3.49*0.05^)^2 ~ 4,300 sessions per variant, and the paper has about 18,000 sessions per variant, so it looks sufficiently powered. - With the corrected SD of 20.86, and you need over 228,000 sessions, so the experiment is highly underpowered.  The authors indeed confirmed that the treatment effect is not significant using a standard t-test. Takeaway: in e-commerce with ~5% conversion rate, the rule of thumb is >= 200K users (http://bit.ly/CH2022Kohavi). When you see far fewer users (or in this case, sessions, which are not even independent), it’s likely underpowered. Paper: https://lnkd.in/gtsZDcS6. Thanks to Nico Neumann for the pointer, and to Xing Fang (no LinkedIn) for confirming the typo and writing that he will notify the journal. Thanks to Alex (Shaojie) Deng and Nathaniel Stevens for the discussion. This “impossible SD” could be a check to add for tools used by Data Colada (Uri Simonsohn, Joseph Simmons, Leif Nelson) and a nice exercise for students learning A/B testing (Dean Eckles, guido imbens, Andrew Gelman).