Shopify CRO When Anyone Can Build Anything

AI made building new features into a Shopify store so much faster than it was a few years ago. An experienced developer with a Claude Code subscription can smash through their backlog in record time.

With this new speed, it's tempting to add every idea you have – leaving you with a cluttered store that’s worse than where you started.

I’ve spent over 15 years repeating my morning design mantras: “keep it simple”, “less is more”, “don’t make me think”. In CRO, some of my most successful A/B tests have been ones that declutter.

For the first half of my career I railed against product and eComm managers trying to add reviews, pushy recommendations, countdown timers, shipping messages, pay-in-4 options etc etc. Now I’m deep in the weeds of CRO with hundreds of A/B tests under my belt I can confidently say two things:

1. The design instinct to simplify is usually the right one.
2. Simple doesn’t always mean minimal.

Simple is what makes life easier for your customers. A review gives them confidence in your product. A robust filter helps wrangle an enormous collection down to a manageable browse. A well-timed and thoughtful recommendation can help build an outfit.

So how do you know what to add and what to remove?

You test it.

Building faster is making your store worse

Shopify stores adding the most features this year are not the ones converting best.

Think about everything best practice says to include on your product page:

• Reviews
• Low stock warnings
• Buy-now-pay-later badges
• Dynamic shipping message
• Relevant upsell recommendations
• Sustainability messaging
• Trending product badges
• Sizes as buttons
• Size guide
• UGC video
• Sticky add-to-cart button
• Wishlist

…all that and more, above the fold!

In the past, getting through that list would take months of work. Now that it takes a fraction of the time, it’s tempting to say yes to everything.

Just because you can, doesn’t mean you should.

None of these are inherently bad ideas. There’s strategic value and best-practice support for each one, but all together? The page becomes a chaotic mess – not to mention the performance impact.

Then you launch and try to figure out if it made a difference. Traditionally you’d publish your new theme, cross your fingers and neurotically refresh your Shopify analytics hoping to see a lift. 

The problem is everything else moves at the same time:

• You launch a new collection
• Your most popular product goes out of stock
• A competitor has a sale
• Your customer's buying habits shift with the season
• You change your paid media spend
• ... the list goes on.

Your month-on-month or year-on-year comparison is never going to accurately answer the question: did that change make a difference?

Conversion rate optimisation helps you figure out what to implement

Conversion rate optimisation means abandoning ‘launch and pray’ and instead using controlled experiments to work out which changes really make an impact.

An A/B test sends half your traffic to the current store and the other half to the new design.

We measure key metrics (add to cart rate, conversion rate, revenue per visitor etc.) to find out:

• Do low stock badges do anything?
• Do customers care about sustainability?
• Is video content worth investing in?
• Is this new search app better than native Shopify?
  

You get the numbers how much each change impacts conversion (up or down) and you decide from there.

An A/B test is the best gatekeeper you have against app bloat, slow page speed and a cluttered UI.

Absolutely – build all the features you want on your product page! But test one at a time before they go in permanently.

The risk of not testing changes

If you don’t test, you can damage your bottom line without knowing it.

Industry benchmarks put A/B test win-rates at around 1 in 3 at best. At face value, that seems like a failing CRO program: why bother if almost everything loses?

If you’re not testing you’re just publishing the same changes with no idea if they’re helping or hindering.

Think about the last 3 changes you made to your store:

• A product page redesign
• A free shipping threshold in the cart
• A new merchandising app

If only one improved conversion, what were the other two doing? At best, nothing at all. At worst, killing your conversion rate without you knowing.

A losing A/B test costs you two to four weeks of slightly lower conversion. An untested, published idea costs you indefinitely.

Great CRO solves problems

Creating a better experience for your customers is the goal. Sleazy conversion tactics may give short term boosts but leave a bad taste and don't do much for your reputation.

Spotting the right problem to solve is the tricky bit. Combine quantitative and qualitative research to spot the gaps in the experience.

Your data tells you what your customers do on your site – the gab between mobile and desktop conversion, where they're leaving the funnel, where they're dead-clicking, how far they scroll etc. That points you to where the problems are.

Qualitative research tells you why they do it. Interviewing your customers, watching as they (struggle to) shop on your site can be very revealing. A CRO audit is an excellent tool to benchmark your store against best practice, and a competitor review spots category norms your customers expect to see.

A test idea backed by both qual and quant has a much better chance of winning than a list of experiments spat out by ChatGPT.

Where to start

If you’ve been building a lot recently and found yourself with a store suffering from clutter and feature bloat, start with an A/B test that removes something. It can go one of three ways:

1. If the test loses, what you removed was helping people and should stay where it is – great validation.
2. If the test wins, it was in the way. Get rid of it.
3. If the test was inconclusive, it was doing next to nothing and can be removed without risking a drop in conversion

If you’re curious about A/B testing or interested in a CRO partnership and don’t know where to start, we’d love to chat. We’re a Shopify Platinum Partner agency that specialises in conversion rate optimisation.

We’d kick things off with a no-commitment CRO audit and go from there.