Our Knowledge Base

The greatest tool of the wise man is knowledge!

Overcoming Data Bias in UX Testing and Stats Analysis

by | Oct 17, 2022 | UX Design

So as a UX designer it’s important to let the evidence guide your decisions.

However sometimes your own bias or ignorance or determination to solve a small issue can influence your interpretation of the data at hand.  In short this hurts your ability to push the User Experience in the right direction.

I’ll give you a few examples which might help explain this and then talk you through some common mistakes UX designers can make.

1. Be careful making snap decisions

So, last year I went for lunch at an outdoor restaurant in Melbourne called Snitz on my way back from a meeting.

I ordered a meal with a drink and chips and then proceeded to find a seat on one of their long communal tables outside.  Once sitting down I put in my earphones and proceeded to watch a show on youtube while I ate.

As I started eating, a small man sat down opposite me. I didn’t pay too much attention to him, until I noticed him take one of my chips and eat it.  I froze in place and stared straight at him.

I don’t know if you know about the chips at Snitz but they are amazing, and I think in some countries you can probably be killed from eating another man’s chips.

Then again he ate another one of my chips. I tried to convey my disgust through my stare to no avail.

I stared at him but didn’t say anything, and then I made a decision to eat them before he managed to eat them all.

I took a chip and he stared at me and smiled in a funny way and then we ate the chips together.

It was a weird lunch, it was like we were both fighting to eat them before the other person did.  I should have said something, I should have pulled the chip bag away from him, but I didn’t.

On the way to work I was furious at myself.  I let this little guy eat half my chips, the nerve of this guy.  I got to the office, went to pull out my laptop, and then realised that my chips were still in my computer bag.

He wasn’t eating my chips… I was eating his chips. OMG!

Sometimes you are the problem, whatever your preconceived notion is about a situation can grossly skew your perspective in life and in UX.

So many times I’ve been sure at how the stats will come out only to be proved completely wrong once I see the data.  Human nature is to try to find any way to make the stats fit your hypothesis, but the reality is when the science shows the reality of a situation you need to embrace it and be open to change.

 And… it helps to remember where you put your chips.

2. Survivor Bias

This is a story I heard from Simon Sinek in one of his brilliant videos, check him out if you haven’t already.  In 1943 during WW2 the US Military was trying to figure out the best place to add amour to their fighter planes and bombers.  If they added too much armour the planes would become too heavy and not enough would risk them being shot down by the enemy in greater numbers.


They performed a statistical inventory of all the bullet holes across all the planes that had come back to determine where the planes were most likely shot.

Essentially, the idea was to strengthen the most commonly damaged areas of the planes.  The study showed that most of the bullet holes were on the wings, fuselage and fuel system, so it was thought that re-enforcing these areas would be the solution.  A brilliant mathematician by the name of Abraham Wald raised some concern and pointed out that they were mis-interpreting the data.

Perhaps the reason that those areas were not covered in bullet holes is that the planes that sustained damage in those other areas are the planes that didn’t make it back.  They were excluding a crucial part of the sample data and their conclusions were entirely wrong.

 In UX we hit this sort of thing all the time, we interview the users that made it through the flows and became customers, often not the ones that dropped out during the process.  We see more drop towards the end of a flow so we assume that there is something wrong with the final step, and not that the previous steps were too time consuming.  

To summarise, survivor bias is also evident in positive flow thinking.  When designing a flow it’s easy to ignore the negative flows, what happens if the credit card is rejected, what happens if the internet cuts out, what do the users see if there are no posts or contact yet on the page?


3. Tightening a flow is not always the best thing for long term retention

One other mistake I see many UX designers making is sacrificing long term retention for a more optimal onboarding flow.  I’ve made this mistake in my career, a good example was the onboarding flow for a social app we were building.


The onboarding process had 7 steps, including things like selecting some shows to follow, other users to follow, selecting categories you like.  After reviewing the stats and the onboarding funnel on the app we saw that we were losing 40% of users on these last 3 steps, so it made sense to cut these steps, tighten up the flow and lose less users during signing them up to the app.

What we found though is that our monthly rolling retention went down as a result.  Not as many people were coming back to use the app again and again like before.

Some of these steps built ownership and habits within the app for users, they gave the users something to follow and content to engage with and kick off their experience, which in tern kept them engaged and coming back.

This same thing happened to Twitter in their earlier years.

In 2006 they had 16,000 users.  They noticed early on that many users were joining but retention was very low, users weren’t returning more than a few times on average.  They did a similar thing we do in the first year of any project.  We interview the Whales.

By this I mean we take a look at the users that just get it, the ones who use the platform regularly, in gaming these are the users that buy the most inapp purchases.  We typically offer them a bottle of wine or gift to participate and we ask 5-7 key questions to determine how and why they use the platform.

Back to Twitter, they noticed that their long term users had followed a handful of accounts right after first signup.  Boom.  They added this step into their onboarding flow, forcing users to follow at least a few accounts and the retention rate skyrocketed.

By 2008 they had 4,120.000 accounts and never looked back.  You can see a similar thing today in most leading social platforms like instagram etc., to try to build habitual dependance in their user base.

In the end, we added these steps back in but after the initial flow had been completed if you’d like to check out the current app check out Goodpods on our work page.  As the user explored the app they were prompted to complete each of the final 4 steps.  So we got the best of both worlds, we got users in fast and we also ensured that they build some personalisation and ownership with the platform to bring the retention rates right back up.

In conclusion, UX testing and strategy is a science based endeavour.  You need to be able to look past your own bias, and keep your eye always on the long term goals of your client and the platform.  You’re not in it to win the battle you want to win the war and good UX is like having Hercules in your army.

Need some help with User Experience Design on your platform? We’d love to have a chat and give you some advice. Get in touch

More Articles

13 Tips for securing your Web Application

13 Tips for securing your Web Application

Web applications are an essential part of the modern internet, providing users with rich and interactive experiences. But as web applications become more complex and sophisticated, they also become more vulnerable to security threats. In this post, we will explore...

Top 5 Rules for Great UX Conversion Optimisation

Top 5 Rules for Great UX Conversion Optimisation

Firstly, what is conversion optimisation and why is it important? The aim is to encourage users to stay and convert by explaining the value proposition, and building in habitual dependance. Basically, every lead your business generates has a cost, let's say you spend...

Don't just take it from us!

We've had the pleasure to work with the AppNative team for the last few years. They were instrumental in redesigning our entire IT solution from a contemporary and cutting edge user interface to a sound and reliable administration system.

They have always gone beyond the scope of work to ensure successful product launches both here in Australia and globally, as well as support Moving Mindz in meeting our deadlines, and getting apps approved on various platforms. We now have a world class product that we are proud of and we are grateful to Appnative.

Shane Bilborough

CEO, Moving Mindz Pty Ltd

AppNative have been fantastic to work with on our new Ineda platform and over the years.

They've helped us build out the front-end of our customer platform as well as run our teams strategy sessions, sprint standups and retros, as well as run quality assurance testing for our products. 

The AppNative team has been easy to work with and a great way to supplement our internal team with the extra skills we need.

Nick Neda

CEO / Founder, Ineda & BeEnergy

We engaged the AppNative team after doing a search in the Australian market for a local company willing and motivated to bring an innovative idea to life that no one else had achieved. We had many hurdles to get through and the fact we were able to apply for both a patent and a research and development grant is a testimony to what we managed to build together.

The team were as passionate as we were, coming up with ideas that would compliment the solution, but also knew when to pull back if there was scope creep to ensure we remained on track with our MVP. We’re extremely proud of the product we built in partnership with Ben and his team. 

Skye Butler

CEO, Smylee

AppNative is made up of industry veterans with incredibly creative minds and have easily the best UI/UX designers in the world.

Having a deep understanding of modern development and technology coupled with genuinely world-class creative abilities means that AppNative creates visually stunning solutions that are beautiful and intuitive.

AppNative has gone above and beyond assisting us with building not only our product but our team. Their integrity, network, talent, and problem solving have rightfully given him the reputation he has, and our experience with AppNative could not have been better. 

Steven Bolton

CEO, Entrepreneur

Clients we've worked with

client logos