New research from the UNSW Business School has revealed that people jump to conclusions when they read studies with small sample sizes.
The study, which had a sample size of nearly 4000 participants, found that everyone from university students to business leaders can be misinformed when reading research drawn from a small sample group.
The research was led by Dr Siran Zhan, senior lecturer in the School of Management and Governance, who wanted to show how people often make incorrect assumptions when reading studies. She wanted to emphasise the significant implications of statistical findings, and encourage both journalists and the general community to read information with an analytical and critical eye, ensuring everyone is properly informed about the world around them.
The study
Dr Zhan, alongside her co-author Dr Krishna Savani, found that people tend to ignore sample sizes and take the conclusions of research papers as fact, even when drawn from studies with as few as three participants.
“What surprised us was that when we examined samples of university-level statistics students and seasoned senior executives who are supposedly trained in their education or professional work to make judgements and decisions according to sound statistical principles, they ignored the sample size just as much as the public,” says Dr Zhan.
“It’s especially appalling to think many important businesses and public policy decisions might have been made based on unreliable results from small samples.”
Dr Zhan also suggests that people aren’t being empowered to use their judgement properly, and understand what counts as evidence. This can lead to the spread of misinformation, and false claims.
However, it’s not all bad news, because the research team have also come up with ways to prevent the spread of incorrect information, and better educate the public on what counts as good research.
The importance of sample sizes
“People’s general tendency to be unduly confident in conclusions drawn from tiny samples is incommensurate with statistical principles and can lead to poor judgement and decisions,” explains Dr Zhan.
To prove this assertion, Dr Zhan and her team conducted six experiments involving a total of 3914 participants. In the experiments, they tested whether people pay attention to variations in sample sizes, and how this changes their trust in studies. As a result, it was found that individuals pay little attention to sample sizes, and it doesn’t contribute to their opinion on a research paper.
“Even with a sample size of three, participants’ mean confidence level was 6.6 out of 10, indicating that people have pretty high confidence in data collected from incredibly small samples, consistent with prior research.”
This is a worry for many, as misinformation continues to be spread rapidly online. Researchers are concerned with the judgements people are making about what they’re presented with in the media, and how this may impact society.
Statistically, bigger sample sizes lead to more accurate data, so it’s more important than ever that the public is made aware of the significance of sample sizes, and how to ensure they’re getting accurate information.
Solutions
Judgements and biases regarding research affect everything from the media, to public policies, to workplaces.
To prevent the continued spread of incorrect conclusions, Dr Zhan recommends that all statistics should be accompanied by an interpretation from a trusted academic
“We recommend more statistical advice (i.e., a layperson interpretation of the strength of evidence statistics) to be provided to aid their interpretation of findings from samples and, ultimately, decision-making.”
However, she also acknowledges that many people don’t read the papers themselves, looking to the media or the internet for their information. To improve the effectiveness of these outlets, she suggests that the public should be better educated on what makes strong evidence and how to question claims made in research. By putting the education of the community first, we can all be better prepared to analyse and make accurate judgements about the information we’re presented with, ensuring everyone is well informed.
To learn about other interesting research projects, click here.