A p-value is not the whole story. To use research well, you need to read the results, not just the headline.
Interpreting a study's results is where many well-meaning practitioners go wrong, either dismissing good evidence or over-trusting a weak finding. This guide covers the five things to look at in a results section: sample size, effect size, statistical significance, confidence intervals, and consistency with other research.
It is part of our short series on the science of hypnotherapy, alongside why evidence matters, how to read a scientific paper, why the randomised controlled trial is the gold standard, and where to find the journals.
1. Sample size
Sample size is the number of participants or observations in a study. Larger samples generally give more reliable results, because they reduce the influence of random variation and make the findings more likely to generalise.
Imagine a study comparing two hypnotherapy techniques for anxiety. With only 10 participants, an apparent difference could easily be a fluke. With 200, the result is far more likely to reflect something real. Small studies are not worthless, but their findings should be held more loosely until larger studies confirm them.
2. Effect size
Effect size measures the magnitude of a difference or the strength of a relationship. It answers a question that statistical significance does not: not "is there an effect?" but "how big is it?"
Suppose a study finds a statistically significant difference in anxiety reduction between two techniques. A small effect size might mean the difference, though real, is too slight to matter to a client. A large effect size means one technique is substantially better. Always look for the effect size, because a real effect can still be too small to be useful.
3. Statistical significance, and what a p-value is not
Statistical significance is a measure of how likely the observed result would be if there were no true effect. It is usually reported as a p-value, and a p-value below 0.05 is the common threshold for calling a result statistically significant.
This is one of the most misunderstood ideas in research, so it is worth being precise. A p-value of 0.03 means that, if there were genuinely no effect, results at least this extreme would occur about 3% of the time. It does not mean there is a 97% chance the treatment works, and it does not tell you the effect is large or important (Wasserstein and Lazar, 2016). Statistical significance and practical significance are different things, and a well-designed study reports both.
4. Confidence intervals
A confidence interval gives a range within which the true value is likely to lie. A 95% confidence interval means that, across many repeated studies, about 95% of such intervals would contain the true value.
For example, a study might report that one technique produces 5 pounds more weight loss than another, with a 95% confidence interval of 2 to 8 pounds. That tells you the likely size of the effect and how precise the estimate is. A narrow interval signals a precise estimate; a wide one signals uncertainty, and an interval that includes zero means the study cannot rule out no effect at all.
5. Consistency with prior research
A single study is rarely the last word. Ask how the result fits with what came before. Do several studies point the same way, or does this one contradict the rest?
If a new study on hypnotherapy for smoking cessation agrees with earlier positive findings, that consistency strengthens confidence in the conclusion. If it contradicts them, look at the differences in design, methods and populations before deciding which to believe. Consistency across independent studies is one of the strongest signals that a finding is real, which is why systematic reviews that pool many studies carry so much weight.
Frequently asked questions
What does a p-value actually mean?
It is the probability of seeing a result at least as extreme as the one observed if there were truly no effect. A value below 0.05 is the usual threshold for statistical significance. It is not the probability that the treatment works, and it says nothing about how large the effect is (Wasserstein and Lazar, 2016).
Why does effect size matter if a result is already significant?
Because significance only tells you an effect is unlikely to be due to chance, not that it is big enough to matter. A large study can find a statistically significant but tiny effect that makes no real difference to clients. The effect size tells you whether it is worth acting on.
What is a confidence interval?
A range that is likely to contain the true value, usually at the 95% level. Narrow intervals mean precise estimates; wide ones mean more uncertainty. If the interval for a difference includes zero, the study cannot confidently say there is any difference.
Should I trust a single study?
Cautiously. Look at its sample size, effect size and confidence intervals, and check whether its findings are consistent with other research. The strongest evidence comes from randomised controlled trials and from systematic reviews that combine many studies.
Keep building your research literacy
At Jacquin Hypnosis Academy we think reading results critically is a core professional skill. Continue with the rest of this series: why evidence matters, how to read a scientific paper, why the randomised controlled trial is the gold standard, and where to find the research. You can also trial our training platform free for 14 days.
Sources and further reading
Wasserstein, R. L., & Lazar, N. A. (2016). The ASA statement on p-values: Context, process, and purpose. The American Statistician, 70(2), 129-133. https://doi.org/10.1080/00031305.2016.1154108