How to Read a Traditional Medicine Study: Questions to Ask
Nine questions to ask about a study of an herb or traditional practice, such as what type of study it was, what was compared, how the product was identified, and what was measured.
Content last updated . This page has not been medically reviewed.
Why a checklist helps
A news story or product page may say "a study shows" a herb or practice helps. That sentence can describe anything from cells in a dish to a large trial in people. This page gives nine questions that work for studies of traditional medicines. They do not replace advice from your doctor or pharmacist, and one study rarely settles a question.Nine questions
1. What kind of study was it? NCCIH describes basic research (testing ideas, often in the lab), translational research (building on basic findings), clinical trials in people, and systematic reviews that bring together many studies. [1] Clinical trials give the clearest information about whether a treatment is effective and safe in humans. [1] A review of Siddha formulas for cancer, for example, found promising results in laboratory cell tests, and its authors still called for well-designed clinical trials. [2] A cell result is a lead, not a treatment.2. Was there a comparison group?
In a controlled trial, one group gets the treatment and another gets standard care or an inactive substance. [3] In a double-blind study neither participants nor researchers know who is in which group. [3] NCCIH calls the randomized, placebo-controlled trial the "gold standard" for testing interventions in people. [4] For acupuncture, the comparison is a sham treatment; in NCCIH's summary of reviews, the gap between acupuncture and no treatment was bigger than the gap versus sham. [5]
3. Was the plant or formula clearly identified?
A consensus statement from editors of seven journals lists the full taxonomic validity of the material under study and appropriate controls as basics of good research. [6] The reporting guideline for Chinese herbal formula trials introduced the idea of the traditional pattern and the features of formulas, and elaborated the item on interventions. [7] If a paper does not say which species, part and preparation were used, you cannot tell whether a product you can buy is the same thing.
4. Did the study use the tradition's own method, or a simplified version?
Traditional systems often tailor treatment to the person. A survey of 378 Kampo trials in Japan found 84.7 percent did not use Kampo diagnosis. [8] That does not make those trials wrong, but it means they tested a formula given to a group, not necessarily the tradition as it is practiced.
5. What was measured, and does it matter to you?
A good trial measures outcomes patients care about. In a Cochrane review of Chinese herbal medicine for infertility with PCOS, no trial reported live birth, the main outcome the review wanted. [9] Check whether the result is a symptom you can feel, a lab number, or a score invented for the study.
6. How big and how long was it?
In the Cochrane vascular dementia review, trial sizes ranged from 26 to 240 participants with a median treatment duration of 12 weeks. [10] Small, short trials can point in a direction but are less reliable.
7. Were harms reported?
In the PCOS review, only one study reported adverse events. [9] In the dementia review, adverse events were incompletely reported. [10] Silence about side effects is not evidence of safety.
8. Do results repeat, and who ran the study?
When many studies come to the same conclusion, that helps show results are reliable. [1] Researchers have also noticed unusually uniform positive results from some countries and suggested publication bias as one possible explanation. [11] Look for several studies by different groups.
9. Does the product on the shelf match the product in the trial?
NCCIH reports Chinese herbal products found contaminated with undeclared drugs, heavy metals or incorrect herbs. [12] A trial of one carefully made preparation does not vouch for every brand. Our page on reading traditional use claims on labels covers product wording.
Reading a review's certainty rating
Cochrane reviews rate how certain the evidence is using the GRADE method. [9] For example, the PCOS review rated most comparisons very low, and the dementia review described moderate to very low quality evidence of benefit and harm. [10]Where to find studies
PubMed, run by the US National Library of Medicine, lists publication information and in most cases brief summaries of articles from scientific and medical journals. [4] Our own Research Library collects study records. If a claim cites no study you can find, treat it as a claim only.Related pages
Does traditional medicine work?, quality of research on traditional medicines and why long use is not proof.Frequently asked questions
Is a study in rats or cells proof that a herb works in people?
No. NCCIH says clinical trials in people give the clearest information on effectiveness and safety. [1]
What is a sham or placebo control?
A placebo is an inactive substance that looks like the treatment, and the placebo effect is a beneficial outcome from the person's expectation that an intervention will help. [4]
Why do reviews call evidence "low certainty"?
Cochrane uses a method called GRADE to rate certainty. [9] The vascular dementia review found generally poor reporting and methods. [10]
Should I trust a single positive trial?
Treat it as a start. When many studies reach the same conclusion, that helps show the results are reliable. [1]
Can I take a herb because a trial was positive?
Ask your doctor or pharmacist first. NCCIH says herbs can interact with medicines, and St. John's wort in particular can weaken the effects of many important medicines. [13]