Process - How Health Feedback works

Our Workflow at a glance:

Select an article for assessment, according to subject matter, relevance, and potential digital influence
Call on scientists with relevant expertise to participate in article analysis
Analyze selected article, with a focus on assessing (claimed) fact-based assertions and scientific reasoning
Evaluate the article’s overall credibility against current state-of-the-art knowledge and thinking in science
Summary & Feedback
Help promote our analyses!

1. Select

Health Feedback editors select articles for review that are the most viral on social media and/or are published in outlets with large readership. Either must contain potentially verifiable claims in the scientific realm.

If you think we missed an important claim or article, you can suggest it for review here.

2. Invite reviewers

The editor in charge of overseeing the review process invites relevant experts to contribute to the review.

3. Article analysis

Reviewers are asked to comment on the article to indicate whether the facts underlying the reasoning are consistent with up-to-date scientific knowledge. Reviewers should represent the state of knowledge in the scientific literature, using strongly supported scientific theories and observations as references, and refrain from pointing to partial/isolated/weakly supported findings.

The primary objective of scientists should be to check the scientific accuracy of the facts and assumptions on which the article is based. However, their contributions need not be limited to fact-checking and could extend to:

Adding Relevant Information: Signaling additional resources or information related to the text, providing context, or indicating a perspective based on a scientific publication or clinical experience can also be the purpose of a comment.
Evaluating Scientific Reasoning: Misleading arguments are often based not on erroneous factual propositions, but on flawed/misleading reasoning or hidden and weakly supported assumptions. Reviewers are also invited to reflect on this aspect: promoting logic, scientific reasoning, and the fact that conclusions should be supported by the argument made…

Reviewers should pay attention to these “credibility criteria” to guide their evaluation:

Factual Accuracy. Does the article build on scientifically accurate information and solid evidence?
Scientific understanding. Beyond accurately describing basic observations, does the article correctly interpret these observations and correctly state what’s currently known/unknown about human physiology?
Context and limitations. If the article is about an interventional treatment (drug, surgery, therapy, screening test, diet change, etc), are the costs, risks and side effects adequately discussed, and are benefits not overstated or overly simplified? Or is the article over-hyping the significance of the benefits?
Logic/Reasoning. Do the conclusions follow from the evidence? Are there gaps or leaps in the reasoning?
Precision/Clarity. Does the article accurately refer to medical concepts and appropriately translate medical jargon? Or is there some confusion, which might mislead the reader?
Sources Quality. Does the article rely on adequate and credible sources (use of relevant/independent experts, studies) to back up important claims? Or is it relying on experts with a conflict of interest?
Fairness/Objectivity. Does the article depict a complete or imbalanced view of the relevant science? Does it give precedence to certain sources without an appropriate reason for doing so (e.g. lacking the necessary expertise and/or credentials)?

4. Evaluation

To help readers better understand the comments made, we invite scientists to provide an assessment regarding the overall scientific credibility of the article. The following criteria are used to help guide the evaluation:

Suggested guidelines for the overall scientific credibility rating

Remember that we do not evaluate the opinion of the author, but instead the scientific accuracy of facts contained within the text, and the scientific quality of reasoning used.

+2 = Very High: Little to no inaccuracies, fairly represents the state of scientific knowledge, contains appropriate references or links. The article provides insights to the reader about health or medical science, mechanisms and implications, as well as limitations and important unknowns surrounding the evidence.
If article is primarily about an interventional treatment (drug, surgery, therapy, screening test, diet change, etc), the costs, risks and side effects are adequately discussed and benefits are not over-stated or overly simplified (for example: Article does not use cause-and-effect statements to describe an observational study).
+1 = High: The article does not contain major scientific inaccuracies and its conclusion follows from the evidence provided. While more detail would have been useful, readers are still accurately informed of the science.
If about an intervention, article at least attempts to establish the risk-to-benefit ratio, albeit briefly.
0 = Neutral: No significant errors, but not enough insight either to inform the reader.
(Ex: Article does not misstate findings from observational study but does not point out experimental research is needed to confirm findings; article doesn’t point out that unpublished research findings aren’t peer-reviewed, article correctly summarizes an interventional trial of a drug, but never mentions its high costs.)
-1 = Low: The article contains significant scientific inaccuracies or misleading statements.
For interventions, article stops short of blatant hype or fear-mongering by providing at least some balance, whether that’s by discussing costs, risks, side effects, alternative well-tested treatments, or study limitations.
-2 = Very Low: The article contains major scientific inaccuracies for key facts supporting argumentation, and/or omits important information, and/or presents logical flaws in using information to reach conclusions.
Information is so inaccurate it may stoke false hope among desperate patients (ex: mouse study extrapolated to have human benefits, predicting benefit from phase 1 trial, etc) or cause unnecessary harm (physical, emotional or financial).
n/a = Not Applicable: The article does not build on scientifically verifiable information (e.g. it is mostly about policy, politics or opinions).

The final Health Feedback rating is the average of all the reviewers’ ratings; in cases when reviewers largely disagree no rating is issued and the review is left as ‘debated’.

Credibility criteria, definition of the keywords

Accurate: free from factual errors, describes reality in a way that is consistent with available data/observations.
Inaccurate: contains statement of fact in direct contradiction with available observations/data.
Insightful: offers a deep understanding of the issue based on accurate information and proper context that clarifies the implications of observations.
Misleading: offers an incorrect impression on some aspect(s) of the science, leaves the reader with false understanding of how things work, for instance by omitting necessary background context.
Biased: holds some ideas (persons) as true (right) without proper justification, lack of objectivity, ideological.
Unbiased: not biased, impartial, weighs evidence for/against ideas
Sound reasoning: conclusion follows from the evidence presented.
Flawed reasoning: conclusion does not follow from the evidence presented.
Cherry-picking: highlights only a subset of all the available relevant evidence that seem to confirm a particular conclusion, ignoring a significant portion of evidence that would contradict it.
Exaggerating: overstates / exaggerates the significance of some findings. (e.g. claims that a new scientific study overturns previous knowledge while it is an incremental update)
Inappropriate sources: relies on low credibility sources, provides no or insufficient evidence in support of claims made.
Imprecise/Unclear: uses ill-defined terms or lacks specifics so that one cannot unambiguously know what is meant without making additional unstated assumptions.
Clickbait headline: article does not appropriately support its title.
Conflates facts and opinions: presents opinion as fact or fact as opinion.
Lack of context: lack of observations or explanations that would change the reader’s takeaway.
Undisclosed conflict of interest: article fails to disclose a conflict of interest with a strong likelihood of influencing a source’s conclusions.
Misrepresentation of sources (strawman): Substitutes a misrepresentation of a source’s conclusion for its actual conclusion, often in order to make it easier to discredit the idea of an “opponent.”
Overstates scientific confidence: Presents a conclusion as conclusive while the hypothesis is still being investigated and there remains genuine scientific uncertainty about it.

The methodology is adapted from Climate Feedback and HealthNewsReview.org.

5. Summary & Feedback

After the review process is complete, the editor writes a summary of the most salient points brought by the scientists, which is promoted on our website, across our social platforms, and shared with key media and scientific partners.

The Health Feedback editor provides feedback to the journalists and/or editors of the outlet of the original article reviewed.

6. Publicize

You can help us amplify the voice of science by following us on social media, sharing our content and signing-up to our email notifications!

Corrections

We aim for our “feedbacks” to be as accurate and up-to-date as possible. If we discover a mistake has been made, we will correct it as soon as possible and a note will be added on the original item. If you think we’ve made an error or missed some relevant information, contact us.

Suggest an item to review

If you wish to submit a suggestion of an article or claim to review, please use this online form. Please note that we focus on reviewing claims and articles that are scientifically verifiable and that reach large audiences.

Process – How Health Feedback works