Peer review explained: Ayurveda Context

This presentation is intended to cover:
 What is ‘peer review’ process?
 Specific points that need special attention in a
research manuscript: Reviewer point of view
 Strengths and weaknesses of the ‘Peer review’ process
 Common problems w.s.r. to Ayurveda research
manuscripts
 Some interesting examples showing the strengths and
weaknesses of Peer Review process
2

Ayurveda research publications during
last few decades:
 Phytochemistry or Phytopharmacology: Major chunk of
the published work
 This neglected basic theories of Ayurveda like Rasa,
Guna, Virya, Vipaka etc.
 Limited work has been published on unique principles of
Ayurveda (Tridosha, Prakriti, Sara, Sattva, Diagnosis
etc.)
 Most of the MD(Ay)/ PhD theses remain unpublished.
 Some have been published as books and not as journal
articles.
3

Why is publication important?
 No publication, no progress in science
 If your results are not available for others, it is as if they
don’t exist
 No publication, no promotion
 Assessment of performance
 UGC /MCI guidelines speak of the importance of
publications
 No publication, no funding
 Funding agencies look for ‘publication profile’ of the
investigator
http://www.the-
aps.org/careers/careers1/EBSymposia/Publishing101EB2009.pdf
4

Reasons for limited number of quality
research publications in Ayurveda
 Limited exclusive journals available
 Limited awareness regarding the importance of journal
publications
 No training given in writing manuscripts during formal
Ayurveda education
 Limited awareness about the differences between Magazines,
Periodicals, Journals.
 Not following international standards/ important guidelines
while planning the research protocols and while writing the
manuscripts.
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2876920/?tool=pubmed
5

Stringent peer review process
6

What is peer review?
 ‘Peer review’ (refereeing) is a scholarly process used
in the publication of manuscripts and in awarding
funds for research.
 Editors and agencies use peer review to select and to
screen submissions.
 Publications and awards that have not undergone
peer review often raise suspicion.
http://www.wordiq.com/definition/Peer_review
7

The Peer Review Process
Scientists study
something
They write their findings
and communicate the same
to a journal
Journal editors review
and send the article for
peer review
Editors send reviewer comments to the
authors, who may then revise and
resubmit the article. Article may be
rejected at this point if it does not
maintain sufficiently high scientific
standards.
Reviewers read the article
and provide feedback
to the editor.
If the article finally
meets the editorial
and peer standards,
it is published in the
journal.
1
2 3
4
5
6
http://undsci.berkeley.edu/article/howscienceworks_16
8

Purpose of peer review
Serves two functions:
 Technical: Ensures that the quality of science is
sound
 Subjective: Is the science interesting, important,
relevant?
 Thus, influences what science enters the public domain,
where it is published and what impact it will have.
 Assists authors and editors in publishing quality
research.
http://www.parliament.uk/documents/post/pn182.pdf
9

Types of Peer Review: Blind (Closed)
 Single Blind:
 The author identity is disclosed to the reviewers
 Reviewer identity is NOT disclosed to the authors.
 Double Blind:
 Identities of neither the author, nor the reviewer are
disclosed.
 In both the cases, even after publication, the authors
won’t know who their reviewers were.
10

Open Peer Review
 Identity of the reviewer is disclosed to the
authors (and/or to the readers too, if
published)
 Journals like Nature and BMJ have
experimented on Open Peer Review process.
 Some journals like BMC Complementary and
Alternative Medicine publish complete
prepublication history of the article.
11

Types of Review (Pre/Post Publication)
 Pre Publication Review (Traditional: either closed or
open)
 Post Publication Review (Always open)
 Publish the manuscript first
 Perform open peer review
 Two good examples:
 Webmedcentral
 Faculty of 1000
12

When a reviewer receives a manuscript, what
questions does he/she ask?
 Am I qualified to perform the review for this manuscript? Is the
manuscript related to my area of specialization / research?
 Do I have any conflict of interest with the authors?
(If the author names and affiliations are known)
 Can I spare sufficient time for performing this review?
 Can I submit the review within the stipulated timeframe?
 If the answer to any of the above questions is a ‘NO’,
he/she declines to perform review.
13

IMRaD Components of a research manuscript
 Based on what was known and unknown, why
did you do the study? (Introduction)
 How did you do the study? (Methods)
 What did you find? (Results)
 What does it mean in the context of the existing
body of knowledge? (Discussion)
http://www.the-aps.org/careers/careers1/EBSymposia/Publishing101EB2009.pdf
14

Introduction
 Statement of the problem
 Has the problem / objective been clearly stated?
 Is the problem specific, important and relevant?
 Are the lacunae in the current understanding of the
problem stated?
 Has the previous work related to the problem been
consulted?
 How is this work different from the one that already
exists?
15

Introduction
 Statement of the hypothesis
 Is the hypothesis clearly stated?
 Is the hypothesis supported by the existing
literature?
 Has the methodology followed in the study
been stated?
 Have the key findings been stated?
16

Methods
 Have the study settings, population, the sampling
strategy, instruments, data collection methods, and
analysis strategies described?
 Are the methods employed appropriate for the study?
 Does this provide the readers with sufficient details
about the study methods so that the study may be
reproduced?
 Do the technical terms really match with the actual
methods followed? (e.g., ‘Randomly selected’)
http://intqhc.oxfordjournals.org/content/16/3/191.full
17

Example:
A simple
method of
Randomization
using MS Excel
http://www.hindawi.com/jo
urnals/ecam/2011/25185
0.html
18

Recommended reading:
 Reporting Randomized, Controlled Trials of Herbal
Interventions: An Elaborated CONSORT Statement
 Standards of reporting Ayurvedic clinical trials - Is
there a need?
 Workshop on a CONSORT statement for Ayuveda
 Four new approaches for validation of Ayurvedic
herbal drugs
24

 Efficacy and safety of Ayurvedic medicines:
Recommending equivalence trial design and
proposing safety index
 Drug and Cosmetic Act, 1940
 Drugs and Cosmetics Rules, 4th Amendment, 2008-
Rule 170
 WHO General Guidelines for Methodologies on
Research and Evaluation of Traditional Medicine
25

Statistics
 What test was used to determine whether differences
between 2 or more groups were significant?
 Is the test used appropriate?
 If multiple comparisons were done, what adjustment was
made in the level of significance accepted?
 Are all details, such as values of test statistics and
degrees of freedom, given along with p value?
 Are the degrees of freedom consistent with the sample
sizes stated or the particular analysis carried out?
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2186371/
26

 Is it clear whether the number given after
‘Mean ±’ is SD or SE?
 Do the numbers add up?
 For example, the sum of all the subgroups should equal
the size of the full group.
 Are the technical terms that are specific to the
methodology defined?
27

How common are such errors?
 11.6% (21 of 181) and 11.1% (7 of 63) of the
statistical results published in Nature and BMJ
respectively during 2001 were incongruent.
 At least one such error appeared in 38% and 25%
of the papers of Nature and BMJ, respectively. In
12% of the cases, the significance level might
change one or more orders of magnitude.
http://www.biomedcentral.com/1471-2288/4/13
28

 Choosing the right statistical test
 Introduction to Study designs
 Basics of Sample size calculation
 A book on Biostatistics
Are placebo-controlled trials in Ayurveda possible?
Daniel E Furst, Manorama M Venkatraman, B G Krishna Swamy, Mary McGann,
Cathryn Booth-LaForce, P Ram Manohar, Reshmi Sarin, Anita Mahapatra, P R
Krishna Kumar. Well controlled, double-blind, placebo-controlled trials of classical
Ayurvedic treatment are possible in rheumatoid arthritis. Ann Rheum Dis. August
24, 2010 doi: 10.1136/ard.2010.136226
29

Research Violations
 Is there any evidence of violations of the Guiding
Principles in the Care and Use of Laboratory
Animals?
 If the research involved human subjects, was the study
performed in accordance with the ethical guidelines?
Has the ethical clearance been obtained?
 Has the informed consent been obtained?
30

Some important guidelines
 ICMR Ethical Guidelines
 International guidelines for care of Animals
 International guidelines for research involving
humans
 Medical Ethics Manual
 WMA Declaration of Helsinki - Ethical Principles for
Medical Research Involving Human Subjects
31

Observations and Results
 Is the ‘Results’ section fairly straightforward and
factual?
 Are all results that relate to the primary question of
research given in detail, including simple counts and
percentages?
 Are the details like timeframe, response rate, sample
characteristics stated?
 Are the key findings and secondary outcomes if any,
stated?
32

Tables and figures
Figures:
Are they really required?
Are they sharp, with lettering proportionate to the size of
the figure?
Are there legends to explain the figures?
Is there any evidence of ‘manipulation’ in the photographs?
Tables:
Is the table understandable even without the text?
Can a table be simplified or condensed?
Can two tables be merged?
Should any table be removed?
33

Discussion
 Have the main findings of the study been stated
correctly?
 Have these findings been discussed with reference to
previous research?
 Do the interpretations match with the data?
 Have the implications of the new findings been discussed?
 Have the strengths and limitations of the study been
analyzed?
 Have the perspectives for future studies been offered?
 Are the data adequate to support these conclusions?
34

References
 Functions:
 To give credit to other workers
 To add credibility
 To help readers find further information
 Contribution of a paper (or an author) to the science is
often assessed on the basis of citations the paper/ the
author has received
 H index
 Citation management software (EndNote, Reference
Manager) are available and can be used.
35

References: Reviewer point of view
 Does the citation style match with the journal style?
 Have all the key assertions been referenced?
 Have the methods and instruments used been
referenced?
 Is the reference section unnecessarily exhaustive?
 Are there references to unpublished work / documents
like technical reports/ theses that the reader will have
difficulty finding?
 Have the references to web content been provided
with the correct links? (Does the link work?)
36

Presentation
 Overall Manuscript: Does it meet the ICMJE’s
‘Uniform Requirements for Manuscripts’ criteria?
 http://www.icmje.org/sop_1about.html
 Research Reporting Guidelines and Initiatives
 http://www.nlm.nih.gov/services/research_report_guide.html
 Writing:
 Is it clear, concise, and in good English?
 Many reviewers concentrate more on ‘language’ than on
‘content’ and simple errors in language annoy them.
 Many journals do not provide copyediting facility to authors
 Great science in an ugly package may still be rejected
37

Title
 Most widely read part of any published article
 Ideally, it should summarize the complete article
 Reader decides whether to read the abstract or not by
looking at the title
 Is the title complete, specific and covers the central
idea?
 Does it effectively summarize the contents of the study?
 Can a reader easily understand the title? (‘Rasayana
effect of Amalaki’ : Is it Immuno-modulatory effect or
anti aging effect?)
 Are there any unnecessary words in the title?
38

Abstract:
 Reader gets the idea of the complete paper by
reading the abstract carefully
 Reader decides to go through the paper only if
he/she finds the Abstract interesting
 Should be brief
 Should state the problem
 Should state the hypothesis
 Should state the methods
 Should summarize the results
 Should include inferences of the study
39

Key words
 Databases use keywords for indexing literature
 Users search a database by keying in the
‘keywords’
 So, more diversity in keyword selection means
more visibility
 Use MeSH thesaurus if not sure of keyword
usage
 E.g., ‘Autistic disorder’ is preferred over ‘Autism’
40

Indexing Methods
 Analytic Indexing:
 Indexers carry out indexing manually according to
their own understanding of the study contents. (e.g.,
MEDLINE, CINAHL, and MANTIS)
 Semantic Indexing:
 Computer programs pool words or phrases and
group them by frequency of use to generate
indexing. (e.g., Web of Science).
41

 The controlled vocabulary for biomedicine has
been developed and continuously updated by the
National Library of Medicine (NLM).
 It is referred to as the NLM Medical Subject
Headings (MeSH). The purpose of MeSH is to
provide uniformity and consistency to the indexing
of the biomedical literature.
42

More on Keywords and terminologies in
CAM literature
http://www.nlm.nih.gov/pubs/factsheets/mesh.html
http://www.biomedcentral.com/1472-6882/3/3/
43

What if reviewers fail to detect some errors?
 Errors may be detected after the publication
 Authors are encouraged to write to the editor if they
find errors after publication
 Communicate the sufficiently detailed Erratum to
rectify the errors.
 http://www.nlm.nih.gov/pubs/factsheets/errata.html
44

Research Violations
 Falsification – where data in manuscripts submitted
for publication are distorted or manipulated in
some way
 This can include ignoring ‘inconvenient’ results and
analysing data in inappropriate ways.
 Fabrication – where data or cases in manuscripts
submitted for publication are simply invented.
 Plagiarism – copying of data, papers or ideas.
45

Falsification: German cancer research
 In 1998, German research funding agency looked at
347 papers published by two researchers, Friedhelm
Herrmann and Marion Brach.
 It concluded that 29 of these contained falsified
material and found evidence of data manipulation
leading to a suspicion of fraud in a further 65 papers.
 In most cases the falsification of illustrations of blood
and other cells was noted.
46

Fabrication: The Pearce case
 In August 1996, Malcom Pearce, published a paper in the
British Journal of Obstetrics and Gynaecology.
 He claimed to have rescued an ectopic pregnancy by
transferring it into the uterus, resulting in a successful birth.
 The work had never taken place and the ‘patient’ did
never exist.
 Four other fraudulent papers were discovered after
investigation, two of which had been published in the BMJ.
 Pearce was terminated and struck off by the General
Medical Council.
47

Plagiarism: US National Science Foundation
 A researcher was asked to peer review a proposal for
research, which was later rejected on the basis of his
comments.
 This reviewer subsequently submitted his own research
proposal to another funding body, which was accepted
for funding.
 This proposal was found to have plagiarised the original
proposal.
 The researcher had submitted a number of other
research proposals plagiarised from proposals he had
been asked to peer review.
48

A common problem with Ayurveda
manuscripts
 ‘Cutting and Pasting’ is dangerous
 You cannot lift some one else’s paragraph without citing
the source
 Even if you cite, paraphrasing is necessary
 Reviewers ‘Google’ for suspicious language or
paragraphs for detecting plagiarism
 If you have to use some sentences as such, use them with
“ ” marks.
 To detect plagiarism, reviewers use different software:
http://www.plagiarismchecker.com/
http://www.ithenticate.com/
49

Research Violations
 Failure to disclose conflicts of interest.
 A study conducted in 1986 found that 96% of studies had
financial relations with the drug manufacturer.
 Other forms of scientific misconduct.
 Gift authorship.
 The senior author (Geoffrey Chamberlain) on the ‘ectopic
pregnancy’ paper had to resign from a number of senior
positions.
 Redundant publication (undisclosed publication)
 Authors publish the same paper in a number of different
journals)
50

Salami Slicing
 Publishing same data in different forms in different
journals
 Submitting neurological findings in a neuro-cysticercosis case
to a neurology journal
 Submitting radiological findings to a radiology journal
 Interferes in meta analyses as the same cases will be
duplicated
 Ethical only if the data is too huge and each paper
addresses substantially different questions
 Editors must be informed in all such cases regarding
prior publications
52

Common problems
 The kind of study is not suitable for the journal
(Decide the journal before writing the manuscript)
 Research question is not specific / multiple non related
questions addressed simultaneously
 The stated aim of the paper is vague (‘Establishing the
effect of Achara Rasayana’, ‘the aim of this paper is to explain
what we did…’)
 The structure of the paper is disordered (e.g. Results are
described in the methods section)
 The introduction is too long and unnecessarily reviews
the entire literature
53

Common problems
 Methods, interventions and instruments are not described
in sufficient detail (Study cannot be reproduced)
 (‘Shirodhara was performed once daily’.)
 What time of the day? For what duration? What was the
clinical setup?
 ‘Coconut oil was used as placebo in XYZ skin disorder’
 Has coconut oil been proven to have no ‘healing’ or
‘aggravating’ effect in this condition?
 ‘Placebo was given in the form of a capsule’
 What was the placebo? What are its effects on biological
system?
54

Common problems
 Results are reported selectively (e.g. percentages
without frequencies, P-values without measures of
effect)
 The repetition of results both in a table and in the
text
 Unrelated tables and figures are provided
 In the Introduction and Discussion, key arguments are
not backed up by appropriate references
55

Common problems
 The references are out of date / cannot be
accessed by most of the readers
 The Discussion does not provide an answer to the
research question
 The Discussion overstates the implications of the
results
 Discussion does not acknowledge the limitations of
the study
56

Common problems
 Though the study is good, it is not suitable for the
particular journal/ particular kind of
communication
 Case reports submitted as Research Articles, research
articles submitted as letters to editor
 Editorial requirements are not fulfilled
 Exceeds the word counts, Violates the citation style,
too many non-text items than that are allowed
57

 Sanskrit verbs used without English translation
 ‘Medhya effect of Mandukaparni’.
 Theoretical arguments that are not backed up by
sufficient evidence
 ‘All Bhasmas are Nanomaterials’
 Indiscriminate use of the technical terms without sufficient
evidence
 ‘Shodhana means Purification’, ‘the drug Triphala showed this
effect’
 No full forms of abbreviations given
 Ca.Su., Ca. Ci.
58
Common problems

Common problems
 ‘Lifting’ complete paragraphs from other resources,
(usually books and websites)
 Using graphs/ figures from other publications
without obtaining prior permission from the
copyright owners
 Even the sentences from translated version of
Caraka Samhita, (e.g., Translator PV Sharma)
cannot be copied without proper referencing
59

Common problems
 Inadequate designing/description of the study
protocol
 What was the qualification of the physician who
prescribed the control drug ‘Phenytoin sodium’?
 Were the antibiotics given along with the trial drug ‘XYZ
ointment’ in the cases of infective ulcers?
 No control groups in clinical trials involving very common
conditions like bronchial asthma, common cold
 What concentration of the solvent was used for extraction?
 Vaguely defined identification procedure of the herb
(Morphology/HPTLC-chemoprofiling/DNA fingerprinting)
60

Common problems
 Wrong citations of books / other documents
 E.g., ‘Anonymous. Ayurvedic Formulary of India’
 [Write ‘Anonymous’ only if the word ‘Anonymous’ appears in
the original source. Else, directly cite the title where there is
no author.]
 Same term spelt differently
 Vagbhata, Bagbhata, Bagbhatta, Charaka, Carak, Charak,
Makshika, Makhika
 Inadequate description of methods
 Inadequate interpretation of statistical results
61

What does a reviewer do?
Confidential comments
 Provides comments regarding the originality and
significance of the manuscript
 Appropriateness of significance (1,2,3,4,5)
 Appropriateness of methods/ study design (1,2,3,4,5)
 Appropriateness of conclusions (1,2,3,4,5)
 Appropriateness of the references (1,2,3,4,5)
 Completeness of the abstract (1,2,3,4,5)
 Language (1,2,3,4,5)
62

Recommendations
 Accept unaltered
 Revise and accept (Minor revision)
 Re-assess after revision (Major revision)
 Reject
63

Comments for authors
A reviewer provides specific comments,
preferably numbered, on the design, presentation of
data, results, and discussion etc.
64

Criticism against Peer Review
 Biased
 Gender Bias
 Confirmational Bias, Promotes only positive results
 Inefficient, slow, doesn’t encourage innovation
 Fails to detect fraudulent research
 Too much burden on a few good reviewers
 Non remunerative
 Abuse of Anonymity (e.g., Non disclosure of conflict
of interest)
65

Exposing the Inefficiency
 http://pdos.csail.mit.edu/scigen/
 SCIgen - An Automatic CS Paper Generator
 SCIgen is a program that generates random
Computer Science research papers, including
graphs, figures, and citations. It uses a hand-
written context-free grammar to form all elements
of the paper.
 (Generates a nonsensical paper)
 Several papers thus generated have been accepted
for conferences and have even been published !
66

One such paper is available on Google scholar
67

A very interesting study
 As test materials, investigators selected 12 already
published research articles, one article from each of 12
highly regarded and widely read American psychology
journals.
 They resubmitted the manuscripts with fictitious names
and institutions substituted for the original ones, to the
journals that had originally published them 18 to 32
months earlier.
68

 Of the sample of 38 editors and reviewers, only three
(8%) detected the resubmissions.
 Nine of the 12 articles continued through the review
process to receive an actual evaluation: eight of the
nine were rejected.
 Sixteen of the 18 referees (89%) recommended
against publication because of “serious methodological
flaws”.
Douglas P. Petersa1 and Stephen J. Cecia. Peer-review practices of Psychological
Journals: The fate of published articles, submitted again. Behavioral and Brain Sciences
(1982), 5: 187-195 Cambridge University Press doi: 10.1017/S0140525X00011183
69

Krebs Cycle Case
70
This discovery subsequently fetched him Nobel Award

Peer review explained: Ayurveda Context

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

Similar a Peer review explained: Ayurveda Context

Similar a Peer review explained: Ayurveda Context (20)

Más de Kishor Patwardhan

Más de Kishor Patwardhan (15)

Último

Último (20)

Peer review explained: Ayurveda Context