1. Introduction to
Signal Detection
by Rodney L. Lemery MPH, PhD
Copyright  BioPharm Systems, Inc. 2011. All rights reserved.

2. Topics…
• Definitions
– Pharmacoepidemiology
– Pharmacovigilance
– Signal
– Signal Detection
• Signal Detection
– Qualitative
• Striking cases
• Periodic reviews
– Quantitative
• Analysis of disproportionality
Introduction to Signal Detection 2

3. Topics…
• Signal Detection (cont.)
– Disproportionality analysis tools
• Signal Prioritization
– WHO Triage
– MHRA Impact Analysis
– Categorization of priority
• Confirmed Signal
• Unconfirmed Signal
• Refuted Signal
• Signal Evaluation
– Causality
– Frequency
– Clinical implications
– Preventability
Introduction to Signal Detection 3

4. Acronyms and General
Definitions
• Prevalence – The total number of cases of a disease in
a given population at a specific time
• Incidence – The number of newly diagnosed cases
during a specific time period
• EMA – European Medicines Agency
• ADR – Adverse Drug Reaction
• APR – Adverse Product Reaction
• ICH – International Conference on Harmonization
• CIOMS – Council for International Organizations of
Medical Sciences
Introduction to Signal Detection 4

5. Definitions…
• Two pervasive definitions (Abenhaim, Moore,
& Begaud, 1999)
1. The collection and scientific evaluation of
adverse drug reactions (ADR), under normal
conditions of use for regulatory purpose.
−Restricts the concept to regulatory
Pharmacovigilance compliance only and only medicinal
products.
2. Watchfulness in guarding against danger from
products or providing for safety of the product
−Expansive beyond just regulations and
frames the construct for use in academia
and the sciences
Introduction to Signal Detection 5

6. Definitions…
•The application of epidemiologic
techniques used to study the effects of
Pharmacoepidemiology drugs in populations
−First mentioned in the early 1980’s
(Abenhaim, Moore, & Begaud, 1999)
Introduction to Signal Detection 6

7. Exercise 1
1. Please segregate into your
Define the small group as assigned by the
index card in your chair
following terms: 2. Using collaborative discussion,
please reach consensus for the
definition of the following terms:
Signal  “Signal”
 “Signal Detection”
3. Nominate a single
spokesperson from the group to
Signal Detection share the definition with the
larger audience.
Introduction to Signal Detection 7

8. Definitions…
•Much debate on the definition (we will use the
following):
•Information that arises from one or multiple sources,
which suggests a new potentially causal association, or
Signal a new aspect of a known association, between an
intervention and an event or set of related events, either
adverse or beneficial, that is judged to be of sufficient
likelihood to justify verificatory actions. (CIOMS, 2010
p.14)
Introduction to Signal Detection 8

9. Definitions
•Much debate on the definition (we will use the
Signal following):
•The act of looking for and/or identifying signals
Detection using event data from any source . (CIOMS, 2010
p.116)
Introduction to Signal Detection 9

10. Simplified Safety Signal Lifecycle
Signal Signal
Detection Prioritization
Signal
Evaluation
CIOMS (2010, p. 9)
Introduction to Signal Detection 10

11. Detailed Signal Management Process
Signal
Detection
Signal
Prioritization
Signal
Evaluation
11
Introduction to Signal Detection

12. Signal Detection…
Introduction to Signal Detection 12

13. Individual Case Safety Reports (ICSR)
• The accumulation of ICSR can occur from multiple
places according to the ICH E2D Guideline (ICH, 2003)
Sources of ICSR Description of Sources
Unsolicited sources Spontaneous reporting
Solicited sources Any organized collection of data
(outcomes research, clinical trials,
registries, surveys, billing databases
etc.)
Contractual agreements Inter-company exchange of safety data
Regulatory authority Any ICSR originating from the
regulatory authority submitted to a
company
Introduction to Signal Detection 13

14. Signal Detection…
Traditional Pharmacovigilance
Methods
· Individual case review
· Aggregate analysis
· Periodic reports
• Traditionally, signals are detected through the
assessment of individual case safety reports
(ICSR) in an individual or cumulative manner
Introduction to Signal Detection 14

15. Signal Detection…
• Qualitative review of ICSR
– Critical method of detection for events where the
background incidence is uncommon or rare and should
not be replaced by quantitative methods (Egberts,
Meyboom, & van Puijenbroek, 2002; Hopstadius,
Norény, Bate, & Edwards, 2008)
– May be implemented as a list of designated medical
events (DME) within a company
• Often manifest in systems as custom Standardized MedDRA
Queries (SMQ)
– FDA has a list of “Interesting PTs”
– EMA follows serious events identified by CIOMS V
• Results in the identification of “index” or “striking”
cases that can be monitored
Introduction to Signal Detection 15

16. Signal Detection…
• Qualitative review of ICSR (cont.)
– Periodic review of case series
• PSUR or equivalent
– Review of the aggregate tables displayed in
these periodic reports can provide indications of
potential “striking” events
• Specific sections such as the “Overall Safety
Evaluation” provide clinical context around the
identification of such signals and become an
important part of signal detection
(Waller, 2009 p. 67)
Introduction to Signal Detection 16

17. Exercise 2
1. Please segregate into your
small group as assigned by the
index card in your chair
2. Using collaborative discussion
and the DSUR provided, please
DSUR AE determine the presence or
Summary Table absence of striking cases in the
table.
Review  Why did you or didn’t you
identify cases of interest?
3. Nominate a single
spokesperson from the group to
share the results with the larger
audience.
Introduction to Signal Detection 17

18. Signal Detection…
• Quantitative assessment of ICSR
– Absolute counts
• Number of reports of an adverse event (AE) or
adverse product reaction (APR)
985
No APR All completed the 1 year
1000 duration of study resulting in
Participants 985 person-years
in an IND
study Steven’s Johnson
Syndrome (SJS)
15
Completed the 6 month mark
of study resulting in 7.5
person-years
Introduction to Signal Detection 18

19. Signal Detection…
• With a total of 992.5 person-years for the denominator of the
study incidence rate, we can calculate the incidence rate of SJS
in the IND population as 15/992.5 which is 0.0151 cases/992.5
person years
– A typical denominator is million person years so we must convert or
IND incidence rate to million person years
• In terms of million person years this becomes 15.1 cases/million person
years
• Using epidemiologic sources, we note that the estimated
background incidence rate of SJS is 1-6 cases/million person
years
• The incidence rate in the study is suspiciously higher than the
expected background rate therefore SJS is a signal
Introduction to Signal Detection 19

20. Signal Detection…
• Quantitative assessment of ICSR (cont.)
– Proportions
• Number of specific AE reports divided by total number of
reports for a given product
– Assume an IND study of 100 patients for a new
Anticonvulsant
– At the end of the study there are 450 ICSR for the product
– Within this 450 are 25 reports of “GI bleeding”
– The proportion would be 25/450 or ~6%
• NOTE: True “expected” proportions are not really possible to
determine from spontaneous reporting data
– However, highly prevalent product/event
combinations for given case series may
be enough to elevate the combination
to a signal
Introduction to Signal Detection 20

21. Exercise 3
Determine the
following with the 1. Please segregate into your
small group as assigned by the
data distributed: index card in your chair
2. Using collaborative discussion
and the information provided,
Is there an please calculate the study
prevalence for “Peripheral
indication that neuropathy” and compare it to
“Peripheral the provided epidemiologic data
neuropathy” may  Has a signal been identified
in the data?
have a relationship 3. Nominate a single
to the spokesperson from the group to
consumption of share the results with the larger
audience.
“Alfenta”?
Introduction to Signal Detection 21

22. Disproportionality…
Data Mining Algorithms
· Disproportional reporting
ratios
• Once the ICSR database becomes large enough*, statistical techniques
(generally referred to as data mining) can be applied
– Usually on large datasets from regulatory agencies or public health
entities
– WHO: Vigibase
– FDA: AERS
– MCA: Sentinel (formally known as ADROIT)
– EMA: EudraVigilance
– Generally these techniques identify
disproportionate reporting ratios
*”Large Enough” is a function of product/event incidence in the population
Introduction to Signal Detection 22

23. “Large Enough”
• Implications on population size for sampling in signal
detection can be reduced to the following (CIOMS, 2010
p.31) :
Event Background Example Ease of Signal Detection
Incidence Incidence proving an Method
in Product of Event association
Takers (method)
Common Rare Phocomelia Easy ICSR or Periodic
due to Thalidomide (clinical observation) Review
Rare Rare Reye’s syndrome Less easy ICSR or Periodic
and Aspirin (clinical observation) Review
Common Common Cough Difficult Data Mining
and ACE inhibitors (large observational
trials/data)
Uncommon Common Breast carcinoma Very difficult Data Mining
to Rare and Hormone (large clinical trials)
Replacement
Therapies
Rare Common None known Virtually impossible Virtually impossible
Introduction to Signal Detection 23

24. Classical Versus Bayesian…
• The basis of disproportionality analysis is
either frequentist statistics or Bayesian
(Gajewski & Simon 2008)
– Classical (frequentist) statistics look at probabilities as long
term frequency with an assumption of repeatable
experiment or sampling methods and a “true” value for a
parameter
• TOTAL INFORMATION=Data from experimentation
– Bayesian statistics look at “true” probabilities as a function
incorporating prior beliefs or knowledge and is updated
• TOTAL INFORMATION=Historical Information + Data from
experimentation
Introduction to Signal Detection 24

25. Classical Versus Bayesian…
(Maggid, 2011)
Introduction to Signal Detection 25

26. Data Mining Algorithms…
• All measures calculated from a 2X2 Contingency Table
– Classical
• Proportional Rate Ratio (PRR)
• Reporting Odds Ratio (ROR)
• Relative Reporting Ratio (RRR)
– Bayesian
• Information Component
• MGPS/EBGM
Event of All other Events TOTAL
Interest
Product of Interest A B A+B
All other Products C D C+D
TOTAL A+C B+D A+B+C+D
Introduction to Signal Detection 26

27. Data Mining Algorithms…
• All measures of disproportionate reporting are
basically calculations of
OBSERVED/EXPECTED
• In PV, the EXPECTED data is also referred to
as the “background”
• What you include in the “background” is a point
of contention in the industry and no real rules
are present (Gogolak, 2003)
Introduction to Signal Detection 27

28. Data Mining Algorithms…
• Since the simple calculation is O/E, the
relationship between background and the
statistic of interest is inversely related:
– As the background increases the resulting
statistic decreases
• Large E results in small PRR
– As the background decreases the
resulting statistic increases
• Small E results in large PRR
Introduction to Signal Detection 28

29. Data Mining Algorithms
• In general, the frequentist or Bayesian methods will
perform similarly when there are five or more reports
of a particular product-event pair (CIOMS, 2010, pp.
59)
• Bayesian methods may provide a solution to “false
positive” indications in large datasets
• It is important to note that the literature does not
demonstrate consensus on cost/utility of various data
mining tools with respect to specificity and sensitivity
– (CIOMS, 2010, pp. 61)
• As such, company specific decisions
must be discussed as part of the
signal management process
Introduction to Signal Detection 29

30. Technical Solutions…
• Use of these databases requires that certain
assumptions be made
– Drugs used in the marketplace are used by a
representative sample of the greater population
• Any information derived from these databases should
be interpreted using the limitations of the data
contained therein (Edwards, 1999)
– Limited clinical quality of data
• USA allows reporting into the AERS system from anyone
(Health care provider {HCP} or not)
• EMEA only allows reporting by HCP thus typically more
complete clinical information
Introduction to Signal Detection 30

31. Technical Solutions…
• Three main products are available for large data
mining opportunities
– Qscan by Drug Logic
– Empirica Signal by Oracle
– agSignals by Aris Global
– dsExplorer by Cerner
• In the spirit of full-disclosure, BioPharm Systems is a
Gold Partner with Oracle and the next slides are taken
from the Empirica Signal product
– Please note that except for very specific functional differences,
these software systems are designed to accomplish similar
tasks. The degree to which one is better than the other is not
discussed in this presentation and the use of slides is not an
endorsement of one product over another
– As already stated, the decision to evaluate
signals and the method used is a company
specific decision
Introduction to Signal Detection 31

32. Technical Solutions…
• Empirica Signal can display disproportionality
results in a sector map fashion which allows for
visual assessment of signal strength.
Introduction to Signal Detection 32

33. Technical Solutions…
• In this example of
“Ziconotide”, we see
that sector 2 is
“Vertigo” with a
disproportionality
ratio of 2.052
– This is a good example though
of a well know substantiated
concern with the product as it
is known to disturb motor
function and balance
Introduction to Signal Detection 33

34. Technical Solutions…
• Sector 1 is “Tinnitus”
with a disproportionality
ratio of 4.194
Introduction to Signal Detection 34

35. Technical Solutions…
• The software solutions available provide a
wonderful way to quickly and comprehensively
analyze marketed data reported to regulatory
agencies.
• These data are subject to the issues surrounding
the collection of the information in them
– Underreporting of serious events
• Changes the number of expected events
• “Weber Effect”: The peak reporting for events in a drug on
market occurs within the first 2 years of approval (Hartnell, &
Wilson, 2004) during the initial 5 year marketing period
– Over reporting of events of
non-interest (expected non-serious)
Introduction to Signal Detection 35

36. Technical Solutions…
• False Causality attribution
– Signals ARE NOT CAUSAL INDICATIONS
– They are disproportionate reporting
indicators
• Mitigation of these limitations can occur if you
establish a signal prioritization method in your
company for dealing with the various signals
identified by manual or automated signal
detection
Introduction to Signal Detection 36

37. Signal Prioritization…
Triage of Outputs
· Interpret the signal in context of other
relevant sources, disease knowledge,
biologic plausibility, alternative
etiologies, etc.
Monitor
If signal is
indeterminate
NO
Impact Is signal
assessment and Need further NO
YES investigation refuted?
prioritization
Close out
YE
S
(if signal is refuted)
Introduction to Signal Detection 37

38. Signal Prioritization
• Prioritization of signals is still a very
controversial aspect of signal
management (Waller, 2010 p. 50)
• Two proposed methods are as follows:
– WHO – Triage (CIOMS, 2010 p. 88;
Lindquist, 2007)
– MHRA – Impact analysis (draft literature)
Introduction to Signal Detection 38

39. WHO - Triage
• Seriousness assessment
– Is the event serious or not?
• Unexpectedness
– Is the event expected or not?
• Disproportionality score
– Is the score high or not?
• Temporal displacement of score
– Has the disproportionality score increased over
time?
Introduction to Signal Detection 39

40. WHO - Triage
• Temporal occurrence
– Is the event occurring within the first few years of
launch
• Careful as Weber effect could contribute to
confounding here
• Multiple signaling countries
– Are more than one country seeing this
issue?
• Positive Rechallange
– Is there evidence of positive rechallange?
• Specialty list of terms of interest
– Is this a term of interest as identified by the
company
Introduction to Signal Detection 40

41. MHRA – Impact analysis…
• A calculated quantitative score based
on “Evidence” and “Public Health”
– Evidence Score
• Degree of disproportionality (PRR, IC etc)
• Strength of evidence
• Biologic Plausibility
– Public Health Score
• # of reported cases per year
• Expected health consequences
• Reporting rate in relationship to the
level of drug exposure
Introduction to Signal Detection 41

42. MHRA – Impact analysis
• Results in the following quantitative
categories:
– High
– Need to gather more information
– Low
– No action
• Additionally the MHRA is looking at a method to
incorporate the following in a prioritization
scheme:
– High profile product (media
attention)
– Risk perception by general
population
– Political obligations
Introduction to Signal Detection 42

43. Signal Prioritization
• Regardless of the method used, every
signal should undergo this type
prioritization
– CIOMS (2010 p. 22) suggest that this effort
results in the following outcomes which I have
labeled in the following manner:
• Confirmed signal
– Results in the movement of the signal to the evaluation
process
• Unconfirmed signal
– Results in the monitoring of this event over time and regularly
re-prioritizing it based on new information
• Refuted signal
– Results in the closing of the signal
» Disease progressions
» Known issues etc.
Introduction to Signal Detection 43

44. Signal Prioritization
• Regardless of the method used, care
should be taken in that this type of
assessment and analysis
should be an iterative process
when new information is
amassed
Introduction to Signal Detection 44

45. Exercise 4
1. Please segregate into your
small group as assigned by the
index card in your chair
2. Using collaborative discussion
and the information provided,
Signal please prioritize the various
Prioritization signals using one of the above
methods (WHO or MHRA)
3. Nominate a single
spokesperson from the group to
share the findings with the larger
audience.
Introduction to Signal Detection 45

46. Signal Evaluation…
Signal Evaluation
· Individual case review
· Aggregate analysis
· Periodic reports
· Non-interventional studies
· Non-clinical studies
· Class analysis
· Other relevant information
Introduction to Signal Detection 46

47. Key Components to Signal
Evaluation…
• Once a signal is prioritized as
“Confirmed” or its equivalent, further
data must be gathered to further the
evaluation of the signal (Waller, 2010 p.
50, 51)
• Causality
– Does the provided evidence support a causal
relationship between the product and event?
• ICSR Causal Relationships
– Probable, possible, unlikely, unrelated, unassessable
Causality is more than this
Introduction to Signal Detection 47

48. Causality…
• Bradford-Hill (Shakir & Layton, 2002)
proposed the following categories should be
used to assess causal relationships found in
data
– Strength
• The stronger an association, the less likely it is
explained by other factors
– Consistency
• Multiple sources demonstrate similar
associations
– Temporality
• Is the event consistently treatment
emergent?
– Positive re or de challenge evidence
Introduction to Signal Detection 48

49. Causality…
• Bradford-Hill (cont.)
– Biologic Gradient
• Evidence of dose or duration related risk
– Dose-dependency or cumulative exposure over time
– Specificity
• Single product exposure results in event
– Presence of this leaves little doubt to causality
– A few adverse drug reactions in an of themselves are
syndromes
• Absence of this does not reflect a non-causal
relationship
– For example nicotine exposure and
lung cancers
Introduction to Signal Detection 49

50. Causality…
• Bradford-Hill (cont.)
– Plausibility
• Does the existing literature support this association?
– Coherence
• Is the association compatible with the existing literature and
knowledge?
– Experimental Evidence
• Is there data that demonstrates the event can be altered or
eliminated by some experimental regimen?
– Converging evidence between post marketed and clinical surveillance
– Analogy
• Are there existing alternative
explanations for the association
in the literature?
• Absence of these strengthens the causal likelihood
• Absence of any or all of these four criteria does not
indicate the absence of a causal relationship since
out data may be describing a newly
seen observation not part of the
literature used in these assessments
Introduction to Signal Detection 50

51. Causality…
• In general, the more of the criteria satisfied,
the stronger the causal relationship. The
assessment of how many and which criteria
are more important than others, is no simple
formula and judgment is required (Waller,
2010 p. 29)
• There are issues with the PV databases and
literature used in signal evaluation and one
must keep these in mind when applying the
Bradford-Hill criteria (Shakir &
Layton, 2002)
Introduction to Signal Detection 51

52. Key Components to Signal
Evaluation…
• In addition to assessments of causality, the
concept of Frequency is important in signal
evaluation (Waller, 2010 p. 51)
– The question of frequency can be categorized
into the following measures of prevalence
• Very Common
– More than 1:10
• Common
– 1:10 to 1:100
• Uncommon
– 1:100 to 1:1000
• Rare
– 1:1000 to 1:10,000
• Very Rare
– Less than 1:10,000
Introduction to Signal Detection 52

53. Key Components to Signal
Evaluation…
• In addition to assessments of causality and
frequency, the concept of Clinical
implications is important in signal evaluation
(Waller, 2010 p. 51)
– Clinical Implications can be rephrased to impact
on patient health
• Is the event life-threatening?
• Does its presence lead to a congenital anomaly?
• Does its presence lead to death?
• Does its presence lead to long term
disability or long term hospitalization?
Introduction to Signal Detection 53

54. Key Components to Signal
Evaluation…
• In addition to assessments of causality,
frequency and clinical implications, the
concept of Preventability is important in
signal evaluation (Waller, 2010 p. 51)
– Preventability
• If an intervention could be applied at this stage, would
the event of interest or its outcomes be prevented?
Introduction to Signal Detection 54

55. Key Components to Signal
Evaluation…
• To summarize, every confirmed signal is
evaluated in terms of the following:
– Causality
– Frequency
– Clinical implications
– Preventability
• Based on the suspected signal’s evaluation
outcome, you may choose to elevate it to the
status of a potential or identified
risk (CIOMS, 2010 p. 93, 94)
Introduction to Signal Detection 55

56. Key Components to Signal
Evaluation
• This signal evaluation process is intimately
tied to risk identification and may result into
the feeding of the signal into the risk
mitigation/management planning processes
at your company.
• The information collected in the signal
evaluation phase will feed the safety
specification section of the risk management
plan.
Introduction to Signal Detection 56

57. Exercise 5
1. Please segregate into your
small group as assigned by the
index card in your chair
2. Using the provided clinical
information, is there evidence
Signal Evaluation that “Ascites” should be
elevated to a risk?
3. Nominate a single
spokesperson from the group to
share the findings with the larger
audience.
Introduction to Signal Detection 57

58. Summary
1. Create a procedure to formalize signal
management in your organization
• Ensure your process includes signal detection,
prioritization and evaluation
2. Signal detection efforts should include both
qualitative and quantitative methods
Top Five 3. Signal prioritization should describe the objective
methods used to categorize the signals
Take- discovered through detection efforts
Aways 4. Signal evaluation should be used to document
and formally evaluate only those critical signals
(from prioritization methods)
5. Information collected from your formal evaluation
strategies can be used to seamlessly parse into
your risk management process
Introduction to Signal Detection 58

59. References
• Council for International Organizations of Medical Sciences (CIOMS). (2010). Practical Aspects of Signal
Detection in Pharmacovigilance. Report of CIOMS Working Group VIII, Geneva .
• Egberts, A.C.G., Meyboom, R.H.B., and van Puijenbroek, E.P. (2002). Use of Measures of Disproportionality in
Pharmacovigilance: Three Dutch Examples Drug Safety 25(6): 453-458
• Gajewski, B.J. and Simon, S.D. (2008). A One-Hour Training Seminar on Bayesian Statistics for Nursing
Graduate Students. The American Statistician, 62 (3)
• Hartnell, N.R. and Wilson, J.P.. (2004). Replication of the Weber Effect Using Postmarketing Adverse Event
Reports Voluntarily Submitted to the United States Food and Drug Administration. Pharmacotherapy. 24:743-
749
• Hopstadius, J., Norény, G.N., Bate, A., and Edwards, R.. (2008). Impact of Stratification on Adverse Drug
Reaction Surveillance. Drug Safety. 31(11)
• International Conference on Harmonisation (ICH). (2003). ICH Harmonised Tripartite Guideline. Post-approval
safety data management: Definitions and standards for expedited reporting E2D. Retrieved on July 26. 2011
from
http://www.ich.org/fileadmin/Public_Web_Site/ICH_Products/Guidelines/Efficacy/E2D/Step4/E2D_Guideline.pdf
• Levitan, B., Yee, C. L., Russo, L., Bayney, R., Thomas, A. P., and Klincewicz, S. L. (2008). A Model for
Decision Support in Signal Triage. Drug Safety, 31 (9)
• Lindquist, M. (2007). Use of triage strategies in the WHO signal-detection process. Drug Safety, 30:635-7
• Maggid. (2011). Retrieved from http://actuary-info.blogspot.com/2011/05/homo-actuarius-bayesianes.html on
August 31, 2011
• Shakir, A.W., and Layton, D.. (2002). Causal Association in Pharmacovigilance and Pharmacoepidemiology:
Thoughts on the Application of the Austin Bradford-Hill Criteria. Drug Safety. 25 (6): 467-471
• Strom, B. L., and Kimmel, S. E. (2006). Textbook of Pharmacoepidemiology. Wiley, West Sussex, UK
• Waller, P. (2010). An Introduction to Pharmacovigilance. Wiley-Blackwell. Oxford, UK
Introduction to Signal Detection 59

60. Contact Information
Rodney has over 15 years experience in clinical research including
laboratory experimentation, clinical data management, clinical trial
design, dictionary coding and pharmacovigilance.
Rodney has worked for BioPharm Systems for eleven years now
serving in a variety of roles all related to the technical and/or clinical
implementations of software systems used in the clinical trial
process.
Prior to coming to BioPharm Systems Rodney worked at
pharmaceutical and technology companies in the Dictionary Coding,
Statistical Programming and Data Management areas.
In addition to his current work at BioPharm Systems, Rodney holds
an Associate faculty position at Walden University teaching Public
Health Informatics and other health information systems courses.
Rodney holds a Bachelor of Science in Genetic Engineering, a
Masters of Public Health in International Epidemiology and a Ph.D. in
Epidemiology focusing on Social Epidemiology
Introduction to Signal Detection 60