SlideShare una empresa de Scribd logo
1 de 19
Descargar para leer sin conexión
www.rti.orgRTI International is a registered trademark and a trade name of Research Triangle Institute.
What is the Optimal Mode?
A Comparison of CATI, IVR, SMS
Pew Research Center Workshop on Public
Opinion in Africa, 30 November 2017
Charles Q. Lau, PhD, MS
1
What is the Optimal Mode?
2
SMS & IVR & Web & CATI: Rapid Data for
Public Health Surveillance (10 countries)
SMS: Tracking Vocational
School Graduates
SMS: Panel Surveys of Youth
SMS & Web: Panel for
Job Accelerator Trainees
SMS: Feasibility of SMS Surveys
IVR: Best Practices in IVR
Introductions
CATI: Technology Adoption
CATI & SMS: Health
Worker Panel Survey
Mobile Survey Modes
3
CATI
IVR SMS
Other Modes: Web and Chatbots
Errors in Mobile Phone Surveys
4
Coverage
Error
CATI, IVR, SMS all
suffer from same
coverage error
(exclude people
without mobiles)
Errors in Mobile Phone Surveys
5
Coverage
Error
Non-
Response
Error
CATI, IVR, SMS all
suffer from same
coverage error
(exclude people
without mobiles)
CATI, IVR, SMS share
common non-response
errors …
-No in-person
interviewer
-Short intros
-Poor network
connection
-Phones turned off
Errors in Mobile Phone Surveys
6
Coverage
Error
Non-
Response
Error
CATI, IVR, SMS all
suffer from same
coverage error
(exclude people
without mobiles)
CATI, IVR, SMS share
common non-response
errors …
... but CATI, IVR, SMS
differ in non-response
errors
-No in-person
interviewer
-Short intros
-Poor network
connection
-Phones turned off
-SMS, IVR lack
interviewer
-SMS, IVR require
tech familiarity
-SMS requires
literacy
-Concern about
cost for SMS
-But: Do SMS at
leisure
Errors in Mobile Phone Surveys
7
Coverage
Error
Non-
Response
Error
Measurement
Error
CATI, IVR, SMS all
suffer from same
coverage error
(exclude people
without mobiles)
CATI, IVR, SMS share
common non-response
errors …
... but CATI, IVR, SMS
differ in non-response
errors
CATI allows for
probing and more
complicated
questions …
… but IVR and
SMS may be better
for sensitive topics
Previous Research
Key Insights
 Response rates are low, especially for IVR and SMS
 Cross-sectional surveys with a single mode (SMS, IVR,
CATI) over-represent young, men, urban, educated
 Panel approach with face-to-face recruitment is promising
 Mixed mode approaches can improve quality
 Weighting might help reduce bias
Knowledge Gaps
 Limited apples-to-apples comparisons
 How biased are survey estimates? Can weighting help?
 How do modes differ with regard to cost and time?
8
Research Questions
9
1. How do response rates
differ among CATI, IVR,
SMS, and FTF surveys?
2. How representative are
respondents from CATI,
IVR, and SMS surveys?
3. Can IVR and SMS
provide an unbiased
estimate of voting
behavior? If there is bias,
can weights reduce bias?
4. How does the cost and
data collection time differ
across survey modes?
Method
 Mobile Phone Surveys
– Modes: CATI, IVR, SMS
– All used RDD sampling
– IVR and SMS conducted in 2017; CATI in 2016
– IVR, SMS used $1 incentive. CATI had experiment ($0 vs. $1)
– IVR and SMS asked same 12 questions; CATI asked 85 questions
 FTF: 2013 Nigeria Demographic and Health Survey (paper-and-
pencil, area probability sample with HH listing, HH quex used)
 Target population for all surveys is general population age 18-64
(though frames differ)
 Approach: Compare estimates from CATI, IVR, SMS to FTF (and to
each other)
10
RQ1. Response Rates
[VALUE]%
[VALUE]%
[VALUE]%
CATI IVR SMS
Response Rates by Mode
11
Response
Rate for FTF
is 99%
RQ2. Representativeness
12
40%
54%
62%
65%
25%
27% 27%
23%
18%
12%
8% 9%
17%
7%
2% 3%
FTF CATI IVR SMS
Age by Mode
18-29 30-39 40-49 50-64
53%
33%
28%
36%
FTF CATI IVR SMS
Female by Mode
RQ2. Representativeness
13
39%
5%
8%
3%
26%
9%
11%
6%
22%
46%
32%
41%
13%
40%
50% 51%
FTF CATI IVR SMS
Education by Mode
No school Primary
Secondary Post-secondary
RQ. Representativeness
14
71%
46%
34%
FTF IVR SMS
Married by Mode
57%
24% 21%
FTF IVR SMS
Village by Mode
70%
79%
71% 69%
FTF CATI IVR SMS
Radio by Mode
81% 85%
IVR SMS
Read Well by Mode
RQ3. Bias
15
31%
68%
59%
Estimates of Voting in 2015 Presidential
Election
SMS IVR True
Preliminary
analysis:
Weighting
reduces some
bias, but does not
eliminate bias.
Weighting by
some factors
(e.g., age)
increases bias.
RQ4. Cost and Time
Data Collection Cost
- SMS is 52% the cost of IVR.
- In this study, IVR is 43% and
SMS is 22% the cost of CATI.
However, IVR and SMS had 12
questions, CATI had 85.
- CATI is cheaper than IVR and
SMS for long (85 qx) surveys
-IVR and SMS is cheaper than
CATI for short (12 qx) surveys
16
Data Collection Time
Data collection days
 CATI: 43 days
 IVR: 42 days
 SMS: 16 days
Completes per day
 CATI: 88
 IVR: 43
 SMS: 172
Takeaways and Discussion
1. Don’t be fooled by incremental improvements in response rates
2. CATI, IVR, SMS can’t match representativeness of FTF surveys
3. Voting estimates are biased; weighting can’t save the day
4. IVR respondents are slightly more representative than SMS (age,
education, village). But are small differences worth the cost?
5. CATI has edge (age, education) over IVR/SMS.
6. For short, simple surveys, IVR & SMS may be best modes if you’re
willing to sacrifice a bit of representativeness
7. For complex or larger surveys (questions, N), CATI is your best bet.
Unresolved issue: Measurement differences between modes
Limitations: Nigeria only; cross-sectional surveys; may be topic specific;
can’t separate coverage, non-response, measurement; house effects
17
Tool: Surveda for Mixed Mode Surveys
18
As part of Bloomberg Philanthropies Data for Health project, RTI and InSTEDD
have built an open-source survey tool for mixed mode (SMS, IVR, Web)
surveys: https://surveda.instedd.org
More Information
Charles Q. Lau, PhD, MS
Survey Methodologist
+1.919.541.8798
clau@rti.org
19

Más contenido relacionado

Similar a 11. RTI International Presentation

Case study of use of IVR systems to empower the visually impaired
Case study of use of IVR systems to empower the visually impairedCase study of use of IVR systems to empower the visually impaired
Case study of use of IVR systems to empower the visually impairedAshish Tandon
 
MCJ 5532, Research Methods in Criminal Justice Administra.docx
MCJ 5532, Research Methods in Criminal Justice Administra.docxMCJ 5532, Research Methods in Criminal Justice Administra.docx
MCJ 5532, Research Methods in Criminal Justice Administra.docxAASTHA76
 
The Sensis Social Media Analysis
The Sensis Social Media AnalysisThe Sensis Social Media Analysis
The Sensis Social Media AnalysisLesly Lockwood
 
A large scale study of daily information needs captured in situ
A large scale study of daily information needs captured in situA large scale study of daily information needs captured in situ
A large scale study of daily information needs captured in situWookjae Maeng
 
Answers to ESOMAR 28 Questions (IMAS Panel 2019)
Answers to ESOMAR 28 Questions (IMAS Panel 2019)Answers to ESOMAR 28 Questions (IMAS Panel 2019)
Answers to ESOMAR 28 Questions (IMAS Panel 2019)IMAS International
 
Evaluating Imagine!’s SmartHome
Evaluating Imagine!’s SmartHomeEvaluating Imagine!’s SmartHome
Evaluating Imagine!’s SmartHomeFred Hobbs
 
A probability-based web panel for the UK: What could it look like?
A probability-based web panel for the UK: What could it look like?A probability-based web panel for the UK: What could it look like?
A probability-based web panel for the UK: What could it look like?Gerry Nicolaas
 
Afroeye Research - ESOMAR 28 QUESTIONS
Afroeye Research - ESOMAR 28 QUESTIONSAfroeye Research - ESOMAR 28 QUESTIONS
Afroeye Research - ESOMAR 28 QUESTIONSAfroeyeResearch
 
SURVEY KARO.ppt
SURVEY KARO.pptSURVEY KARO.ppt
SURVEY KARO.pptRavi Kumar
 
The Armed Services Vocational Aptitude Battery (ASVAB) Is...
The Armed Services Vocational Aptitude Battery (ASVAB) Is...The Armed Services Vocational Aptitude Battery (ASVAB) Is...
The Armed Services Vocational Aptitude Battery (ASVAB) Is...Tina Jordan
 
ESOMAR Telephone and Internet Coverage around the World 2016
ESOMAR Telephone and Internet Coverage around the World 2016ESOMAR Telephone and Internet Coverage around the World 2016
ESOMAR Telephone and Internet Coverage around the World 2016T.S. Lim
 
Teledentistry; Next step into digitalization
Teledentistry; Next step into digitalizationTeledentistry; Next step into digitalization
Teledentistry; Next step into digitalizationKathan Mehta
 
Teledentistry; Next step into digitalization
Teledentistry; Next step into digitalizationTeledentistry; Next step into digitalization
Teledentistry; Next step into digitalizationKathan Mehta
 
Workshop on public relations measurement
Workshop on public relations measurementWorkshop on public relations measurement
Workshop on public relations measurementDavid Geddes
 
Pan All Ia Learning Clinic
Pan All Ia Learning ClinicPan All Ia Learning Clinic
Pan All Ia Learning Clinicpanall2009
 

Similar a 11. RTI International Presentation (20)

Case study of use of IVR systems to empower the visually impaired
Case study of use of IVR systems to empower the visually impairedCase study of use of IVR systems to empower the visually impaired
Case study of use of IVR systems to empower the visually impaired
 
MCJ 5532, Research Methods in Criminal Justice Administra.docx
MCJ 5532, Research Methods in Criminal Justice Administra.docxMCJ 5532, Research Methods in Criminal Justice Administra.docx
MCJ 5532, Research Methods in Criminal Justice Administra.docx
 
Internet Pilot Paper2
Internet Pilot Paper2Internet Pilot Paper2
Internet Pilot Paper2
 
The Sensis Social Media Analysis
The Sensis Social Media AnalysisThe Sensis Social Media Analysis
The Sensis Social Media Analysis
 
A large scale study of daily information needs captured in situ
A large scale study of daily information needs captured in situA large scale study of daily information needs captured in situ
A large scale study of daily information needs captured in situ
 
Answers to ESOMAR 28 Questions (IMAS Panel 2019)
Answers to ESOMAR 28 Questions (IMAS Panel 2019)Answers to ESOMAR 28 Questions (IMAS Panel 2019)
Answers to ESOMAR 28 Questions (IMAS Panel 2019)
 
Evaluating Imagine!’s SmartHome
Evaluating Imagine!’s SmartHomeEvaluating Imagine!’s SmartHome
Evaluating Imagine!’s SmartHome
 
A probability-based web panel for the UK: What could it look like?
A probability-based web panel for the UK: What could it look like?A probability-based web panel for the UK: What could it look like?
A probability-based web panel for the UK: What could it look like?
 
Afroeye Research - ESOMAR 28 QUESTIONS
Afroeye Research - ESOMAR 28 QUESTIONSAfroeye Research - ESOMAR 28 QUESTIONS
Afroeye Research - ESOMAR 28 QUESTIONS
 
SURVEY KARO.ppt
SURVEY KARO.pptSURVEY KARO.ppt
SURVEY KARO.ppt
 
The Armed Services Vocational Aptitude Battery (ASVAB) Is...
The Armed Services Vocational Aptitude Battery (ASVAB) Is...The Armed Services Vocational Aptitude Battery (ASVAB) Is...
The Armed Services Vocational Aptitude Battery (ASVAB) Is...
 
ESOMAR Telephone and Internet Coverage around the World 2016
ESOMAR Telephone and Internet Coverage around the World 2016ESOMAR Telephone and Internet Coverage around the World 2016
ESOMAR Telephone and Internet Coverage around the World 2016
 
Paper 3 (iftikhar alam)
Paper 3 (iftikhar alam)Paper 3 (iftikhar alam)
Paper 3 (iftikhar alam)
 
An Exploratory Case Study
An Exploratory Case StudyAn Exploratory Case Study
An Exploratory Case Study
 
Teledentistry; Next step into digitalization
Teledentistry; Next step into digitalizationTeledentistry; Next step into digitalization
Teledentistry; Next step into digitalization
 
Teledentistry; Next step into digitalization
Teledentistry; Next step into digitalizationTeledentistry; Next step into digitalization
Teledentistry; Next step into digitalization
 
Workshop on public relations measurement
Workshop on public relations measurementWorkshop on public relations measurement
Workshop on public relations measurement
 
Pan All Ia Learning Clinic
Pan All Ia Learning ClinicPan All Ia Learning Clinic
Pan All Ia Learning Clinic
 
survey techniques
survey techniquessurvey techniques
survey techniques
 
Jack Trout And Ries
Jack Trout And RiesJack Trout And Ries
Jack Trout And Ries
 

Más de Pew Research Center

Virtual Event: Jewish Americans in 2020
Virtual Event: Jewish Americans in 2020Virtual Event: Jewish Americans in 2020
Virtual Event: Jewish Americans in 2020Pew Research Center
 
Between the Data: The Unseen Stories of Asian Americans
Between the Data: The Unseen Stories of Asian AmericansBetween the Data: The Unseen Stories of Asian Americans
Between the Data: The Unseen Stories of Asian AmericansPew Research Center
 
How Do OECD Forum Attendees Compare with Citizens Around the World on Views A...
How Do OECD Forum Attendees Compare with Citizens Around the World on Views A...How Do OECD Forum Attendees Compare with Citizens Around the World on Views A...
How Do OECD Forum Attendees Compare with Citizens Around the World on Views A...Pew Research Center
 
Once in a decade: Covering the 2020 census
Once in a decade: Covering the 2020 censusOnce in a decade: Covering the 2020 census
Once in a decade: Covering the 2020 censusPew Research Center
 
How Do OECD Forum Attendees Compare With General Publics Around the World on ...
How Do OECD Forum Attendees Compare With General Publics Around the World on ...How Do OECD Forum Attendees Compare With General Publics Around the World on ...
How Do OECD Forum Attendees Compare With General Publics Around the World on ...Pew Research Center
 
How has populism disrupted the left right divide in western europe
How has populism disrupted the left right divide in western europeHow has populism disrupted the left right divide in western europe
How has populism disrupted the left right divide in western europePew Research Center
 
The Roots of Modern Nationalism and the Populist Backlash
The Roots of Modern Nationalism and the Populist BacklashThe Roots of Modern Nationalism and the Populist Backlash
The Roots of Modern Nationalism and the Populist BacklashPew Research Center
 
OECD Forum 2018 Thought Leader Survey
OECD Forum 2018 Thought Leader SurveyOECD Forum 2018 Thought Leader Survey
OECD Forum 2018 Thought Leader SurveyPew Research Center
 
Religious demography maps and data from Pew Research Center
Religious demography maps and data from Pew Research CenterReligious demography maps and data from Pew Research Center
Religious demography maps and data from Pew Research CenterPew Research Center
 
Thought Leader Survey: Issues Impacting the Transatlantic Relationship, 2018
Thought Leader Survey: Issues Impacting the Transatlantic Relationship, 2018Thought Leader Survey: Issues Impacting the Transatlantic Relationship, 2018
Thought Leader Survey: Issues Impacting the Transatlantic Relationship, 2018Pew Research Center
 
Thought Leader Survey: Issues Impacting the Transatlantic Relationship
Thought Leader Survey: Issues Impacting the Transatlantic RelationshipThought Leader Survey: Issues Impacting the Transatlantic Relationship
Thought Leader Survey: Issues Impacting the Transatlantic RelationshipPew Research Center
 
Origins and Destinations of Foreign Students in the United States
Origins and Destinations of Foreign Students in the United StatesOrigins and Destinations of Foreign Students in the United States
Origins and Destinations of Foreign Students in the United StatesPew Research Center
 
8. Greenberg Quinlan Rosner Presentation
8. Greenberg Quinlan Rosner Presentation8. Greenberg Quinlan Rosner Presentation
8. Greenberg Quinlan Rosner PresentationPew Research Center
 
3. ORB International Presentation
3. ORB International Presentation3. ORB International Presentation
3. ORB International PresentationPew Research Center
 
2. Research Plus Africa Presentation
2. Research Plus Africa Presentation2. Research Plus Africa Presentation
2. Research Plus Africa PresentationPew Research Center
 
Global Attitudes Toward China and the U.S.
Global Attitudes Toward China and the U.S.Global Attitudes Toward China and the U.S.
Global Attitudes Toward China and the U.S.Pew Research Center
 

Más de Pew Research Center (20)

Virtual Event: Jewish Americans in 2020
Virtual Event: Jewish Americans in 2020Virtual Event: Jewish Americans in 2020
Virtual Event: Jewish Americans in 2020
 
Between the Data: The Unseen Stories of Asian Americans
Between the Data: The Unseen Stories of Asian AmericansBetween the Data: The Unseen Stories of Asian Americans
Between the Data: The Unseen Stories of Asian Americans
 
How Do OECD Forum Attendees Compare with Citizens Around the World on Views A...
How Do OECD Forum Attendees Compare with Citizens Around the World on Views A...How Do OECD Forum Attendees Compare with Citizens Around the World on Views A...
How Do OECD Forum Attendees Compare with Citizens Around the World on Views A...
 
Once in a decade: Covering the 2020 census
Once in a decade: Covering the 2020 censusOnce in a decade: Covering the 2020 census
Once in a decade: Covering the 2020 census
 
How Do OECD Forum Attendees Compare With General Publics Around the World on ...
How Do OECD Forum Attendees Compare With General Publics Around the World on ...How Do OECD Forum Attendees Compare With General Publics Around the World on ...
How Do OECD Forum Attendees Compare With General Publics Around the World on ...
 
Trust in the Digital Era
Trust in the Digital EraTrust in the Digital Era
Trust in the Digital Era
 
How has populism disrupted the left right divide in western europe
How has populism disrupted the left right divide in western europeHow has populism disrupted the left right divide in western europe
How has populism disrupted the left right divide in western europe
 
The Roots of Modern Nationalism and the Populist Backlash
The Roots of Modern Nationalism and the Populist BacklashThe Roots of Modern Nationalism and the Populist Backlash
The Roots of Modern Nationalism and the Populist Backlash
 
OECD Forum 2018 Thought Leader Survey
OECD Forum 2018 Thought Leader SurveyOECD Forum 2018 Thought Leader Survey
OECD Forum 2018 Thought Leader Survey
 
Religious demography maps and data from Pew Research Center
Religious demography maps and data from Pew Research CenterReligious demography maps and data from Pew Research Center
Religious demography maps and data from Pew Research Center
 
Thought Leader Survey: Issues Impacting the Transatlantic Relationship, 2018
Thought Leader Survey: Issues Impacting the Transatlantic Relationship, 2018Thought Leader Survey: Issues Impacting the Transatlantic Relationship, 2018
Thought Leader Survey: Issues Impacting the Transatlantic Relationship, 2018
 
Thought Leader Survey: Issues Impacting the Transatlantic Relationship
Thought Leader Survey: Issues Impacting the Transatlantic RelationshipThought Leader Survey: Issues Impacting the Transatlantic Relationship
Thought Leader Survey: Issues Impacting the Transatlantic Relationship
 
Origins and Destinations of Foreign Students in the United States
Origins and Destinations of Foreign Students in the United StatesOrigins and Destinations of Foreign Students in the United States
Origins and Destinations of Foreign Students in the United States
 
7. McCauley Presentation
7. McCauley Presentation7. McCauley Presentation
7. McCauley Presentation
 
8. Greenberg Quinlan Rosner Presentation
8. Greenberg Quinlan Rosner Presentation8. Greenberg Quinlan Rosner Presentation
8. Greenberg Quinlan Rosner Presentation
 
6. Mathematica Presentation
6. Mathematica Presentation6. Mathematica Presentation
6. Mathematica Presentation
 
4. BioStat Global Presentation
4. BioStat Global Presentation4. BioStat Global Presentation
4. BioStat Global Presentation
 
3. ORB International Presentation
3. ORB International Presentation3. ORB International Presentation
3. ORB International Presentation
 
2. Research Plus Africa Presentation
2. Research Plus Africa Presentation2. Research Plus Africa Presentation
2. Research Plus Africa Presentation
 
Global Attitudes Toward China and the U.S.
Global Attitudes Toward China and the U.S.Global Attitudes Toward China and the U.S.
Global Attitudes Toward China and the U.S.
 

Último

Optimal Decision Making - Cost Reduction in Logistics
Optimal Decision Making - Cost Reduction in LogisticsOptimal Decision Making - Cost Reduction in Logistics
Optimal Decision Making - Cost Reduction in LogisticsThinkInnovation
 
Mapping the pubmed data under different suptopics using NLP.pptx
Mapping the pubmed data under different suptopics using NLP.pptxMapping the pubmed data under different suptopics using NLP.pptx
Mapping the pubmed data under different suptopics using NLP.pptxVenkatasubramani13
 
Elements of language learning - an analysis of how different elements of lang...
Elements of language learning - an analysis of how different elements of lang...Elements of language learning - an analysis of how different elements of lang...
Elements of language learning - an analysis of how different elements of lang...PrithaVashisht1
 
Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024
Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024
Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024Guido X Jansen
 
TINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptx
TINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptxTINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptx
TINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptxDwiAyuSitiHartinah
 
Rock Songs common codes and conventions.pptx
Rock Songs common codes and conventions.pptxRock Songs common codes and conventions.pptx
Rock Songs common codes and conventions.pptxFinatron037
 
CI, CD -Tools to integrate without manual intervention
CI, CD -Tools to integrate without manual interventionCI, CD -Tools to integrate without manual intervention
CI, CD -Tools to integrate without manual interventionajayrajaganeshkayala
 
CCS336-Cloud-Services-Management-Lecture-Notes-1.pptx
CCS336-Cloud-Services-Management-Lecture-Notes-1.pptxCCS336-Cloud-Services-Management-Lecture-Notes-1.pptx
CCS336-Cloud-Services-Management-Lecture-Notes-1.pptxdhiyaneswaranv1
 
Virtuosoft SmartSync Product Introduction
Virtuosoft SmartSync Product IntroductionVirtuosoft SmartSync Product Introduction
Virtuosoft SmartSync Product Introductionsanjaymuralee1
 
The Universal GTM - how we design GTM and dataLayer
The Universal GTM - how we design GTM and dataLayerThe Universal GTM - how we design GTM and dataLayer
The Universal GTM - how we design GTM and dataLayerPavel Šabatka
 
5 Ds to Define Data Archiving Best Practices
5 Ds to Define Data Archiving Best Practices5 Ds to Define Data Archiving Best Practices
5 Ds to Define Data Archiving Best PracticesDataArchiva
 
ChistaDATA Real-Time DATA Analytics Infrastructure
ChistaDATA Real-Time DATA Analytics InfrastructureChistaDATA Real-Time DATA Analytics Infrastructure
ChistaDATA Real-Time DATA Analytics Infrastructuresonikadigital1
 
Cash Is Still King: ATM market research '2023
Cash Is Still King: ATM market research '2023Cash Is Still King: ATM market research '2023
Cash Is Still King: ATM market research '2023Vladislav Solodkiy
 
Master's Thesis - Data Science - Presentation
Master's Thesis - Data Science - PresentationMaster's Thesis - Data Science - Presentation
Master's Thesis - Data Science - PresentationGiorgio Carbone
 
How is Real-Time Analytics Different from Traditional OLAP?
How is Real-Time Analytics Different from Traditional OLAP?How is Real-Time Analytics Different from Traditional OLAP?
How is Real-Time Analytics Different from Traditional OLAP?sonikadigital1
 
Strategic CX: A Deep Dive into Voice of the Customer Insights for Clarity
Strategic CX: A Deep Dive into Voice of the Customer Insights for ClarityStrategic CX: A Deep Dive into Voice of the Customer Insights for Clarity
Strategic CX: A Deep Dive into Voice of the Customer Insights for ClarityAggregage
 

Último (16)

Optimal Decision Making - Cost Reduction in Logistics
Optimal Decision Making - Cost Reduction in LogisticsOptimal Decision Making - Cost Reduction in Logistics
Optimal Decision Making - Cost Reduction in Logistics
 
Mapping the pubmed data under different suptopics using NLP.pptx
Mapping the pubmed data under different suptopics using NLP.pptxMapping the pubmed data under different suptopics using NLP.pptx
Mapping the pubmed data under different suptopics using NLP.pptx
 
Elements of language learning - an analysis of how different elements of lang...
Elements of language learning - an analysis of how different elements of lang...Elements of language learning - an analysis of how different elements of lang...
Elements of language learning - an analysis of how different elements of lang...
 
Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024
Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024
Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024
 
TINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptx
TINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptxTINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptx
TINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptx
 
Rock Songs common codes and conventions.pptx
Rock Songs common codes and conventions.pptxRock Songs common codes and conventions.pptx
Rock Songs common codes and conventions.pptx
 
CI, CD -Tools to integrate without manual intervention
CI, CD -Tools to integrate without manual interventionCI, CD -Tools to integrate without manual intervention
CI, CD -Tools to integrate without manual intervention
 
CCS336-Cloud-Services-Management-Lecture-Notes-1.pptx
CCS336-Cloud-Services-Management-Lecture-Notes-1.pptxCCS336-Cloud-Services-Management-Lecture-Notes-1.pptx
CCS336-Cloud-Services-Management-Lecture-Notes-1.pptx
 
Virtuosoft SmartSync Product Introduction
Virtuosoft SmartSync Product IntroductionVirtuosoft SmartSync Product Introduction
Virtuosoft SmartSync Product Introduction
 
The Universal GTM - how we design GTM and dataLayer
The Universal GTM - how we design GTM and dataLayerThe Universal GTM - how we design GTM and dataLayer
The Universal GTM - how we design GTM and dataLayer
 
5 Ds to Define Data Archiving Best Practices
5 Ds to Define Data Archiving Best Practices5 Ds to Define Data Archiving Best Practices
5 Ds to Define Data Archiving Best Practices
 
ChistaDATA Real-Time DATA Analytics Infrastructure
ChistaDATA Real-Time DATA Analytics InfrastructureChistaDATA Real-Time DATA Analytics Infrastructure
ChistaDATA Real-Time DATA Analytics Infrastructure
 
Cash Is Still King: ATM market research '2023
Cash Is Still King: ATM market research '2023Cash Is Still King: ATM market research '2023
Cash Is Still King: ATM market research '2023
 
Master's Thesis - Data Science - Presentation
Master's Thesis - Data Science - PresentationMaster's Thesis - Data Science - Presentation
Master's Thesis - Data Science - Presentation
 
How is Real-Time Analytics Different from Traditional OLAP?
How is Real-Time Analytics Different from Traditional OLAP?How is Real-Time Analytics Different from Traditional OLAP?
How is Real-Time Analytics Different from Traditional OLAP?
 
Strategic CX: A Deep Dive into Voice of the Customer Insights for Clarity
Strategic CX: A Deep Dive into Voice of the Customer Insights for ClarityStrategic CX: A Deep Dive into Voice of the Customer Insights for Clarity
Strategic CX: A Deep Dive into Voice of the Customer Insights for Clarity
 

11. RTI International Presentation

  • 1. www.rti.orgRTI International is a registered trademark and a trade name of Research Triangle Institute. What is the Optimal Mode? A Comparison of CATI, IVR, SMS Pew Research Center Workshop on Public Opinion in Africa, 30 November 2017 Charles Q. Lau, PhD, MS 1
  • 2. What is the Optimal Mode? 2 SMS & IVR & Web & CATI: Rapid Data for Public Health Surveillance (10 countries) SMS: Tracking Vocational School Graduates SMS: Panel Surveys of Youth SMS & Web: Panel for Job Accelerator Trainees SMS: Feasibility of SMS Surveys IVR: Best Practices in IVR Introductions CATI: Technology Adoption CATI & SMS: Health Worker Panel Survey
  • 3. Mobile Survey Modes 3 CATI IVR SMS Other Modes: Web and Chatbots
  • 4. Errors in Mobile Phone Surveys 4 Coverage Error CATI, IVR, SMS all suffer from same coverage error (exclude people without mobiles)
  • 5. Errors in Mobile Phone Surveys 5 Coverage Error Non- Response Error CATI, IVR, SMS all suffer from same coverage error (exclude people without mobiles) CATI, IVR, SMS share common non-response errors … -No in-person interviewer -Short intros -Poor network connection -Phones turned off
  • 6. Errors in Mobile Phone Surveys 6 Coverage Error Non- Response Error CATI, IVR, SMS all suffer from same coverage error (exclude people without mobiles) CATI, IVR, SMS share common non-response errors … ... but CATI, IVR, SMS differ in non-response errors -No in-person interviewer -Short intros -Poor network connection -Phones turned off -SMS, IVR lack interviewer -SMS, IVR require tech familiarity -SMS requires literacy -Concern about cost for SMS -But: Do SMS at leisure
  • 7. Errors in Mobile Phone Surveys 7 Coverage Error Non- Response Error Measurement Error CATI, IVR, SMS all suffer from same coverage error (exclude people without mobiles) CATI, IVR, SMS share common non-response errors … ... but CATI, IVR, SMS differ in non-response errors CATI allows for probing and more complicated questions … … but IVR and SMS may be better for sensitive topics
  • 8. Previous Research Key Insights  Response rates are low, especially for IVR and SMS  Cross-sectional surveys with a single mode (SMS, IVR, CATI) over-represent young, men, urban, educated  Panel approach with face-to-face recruitment is promising  Mixed mode approaches can improve quality  Weighting might help reduce bias Knowledge Gaps  Limited apples-to-apples comparisons  How biased are survey estimates? Can weighting help?  How do modes differ with regard to cost and time? 8
  • 9. Research Questions 9 1. How do response rates differ among CATI, IVR, SMS, and FTF surveys? 2. How representative are respondents from CATI, IVR, and SMS surveys? 3. Can IVR and SMS provide an unbiased estimate of voting behavior? If there is bias, can weights reduce bias? 4. How does the cost and data collection time differ across survey modes?
  • 10. Method  Mobile Phone Surveys – Modes: CATI, IVR, SMS – All used RDD sampling – IVR and SMS conducted in 2017; CATI in 2016 – IVR, SMS used $1 incentive. CATI had experiment ($0 vs. $1) – IVR and SMS asked same 12 questions; CATI asked 85 questions  FTF: 2013 Nigeria Demographic and Health Survey (paper-and- pencil, area probability sample with HH listing, HH quex used)  Target population for all surveys is general population age 18-64 (though frames differ)  Approach: Compare estimates from CATI, IVR, SMS to FTF (and to each other) 10
  • 11. RQ1. Response Rates [VALUE]% [VALUE]% [VALUE]% CATI IVR SMS Response Rates by Mode 11 Response Rate for FTF is 99%
  • 12. RQ2. Representativeness 12 40% 54% 62% 65% 25% 27% 27% 23% 18% 12% 8% 9% 17% 7% 2% 3% FTF CATI IVR SMS Age by Mode 18-29 30-39 40-49 50-64 53% 33% 28% 36% FTF CATI IVR SMS Female by Mode
  • 13. RQ2. Representativeness 13 39% 5% 8% 3% 26% 9% 11% 6% 22% 46% 32% 41% 13% 40% 50% 51% FTF CATI IVR SMS Education by Mode No school Primary Secondary Post-secondary
  • 14. RQ. Representativeness 14 71% 46% 34% FTF IVR SMS Married by Mode 57% 24% 21% FTF IVR SMS Village by Mode 70% 79% 71% 69% FTF CATI IVR SMS Radio by Mode 81% 85% IVR SMS Read Well by Mode
  • 15. RQ3. Bias 15 31% 68% 59% Estimates of Voting in 2015 Presidential Election SMS IVR True Preliminary analysis: Weighting reduces some bias, but does not eliminate bias. Weighting by some factors (e.g., age) increases bias.
  • 16. RQ4. Cost and Time Data Collection Cost - SMS is 52% the cost of IVR. - In this study, IVR is 43% and SMS is 22% the cost of CATI. However, IVR and SMS had 12 questions, CATI had 85. - CATI is cheaper than IVR and SMS for long (85 qx) surveys -IVR and SMS is cheaper than CATI for short (12 qx) surveys 16 Data Collection Time Data collection days  CATI: 43 days  IVR: 42 days  SMS: 16 days Completes per day  CATI: 88  IVR: 43  SMS: 172
  • 17. Takeaways and Discussion 1. Don’t be fooled by incremental improvements in response rates 2. CATI, IVR, SMS can’t match representativeness of FTF surveys 3. Voting estimates are biased; weighting can’t save the day 4. IVR respondents are slightly more representative than SMS (age, education, village). But are small differences worth the cost? 5. CATI has edge (age, education) over IVR/SMS. 6. For short, simple surveys, IVR & SMS may be best modes if you’re willing to sacrifice a bit of representativeness 7. For complex or larger surveys (questions, N), CATI is your best bet. Unresolved issue: Measurement differences between modes Limitations: Nigeria only; cross-sectional surveys; may be topic specific; can’t separate coverage, non-response, measurement; house effects 17
  • 18. Tool: Surveda for Mixed Mode Surveys 18 As part of Bloomberg Philanthropies Data for Health project, RTI and InSTEDD have built an open-source survey tool for mixed mode (SMS, IVR, Web) surveys: https://surveda.instedd.org
  • 19. More Information Charles Q. Lau, PhD, MS Survey Methodologist +1.919.541.8798 clau@rti.org 19