Presentation for Pew Research Center's Methodological Workshop on Public Opinion Research in Sub-Saharan Africa

1. Using Mobiles for Survey Research
in Africa
Lessons Learned
Bill Bell
Director of Research
Voice of America

2. 2
Voice of America
Federally funded; multi-platform content
provided in 47 languages
Target countries are those lacking free media
29 % of audience is in Africa
Biggest audiences in Nigeria, Tanzania, Kenya,
Ethiopia
Most recent language additions have been in
Africa
2

3. 3
VOA Research
Robust program of survey-based research
designed to:
• Assess audience size, composition, and
programming preferences
• Understand competitive factors
• Track changes in overall media consumption
patterns
Usually national samples of 1500-2000
All F2F; traditionally P&P but increasingly CAPI
3

4. 4
What’s Wrong with the Status Quo?

5. 5
Are we there yet?...

6. 6
Which way was that
sampling point?

7. 7
Anyone home?

8. 8
Are Mobile Surveys the Answer?
Definition: Administration of survey instrument
to randomly selected owners of mobile devices
via SMS, IVR, or mobile web
8

9. 9
How Mobile Surveys Work
Vendors obtain operator-provided lists of numbers,
which serve as sampling frame
Survey invitations sent to randomly selected numbers
SMS: Respondents opting in are sent short list of
questions as individual text messages with numeric
response categories
IVR: Questions sent in audio form; numeric responses
No cost to respondent; and respondents are
incentivized with free air time where possible

10. 10
Potential Advantages of Mobile
Speed: (weeks vs. months)
Cost: ($5-10K/country vs. $100K +)
Overcome geographic limits due to conflict or
logistics
No clustering effects
Pre-existing sampling frames (lists of mobile
numbers)
Easy to incentivize participation via air time
credits
10

11. 11
Disadvantages
Limits on number/type of question and
response categories
Cost per question
Uncertain quality of sampling frames
• Quality depends on accuracy of lists provided by
mobile operators and number of operators making
information accessible
Coverage limits (geographic, demographic)
11

12. 12
Mobile Ownership in Selected African Countries
87
85
79
59
55
47
45
0
10
20
30
40
50
60
70
80
90
100
Nigeria (2016) Kenya (2017) Cote d'Ivoire
(2017)
Mali (2017) Tanzania
(2015)
DRC (2016) Zambia (2016)
Source: BBG surveys in years shown
12
Percentage of adults (15+) who personally own a mobile

13. 13
Demographics of Mobile Ownership
Percentageofeachgroupowningamobiledevice
62
68
47
50
60
69
53 54
0
10
20
30
40
50
60
70
80
Tanzania Mali
Male Female Urban Rural
13

14. 14
Our Experiment
Goal: Explore whether mobile surveys could
substitute for F2F in some markets
Methods: Conduct mobile surveys in markets
where recent F2F results were available
Compare sample composition, media
access/use findings; basic audience data
14

15. HOW DO MOBILE SURVEY RESULTS STACK
UP AGAINST TRADITIONAL F2F?

16. 16
Background and Comparison
Mobile surveys run in Tanzania, DRC, Ghana
Kenya); only SMS surveys shown here
Sample sizes ca. 500
Note questions and response categories not
fully comparable due to limitations of SMS
format
Mobile survey results are compared here to
that portion of the F2F samples that owned
mobiles and used them for SMS

17. 17
Methods
DRC Ghana Kenya Tanzania
N = 555 552 613 617
Languages(s) French English English/Swahili 50/50 Swahili
Completion
rate*
1.02% 1.99% 1.3% 0.29%
No. of poll
days
4 1 7 1
F2F n= 620 602 1088 1210
17
*Completion rate = number of complete interviews/number of
invitations sent out

18. 18
Question Form
Constraints of SMS format required
“compression” of questions and responses,
possibly affecting respondents’ comprehension
and data quality
18
Do you ever get news about current events in your country or around the
world on your mobile phone? 1) Yes 2) No [Reply 1 or 2]
Which of these items do you have in your household? 1) Radio 2) TV 3)
Computer [Choose all numbers that apply]

19. 19
Age
15-24
25-34
35+
Mobile Survey
F2F
TanzaniaKenyaGhanaDRC
34%
28%
38%
24%
37%
37%
35%
29%
36%
5%
26%
69%
30%
38%
32%
17%
28%
57%
39%
29%
32%
11%
24%
65%
Avg Age:
Mobile
F2F
29.7 23.7 26.3 24.9
32.3 32.6 30.9 33.4

20. 20
Gender
Male
Female
TanzaniaKenyaGhanaDRC
41%
59%
21%
79%
49%
51%
24%
76%
53%
47%
30%
70%
46%
54%
30%
70%
Mobile Survey
F2F

21. 21
Home Ownership of Media Technologies
Computer
Radio
TV
TanzaniaKenyaGhanaDRC
54%
85%
9%
64%
57%
37%
83%
94%
21%
59%
36%
25%
45%
94%
5%
41%
58%
22%
27%
91%
3%
38%
51%
4%
Mobile Survey
F2F

22. 22
Use of Mobile Phone Features
View a video
Access the Internet
Take a picture
TanzaniaKenyaGhanaDRC
56%
10%
13%
11%
11%
50%
41%
22%
28%
32%
28%
28%
24%
10%
9%
10%
36%
42%
17%
28%
31%
35%
25%
13%
7%
4%
13%
40%
32%
26%
23%
36%
31%
17%
3%
4%
3%
3%
43%
25%
24%
22%
27%
19%
Access FB
Send emails
Listen to radio
N/A N/A
Mobile Survey
F2F
N/A N/A

23. 23
Radio Station Listenership
China Radio
GBC1
RFI
TanzaniaKenyaGhanaDRC
9.6%
6.5%
23.1%
5.4%
25.7%
20.1%
15.1%
7.2%
34.0%
8.5%
12.6%
22.2%
23.3%
39.7%
46.8%
8.9%
12.3%
44.1%
BBC
DW
VOA
Mobile Survey
F2F
9.3%
48.8%
14.1%
15.1%
48.3%
19.3%

24. 24
TV Viewership
CCTV
France24
TanzaniaKenya
GhanaDRC
2.5%
28.3%
18.0%
1.5%
7.2%
47.9%
26.5%
7.0%
2.0%
4.6%
5.2%
6.2%
17.6%
7.2%
21.0%
0.7%
3.2%
2.7%
14.0%
15.0%
26.8%
7.6%
7.8%
13.7%
7.9%
5.5%
6.8%
32.6%
12.0%
BBC TV
DW TV
VOA
TV5 Monde
Al Jazeera
Mobile Survey
F2F

25. 25
Why Such Large Differences?
SMS surveys may engage segment of
population that is very different, not just in
terms of demos but also behavior
Changes in question format and response
categories may affect results
Low response rates on mobile especially
problematical, though direction of bias is
unclear
25

26. 26
Moving Forward …
 At present mobile surveys don’t permit sufficient statistical
rigor to be used for KPI estimates
 They may become more useful to the extent that we’re
interested in the behavior of highly specific population
segments , especially if we’re able to boost participation
 The methodology could be particularly useful in instances
where just need answers to one or two quick questions,
especially if we’re willing to live with less rigorously derived
results
 Mobiles also offer the opportunity for panel based
evaluation cheaply and quickly