The Internet of Things is evolving and the number of connected devices growing, but with each new device and form factor created comes some novel way to interact with it – different inputs and outputs. As designers, we have an exciting opportunity in front of us to inform how speech, alongside other modalities, manifests itself on these different devices, and what that ultimately means for the people using them. This presentation was originally given by Nuance's Tim Lynch at SXSW Interactive 2015. Find the 'tweet' buttons throughout the presentation to share your favorite parts.

1. Designing the UI for the
Internet of Things
Tim Lynch
Design lead, mobile and consumer products
Nuance Communications
@clampants | @NuanceInc
Voice interfaces for connected
experiences
#IoTUI
#SXSW

2. A little bit about me…
(Yes, we beat the record.)
I live in Boston, where I
am the lead UX
Designer for web/mobile
devices at Nuance. I
love Austin, have family
here, and almost moved
here, yet somehow I
ended up living in a city
with 800 ft. of snow in a
month and below 0° F
winters. So thank you for
having me and letting
me thaw.

3. A little bit about Nuance…
We innovate technology to reduce the distance between
want and get.
“Tweet this and say not a
bad way to start the day”
“Bring up the record for
patient Nicole Redman”

4. A little bit about our design team…
We are a diverse team of
interaction and visual
designers, user
researchers, dialog
designers across the
country, redefining speech
experiences across
handsets and tablets,
automotive, television,
desktop, IoT, wearables,
augmented reality,
gaming… The list goes on.
Did I mention it’s fun?

5. Narrative design is core to what we
do as designers. We have the ability
to pull from general design experience
to create meaningful voice
experiences for users.

6. The landscape of the
internet of things

7. We – designers and consumers – face
challenges each day when designing
for the Internet of Things:
• Smaller (or varying)
screen sizes
• Designing for
transmodal
experiences
• Disparate device
branding and cues
• User expectations of
immediacy
• And so on.

8. When we talk about devices that make
up the Internet of Things, what comes
to mind?
Light bulbs, thermostats, smart hubs, speakers, Crock Pots,
refrigerators…

9. These devices are the poster children of
connected devices – the IoT we know
and love today. They are generally
“smart devices” that…
a) Are loaded with sensors
b) Can “talk” (convey data and information) to us and
each other

10. But when we consider all the things that
are taking in and communicating data,
we often overlook the things we already
know –things that were smart and
sensor-laden well before the phrase
“Internet of Things.”

11. Sometimes, these things act as the
interface for other connected devices.
Other times, these things unto
themselves have become connected
devices.

12. Then there are
connected things
that feel like
science fiction…
By 2020, there could be
200-billion connected
things, from smart dust
to entire cities.*
How will we interact
with it all?
* http://www.intel.com/content/www/us/en/internet-of-things/infographics/guide-to-iot.html

13. This shift in ecosystem presents a
fundamental challenge.
As people living in this world of connected experiences, we need to
interact with these things in personal, meaningful ways… and our
devices need to interact with us similarly.

14. Interacting with
all these things

15. Small screens
Pebble Time
Consider different form factors
(screens).
Screens
Honeywell WiFi Smart
Thermostat
No screens
Kohler Moxie
Showerhead &
Speaker

16. As these devices pervade our
everyday lives – and become
smaller, pushed into the background,
and more personal – our interactions
with them become more enmeshed
in the day-to-day.

17. Context: How are things used?
Life becomes the context.

18. Different inputs and outputs.
From Nicolas Nova’s “Curious Rituals” - https://curiousrituals.wordpress.com/

19. We’re in the wild-west with these
devices. Each device seems to put a
stake in the ground with some novel
way to interact with it.
Touches, gestures, and swipes on one device trigger one
response, while on another they trigger something
completely different.

20. Method’s Henri (via FastCoDesign) - http://method.com/work/ixda15
“With the rise of smart objects and the connected home,
we’ve found that products increasingly need to
communicate even without a screen, through things like
light and sound patterns.” –Daniel Nacamuli

21. Speech as a
unifying modality

22. Speech is the
simplest and
most human
communication
method.
Using speech, the user
interface becomes almost
invisible and the
experience is as natural as
part of the day-to-day.

23. The goal is to
communicate with
devices as we
would each other
– as humans.
How Voice Activates and Advances the Human-Computer Relationship - by Clifford Nass - http://goo.gl/YYozKs
Dialog evokes meaning,
identity, emotion, and trust.

24. Let’s pretend you have a connected
mattress.
The Chattress
Hey friend
:)

25. Foundations of the ‘voice interface’
The Chattress
You seem down
today…what’s
wrong?
ASR
Recognize the
words
NLU
Meaning
behind the
words
Dialog
Appropriate
response
Nothing,
mattress :(

26. Intelligence is the secret sauce that
permeates the entire experience,
influencing the conversational
partnership we have and creating a
much more personalized experience.

27. Intelligence & personalization
The intelligent Chattress
Intelligence
Context Knowledge
Memory
‘Smarts’ can manifest itself
through context, memory,
and knowledge.

28. Design considerations
(Tweet this section)

29. A speech experience itself, though
– without a holistic and thoughtful
design – won’t make for a good
user experience.
Speech experiences will fail (or fail to be adopted)
when they:
• Fail to meet people’s expectations
• Don’t take into account context or other modalities
• Are hidden or unclear in purpose
• Ignore conversational norms
• Treat errors as dead-ends

30. Understand
expectations
Leverage the
strengths of
speech
Partner with
other modalities
Introduce
yourself
Frame the
scope
Support what
is natural
Provide
conversational
feedback
Identify errors
as opportunities
Deliver a
consistent point
of view
To remedy this, consider these speech
design factors.

31. 1. Understand expectations.
People don’t want to
speak to their
devices simply to
have a
conversation…
they want to get
things done.
Design with those
things in mind.
Police Dog, Tess (via the State Library of New South Wales, NZ - https://flic.kr/p/5TJoyH)

32. 2. Leverage the strengths of speech.
Speech can empower goal-oriented tasks, streamline existing
flows, and improve the experience in certain contexts.

33. 3. Partner with other modalities.
Rarely should speech be
thought about as the only
modality.
Use it to support and
amplify other modalities.

34. 4. Introduce yourself.
Make speech obvious and well-integrated into the full
experience. Once discovered, people will experiment.

35. 5. Frame the scope with guidance.
The promise of natural language is you can say anything.
A challenge is you think you can say anything…

36. 6. Support what is natural.
The “natural” in “natural language” can mean lengthy
phrases… but it can also mean simple fragments. Natural
language should encompass structured commands, but be
able to extend out to full, grammatically correct sentence
structure.

37. 7. Provide conversational feedback.
Speech systems should follow our own conversational
norms, conveying they are listening and understanding.

38. 8. Identify “errors” as opportunities.
“Sorry, I’m not
hearing anything.
Try checking your
mic settings.”
Audio
“Sorry, what time
was that?”
Recognition
“Did you mean
two people or two
o’clock?”
Interpretation
“Ah…I can’t help
you with that yet,
but try this…”
Dialog
It’s OK if something goes wrong – as long as people
understand what happened (and what to do to fix it).
(Tweet this)

39. 9. Deliver a consistent
point of view.
Dialog, TTS, visual, audio, interaction, scope, content, form
factor… should all work in concert across devices.

40. So, if you take away one thing,
it should be…
Thoughtfully-designed speech
systems allow us to meaningfully
interact with our connected
devices.

41. Thank you!
Tim Lynch
Design lead, mobile and consumer products
Nuance Communications
@clampants | @NuanceInc
#IoTUI
#SXSW