Thoughts on the past, present and future of device APIs for (mostly mobile) web developers. NB: is probably out of date already.

1. (Mobile) Device APIs
James Pearce
@jamespearce
http://tripleodeon.com

2. What are Device APIs?
“Client-side APIs that enable the
development of Web Applications that
interact with device hardware”
...and other capabilities outside of the
browser’s traditional remit

3. Why are they important?
• They’re not...
...if you’re happy with a web of documents
• They are...
... if you want the the web to be
competitive with native app platforms

4. History
Current Work
Use Cases

7. WTAI Make Call
<a
href="wtai://wp/mc;16505551234">
Call
us
</a>
<a
href="call.wmls#makeCall()">
Call
us
</a>
//
call.wmls
extern
function
makeCall()
{
WTAPublic.makeCall("16505551234");
}
http://www.wapforum.org/what/technical/wtai-30-apr-98.pdf

8. WTAI Call Control
<a
href="calls.wmls#setupCall()">Call
us!</a>
<a
href="calls.wmls#acceptCall()">Brring!</a>
<a
href="wtai://cc/sd;1*2*3456">Enter
code</a>
//
calls.wmls
extern
function
setupCall()
{
WTACallCont.setup("16505551234",
1);
}
extern
function
acceptCall()
{
WTACallCont.accept("1",
1);
}

9. WTAI Messaging
//
sms.wmls
extern
function
sendMsg()
{
WTANetText.send("16505551234",
"WML
rocks");
}
extern
function
readMsg(i)
{
var
sms
=
WTANetText.read(i);
return
WTANetText.getFieldValue(
sms,
"body"
);
}

10. WTAI Contacts
//
contacts.wmls
extern
function
addFriend(i,
num,
name)
{
WTAPhoneBook.write(i,
num,
name);
}
//
return
structs
of
contact
details
extern
function
getFriendById(i)
{
return
WTAPhoneBook.read("i",
i);
}
extern
function
getFriendByName(name)
{
return
WTAPhoneBook.read("t",
name);
}

11. I may be laboring the point
But that was 1998...
... and you still can’t do any of this with a
contemporary mobile browser

13. BONDI
“enables web based content to access native
device capability, intermediated through a
robust, but flexible security framework”
http://www.omtp.org/1.11

14. Messaging
var
sms
=
bondi.messaging.createSMS({
to:
["16505551234"],
body:
"Home
soon"
});
var
mms
=
bondi.messaging.createMMS({
to:
["16505551234"],
subject:
"I
will
be...",
body:
"...home
soon"
});

15. Messaging
bondi.messaging.sendSMS(
function
()
{},
//
success
function
()
{},
//
error
sms,
true
);
bondi.messaging.sendMMS(
function
()
{},
//
success
function
()
{},
//
error
mms,
true
);

16. Messaging
bondi.messaging.subscribeToSMS(
function
()
{},
//
success
function
()
{},
//
error
function
()
{
//
listener
alert("They
replied!");
},
{
from:
"16505551234"
//
filter
},
true
//
exclusive
);

17. Contacts
bondi.pim.contact.getAddressBooks(
function(books)
{
//
success
var
contact
=
books[0].createContact({
name:
'James
Pearce',
emails:
[{email:'j@tripleodeon.com'}],
phoneNumbers:
[{number:'16505551234'}]
});
},
function
()
{}
//
error
);

18. Telephony
bondi.telephonyLog.findLogEntries(
function
()
{},
//
success
function
()
{},
//
error,
[
bondi.telephonyLog.MISSED_CALLS_FOLDER,
bondi.telephonyLog.RECEIVED_CALLS_FOLDER
],
{
phoneNumber:
"16505551234"
}
);
* Yes, it’s true: you can’t actually make or receive calls

19. App Launcher
bondi.applauncher.launchApplication(
function
()
{},
//
success
function
()
{},
//
error
"tel:16505551234"
);
//
mailto:...
//
sms:...
//
file:...

20. On one hand...
• Security-conscious architecture
• Comprehensive API support
• Consistent API patterns
• Conformance specifications
• Reasonable test coverage
http://www.omtp.org/1.11/apis

21. On the other...

22. bondi.*
deviceapis.*

23. WAC 2.0
• Accelerometer • File system
• Calendar • Geolocation
• Camera • Messaging
• Contacts • Orientation
• Core • Tasks
• Device interaction • Viewport
• Device status • Webview
http://specs.wacapps.net/

24. Which is all awesome
Except...

27. Device APIs Working Group
“to create client-side APIs that enable the
development of Web Applications and Web
Widgets that interact with devices services
such as Calendar, Contacts, Camera, etc”
Joint efforts with WebRTC Working Group,
WebApps Working Group
http://www.w3.org/2009/dap/

28. W3C Device APIs
• Media Capture(HTML)
• Vibration
• Media Capture(gUM)
• Calendar
• Contact • Permissions
• Messaging • Menu
• Battery Status • Gallery
• Network Information • System info
• Sensor • Media Stream

29. Media Capture (HTML)
<input
type="file"
accept="image/*"
/>
<!-­‐-­‐
The
accept
attribute
is
'a
hint'.
Specification
suggests
that
it
can
be
used
to
offer
alternative
picker
UIs.
-­‐-­‐>
Public working draft, http://www.w3.org/TR/2011/WD-html-media-capture-20110414

30. Brad Lassey, September 2008

31. Media Capture (HTML)
<input
type="file"
accept="image/*"
capture="camera"
/>
<!-­‐-­‐
Again,
a
hint.
Valid
options
are:
camera,
camcorder,
microphone,
filesystem
-­‐-­‐>

33. Media Capture (HTML)
<input
type="file"
accept="image/*"
capture="camera"
id="photo"
/>
var
photo
=
document.getElementById('photo');
photo.files[0].getFormatData(
function
(data)
{},
//
success
function
()
{}
//
error
);
//
file
is
a
MediaFile,
extending
File
//
data
is
a
MediaFileData

34. MediaFileData
interface
MediaFileData
{
attribute
DOMString
codecs;
attribute
unsigned
long
bitrate;
attribute
unsigned
long
height;
attribute
unsigned
long
width;
attribute
float
duration;
};
...seems to be under discussion

35. Media capture (gUM)
navigator.getUserMedia(
{
audio:
true,
video:
false
},
function
(stream)
{},
//
success
function
()
{}
//
error
);
//
stream
is
a
LocalMediaStream
Editor's draft, http://dev.w3.org/2011/webrtc/editor/getusermedia.html

36. LocalMediaStream
var
recorder
=
stream.record();
//
recorder
is
a
MediaStreamRecorder
recorder.getRecordedData(
function
(file)
{}
//
success
);
//
file
is
a
File
URL.createObjectURL(stream);
//
a
Blob
URI
for,
say
<video
src='...'
/>

37. Messaging
navigator.sendMessage(
"sms:16505551234?"
+
"body=JavaScript%20says%20hi",
[],
//
attachments
as
File
array
function
()
{},
//
success
function
()
{}
//
error
);
//
also
mms:
and
mailto:
//
oh...
were
you
looking
for
receipt?
Editor's draft, http://dev.w3.org/2009/dap/messaging/

38. Sensors
navigator.findSensors().onsuccess
=
function()
{
this.result.forEach(function
(sensor)
{
//
...
});
};
var
sensor
=
new
SensorConnection('Temperature');
sensor.addEventListener('sensordata',
function(e)
{
if(e.data
>
20.0)
{
window.console.log('Hawt');
}
});
sensor.startWatch({
interval:
2000
});
Editor's draft, http://dev.w3.org/2009/dap/system-info/Sensors.html

39. Sensors
• Temperature, ºC
• AmbientLight, Lux
• AmbientNoise, dbA
• MagneticField, uTesla
• Proximity, cm
• AtmPressure, kP
• RelHumidity, %

40. Contacts
navigator.contacts.find(
[
//
fields
'name',
'emails'
],
function
(contacts)
{},
//
success
function
()
{},
//
error
{
//
find
options
filter:
'James',
multiple:
true
}
);
Editor's draft, http://w3c-test.org/dap/contacts/

41. Contact
interface
Contact
{
readonly
attribute
DOMString
id;
attribute
DOMString?
displayName;
attribute
ContactName?
name;
attribute
DOMString?
nickname;
attribute
ContactField[]?
phoneNumbers;
attribute
ContactField[]?
emails;
attribute
ContactAddress[]?
addresses;
attribute
ContactField[]?
ims;
attribute
ContactOrganization[]?
organizations;
attribute
Date?
birthday;
attribute
DOMString?
note;
attribute
ContactField[]?
photos;
attribute
DOMString[]?
categories;
attribute
ContactField[]?
urls;
};

42. Cool, but...
“The ability to add and update contact
information is not a function of the API
provided in this specification.”

43. Holy Contacts Editing WTF
var
vcard
=
'BEGIN:VCARDrn'
+
'VERSION:3.0rn'
+
'UID:'
+
contact.id
+
'rn'
+
'N:Pearce;Jamesrn'
+
'FN:James
Pearcern'
+
'END:VCARD';
document.getElementById('vcard').href
=
'data:text/x-­‐vcard;charset=utf-­‐8,'
+
encodeURIComponent(vcard);

46. WebAPI
“We are aiming at providing all the necessary
APIs to build a basic HTML5 phone
experience within the next 3-6 months”
https://wiki.mozilla.org/WebAPI

47. WebAPI
• Telephony • Embedded browser
• Messaging • Battery
• Contacts • Device Status
• Camera • Settings
• Network • Mouse Lock
• USB • Vibrator

48. Hopefully W3C++
• “Our proposal might or might not look like
it for the moment”

49. Telephony!
var
tcall
=
navigator.telephony.dial("16505551234");
tcall.readyState;
//
dialing,
ringing,
busy,
connecting,
//
connected,
disconnecting,
disconnected,
//
incoming
tcall.onconnected
=
function
()
{};
//
onbusy,
ondisconnected
etc
tcall2.answer();
http://www.youtube.com/watch?v=RuIQskGD3u0#t=19s

50. Contacts
“We need read/write”

51. USB
var
usb
=
new
MozUSBManager();
usb.addEventListener("attachdevice",
function
()
{}
);
usb.addEventListener("detachdevice",
function
()
{}
);
usb.claimDevice({
deviceClass:
...,
deviceSubClass:
...
});

53. We deserve Device APIs...
• As forward-looking as WTAI
• As well-structured as BONDI
• As comprehensive as WAC
• As respectable as W3C DAP
• As agile as WebAPI

54. And we’ve got...

55. PhoneGap
• Accelerometer • Events
• Camera • File
• Capture • Geolocation
• Compass • Media
• Connection • Notification
• Contacts • Storage
• Device

56. Use Cases

57. Useless Demos
https://github.com/jamesgpearce/compios5

58. Fixing layout bugs, FML
window.addEventListener("deviceorientation",
function
(e)
{
orientation
=
Math.abs(w.orientation);
rotation
=
Math.abs(e.gamma);
if
(rotation
>
8
&&
orientation
===
0)
{
if
(enabled)
{
disableZoom();
}
else
{
restoreZoom();
}
}
},
false
); http://vimeo.com/adactio/iosbug
https://github.com/scottjehl/iOS-Orientationchange-Fix

59. Camera
• Profile pictures
• Review sites
• Photo sharing
• Recognition
• Bar-codes
• Text
• Faces

60. Filters!
$('img').vintage({
vignette:
{
black:
0.8,
white:
0.2
},
noise:
20,
screen:
{
red:
12,
...
},
desaturate:
0.05
});
https://github.com/rendro/vintageJS

61. Augmented reality?
http://fhtr.org/JSARToolKit/

62. Contacts & calendaring
• Add business details to address book
• Web-based messaging
• Facial recognition again...
• Presence
• Context-sensitivity of content

63. Messaging
• Sharing
• Notifications
• Offline transport fallback
• Frictionless web sign-on

64. Expect more

65. I still want...
• Audio processing
• Bluetooth
• RFID & NFC
• Both cameras
• Memory & power
• Native recognition

66. window.addEventListener(
'appointmentdue',
function(appt)
{
if
(device.near(WORK))
{
siri.remind(
contacts.get('Peter'),
calendar.getDetails(appt)
);
}
},
false
);
PS: does not work

67. James Pearce
@jamespearce
http://tripleodeon.com