Discover, manage, deploy, monitor – rinse and repeat.  In this session we show how Azure Machine Learning can be used to create the right AI model for your challenge and then easily customize it using your development tools while relying on Azure ML to optimize them to run in hardware accelerated environments for the cloud and the edge using FPGAs and Neural Network accelerators.  We then show you how to deploy the model to highly scalable web services and nimble edge applications that Azure can manage and monitor for you.  Finally, we illustrate how you can leverage the model telemetry to retrain and improve your content.

1. AI for Intelligent Cloud and
Intelligent Edge:
Discover, deploy, and manage
with Azure ML Services
James Serra
Microsoft
Technical Architect, Data & AI
Blog: JamesSerra.com

2. About Me
 Microsoft, Big Data Evangelist
 In IT for 30 years, worked on many BI and DW projects
 Worked as desktop/web/database developer, DBA, BI and DW architect and developer, MDM
architect, PDW/APS developer
 Been perm employee, contractor, consultant, business owner
 Presenter at PASS Business Analytics Conference, PASS Summit, Enterprise Data World conference
 Certifications: MCSE: Data Platform, Business Intelligence; MS: Architecting Microsoft Azure
Solutions, Design and Implement Big Data Analytics Solutions, Design and Implement Cloud Data
Platform Solutions
 Blog at JamesSerra.com
 Former SQL Server MVP
 Author of book “Reporting with Microsoft SQL Server 2012”

3. I tried to understand AI products on my own…
And felt like I was body slammed by Randy
Savage:
Let’s prevent that from happening…

4. Intro to Azure Machine Learning
Model Management
Hardware Acceleration
Edge Integration
AICloud+Edge

5. Machine learning is a data science technique that allows computers to
use existing data to forecast future behaviors, outcomes, and trends.

6. Prepare Data Build & Train Deploy
Custom AI
Building your own AI models for Transforming Data into Intelligence

7. © Microsoft Corporation
Advanced analytics pattern in Azure
Azure Data
Lake store
Azure
Storage
HDInsightAzure Databricks
Azure ML
Services
ML server
Model training
Long-term storage Data processing
Azure Data
Lake Analytics
Azure ML
Studio
SQL Server
(in-database ML)
Azure Databricks
(Spark ML)
Data Science
VM
Cosmos DB
Serving storage
SQL DB
SQL DW
Azure Analysis
Services
Cosmos DB
Batch AI
SQL DB
Azure Data
Factory
Orchestration
Azure Container
Service
Trained model hosting
SQL Server
(in-database ML)
Data collection and understanding, modeling, and deployment
Sensors and IoT
(unstructured)
Logs, files, and media
(unstructured)
Business/custom apps
(structured)
Applications
Dashboards
Power BI

8. © Microsoft Corporation
Power BI – build your own ML models

9. Azure AI Services
Azure Infrastructure
Tools

11. © Microsoft Corporation
Machine learning and AI portfolio
What engines do you want to use?
Deployment target
Which experience do you want?
Build your own or consume
pre-trained models?
Microsoft ML &
AI products
Build your own
Azure Machine
Learning
Code first
(On-prem)
ML Server
On-prem
Hadoop
SQL Server
(cloud)
BYOT
SQL Server Hadoop Azure Batch DSVM Spark
Visual tooling
(cloud)
AML Studio
Consume
Cognitive
services, bots
Spark ML,
SparkR, SparklyR
Notebooks Jobs
Azure Databricks
Spark
When to use what
AI Decision tree: https://biz-excellence.com/2019/02/15/ai-dt/

12. Cognitive Services capabilities
Infuse your apps, websites, and bots with human-like intelligence
https://azure.microsoft.com/en-us/services/cognitive-services

13. Model Management
Hardware Acceleration
Edge Integration

14. Data Science
Lifecycle
TEAM DATA SCIENCE PROCESS (TDSP) IS MICROSOFT’S AGILE,
ITERATIVE METHODOLOGY TO DELIVER PREDICTIVE ANALYTICS
SOLUTIONS AND INTELLIGENT APPLICATIONS EFFICIENTLY
https://docs.microsoft.com/en-us/azure/machine-learning/team-data-science-process/overview

15. WHAT IS AZURE MACHINE LEARNING SERVICE?
Set of Azure Cloud
Services
Python
SDK
 Prepare Data
 Build Models
 Train Models
 Manage Models
 Track Experiments
 Deploy Models
That enables
you to:

16. What is Azure Machine Learning service?
Start training on your local machine and then scale out to the cloud

17. Azure Machine Learning Services
• Deprecate Azure Machine Learning Workbench
• Unified SDK, CLI and UX for training and deploying models
• Full Integration with Visual Studio Code and Azure DevOps
• Improved support for multiple compute targets
• Four new models for FPGA
• Vision AI Dev Kit available to order on Oct 1

18. Azure Machine Learning Workspace layout

20. The Azure ML Deployment Pipeline

21. Understanding the Edge: Heavy Edge vs Light Edge
Cloud: Azure Heavy Edge Light Edge
Descriptio
n
An Azure host that
spans from CPU to
GPU and FPGA VMs
A server with slots to insert CPUs, GPUs, and FPGAs or a X64 or ARM system that
needs to be plugged in to work
A Sensor with a SOC (ARM CPU, NNA, MCU) and memory
that can operate on batteries
Example
DSVM / ACI / AKS /
Batch AI
- DataBox Edge
- HPE
- Azure Stack
- DataBox Edge - Industrial PC
-Video Gateway
-DVR
-Mobile Phones
-VAIDK
-Mobile Phones
-IP Cameras
-Azure Sphere
- Appliances
What runs
model
CPU,GPU or FPGA
CPU,GPU or
FPGA
CPU, GPU x64 CPU Multi-ARM CPU
Hw accelerated
NNA
CPU/GPU MCU
Why Edge? latency, less data sent, filter, aggregate, work offline

22. Data Science Virtual Machines

25. AZURE MACHINE LEARNING STUDIO
Platform for emerging data scientists
to graphically build and deploy
experiments
• Rapid experiment composition
• > 100 easily configured modules
for data prep, training, evaluation
• Extensibility through R & Python
• Serverless training and
deployment
Some numbers:
• 100’s of thousands of deployed
models serving billions of requests

26. Comparable Table
Azure Machine Learning Studio Machine Learning Services
Pros • Rapid development (Drag and Drop)
• Works well with relatively simple
datasets
• Pre-built ML algorithms
• Cheap
• Fast (VMs with GPUs)
• Different optimization methods,
CI/CD pipeline
• Full control during training
• Manage computing resources
(choose VM size)
• Use open source ML libraries
Cons • Can be slow
• Limited optimization methods,
operationalized architecture
• Less control during training
• Fixed computing resources
• More elaborate to build, require
deeper knowledge of machine
learning
• Deeper models need much more
data with much more memory
• Higher costs for VM with GPU

27. https://gallery.azure.ai/

28. Model Management and Deployment Demo
UX, Azure Dev Ops and CLI

29. Model Management
Hardware Acceleration
Edge Integration

30. AI in Action
https://github.com/Microsoft/Cognitive-Samples-IntelligentKiosk

31. Breakthroughs in deep
learning demand real-time AI
Convolutional Neural Networks (CNN)
ht-1 ht ht+1
xt-1 xt xt+1
ht-1 ht ht+1
yt-1 yt yt+1
Recurrent Neural Networks (RNN)
Deep neural networks (DNN) have enabled major
advances in machine learning and AI
Computer vision
Language translation
Speech recognition
Question answering
And more…
Problem
DNNs are challenging to serve and deploy
in large-scale online services
Heavily constrained by latency, cost, and power
Size and complexity outpacing growth of commodity CPUs

32. Microsoft Hardware Acceleration
FPGAs
EFFICIENCYFLEXIBILITY
CPUs GPUs
ASICs
INFERENCING
CPUs, GPUS, FPGAs
TRAINING
CPUs and GPUs
Cloud
Edge
INFERENCING
CPUs, GPUS, FPGAs,
ASICs
TRAINING
(HEAVY EDGE)
CPUs and GPUs

33. hardware architecture designed to accelerate real-time AI
calculations
Project Brainwave unique
advantage (DNN models and
FPGA)
No batching required
Brainwave delivers the ideal combination:
High hardware utilization
Low latency
Low batch sizes
In short, it is a hardware architecture and learning platform
designed to accelerate real-time AI calculations
Batch Size
Performance
Brainwave
NPU
1256

34. Credit: Henk Monster. Licensed under the Creative Commons Attribution 3.0 Unported license.

35. Licensed under the Creative Commons Attribution 2.0 Generic license

36. Project Brainwave use cases
For accelerated real-time image processing:
- Identify manufacturing defects
- Detect spills or open freezer doors in a retail store
- Conduct real-time medical image analysis
- Tract endangered species
- Detect cars parked in a fire lane

38. The power of deep learning on FPGA
Performance Flexibility Scale
Rapidly adapt to evolving ML
Inference-optimized numerical precision
Exploit sparsity, deep compression
Excellent inference at low batch sizes
Ultra-low latency | 10x < CPU/GPU
World’s largest cloud investment in FPGAs
Multiple Exa-Ops of aggregate AI capacity
Runs on Microsoft’s scale infrastructure
Low cost
$0.21/million images on Azure FPGA (inferencing)

39. Project BrainWave
A Scalable FPGA-Powered DNN Serving Platform
Fast:
Flexible
Friendly:
F F F
L0
L1
F F F
L0
Pretrained DNN Model
in TensorFlow, CNTK, etc.
Scalable DNN Hardware
Microservice
BrainWave
Soft DPU
Instr Decoder
& Control
Neural FU
Network switches
FPGAs

40. Azure ML and Project Brainwave
• New DNN models
• ResNet 152, DenseNet-121, VGG-16, SSD-VGG
• Customizable weights
http://aka.ms/aml-real-time-ai
Easily deploy models to FPGAs for ultra-low latency with
Azure Machine Learning powered by Project Brainwave

41. DAWNBench

42. Model Management
Hardware Acceleration
Edge Integration

43. Why Intelligent Edge?
High-speed data processing,
analytics and shorter response
times are more essential than ever.
Intelligent Cloud
• Business agility and scalability: unlimited computing
power available on demand.
Intelligent Edge
• Can handle priority-one tasks locally
even without cloud connection.
• Can handle generated data that is too
large to pull rapidly from the cloud.
• Enables real-time processing through
intelligence in or near to local devices.
• Flexibility to accommodate data privacy related
requirements.

44. The components of a ML application
Vision
AI dev
kit
Vision
AI dev
kit

45. Building
AI
Solutions
for the
Intelligent
Edge

46. Vision AI Developer Kit
Hardware Specification
Tutorial: Develop a C# IoT Edge module and deploy to your simulated device
https://docs.microsoft.com/en-us/azure/iot-edge/tutorial-csharp-module

47. Vision AI Developer Kit

48. Vision AI Developer Kit
A connected camera reference solution
Altek version available to order soon at https://visionaidevkit.com

49. Q & A ?
James Serra, Big Data Evangelist
Email me at: JamesSerra3@gmail.com
Follow me at: @JamesSerra
Link to me at: www.linkedin.com/in/JamesSerra
Visit my blog at: JamesSerra.com (where this slide deck is posted via the “Presentations” link on the top menu)