This document provides an overview and agenda for a workshop on end-to-end machine learning pipelines using TFX, Kubeflow, Airflow and MLflow. The agenda covers setting up an environment with Kubernetes, using TensorFlow Extended (TFX) components to build pipelines, ML pipelines with Airflow and Kubeflow, hyperparameter tuning with Kubeflow, and deploying notebooks with Kubernetes. Hands-on exercises are also provided to explore key areas like TensorFlow Data Validation, TensorFlow Transform, TensorFlow Model Analysis and Airflow ML pipelines.
2. Founder @ PipelineAI
Continuous Machine Learning in Production
Former Databricks, Netflix
Apache Spark Contributor
O’Reilly Author
High Performance TensorFlow in Production
Meetup Organizer
Advanced Kubeflow Meetup
Who Am I? (@cfregly)
9. After Pushing Your Model to Production, Your Model is…
1 Already Out of Date – Need to Re-Train
Biased – Need to Validate Before Pushing
Broken – Need to A/B Test in Production
Hacked – Need to Train With Data Privacy
Slow – Need to Quantize and Speed Up Predictions
2
3
4
5
10. 1 Setup Environment with Kubernetes
TensorFlow Extended (TFX)
ML Pipelines with Airflow and KubeFlow
Agenda – Part 1 of 2
Hyper-Parameter Tuning with KubeFlow
Deploy Notebook with Kubernetes
2
3
4
5
11. Agenda – Part 2 of 2
6 MLflow
7
8
9
10
TensorFlow Privacy
Model Serving & A/B Tests
Model Optimization
Papermill
12. Note #1 of 11
IGNORE WARNINGS & ERRORS
(Everything will be OK!)
13. Note #2 of 11
THERE IS A LOT OF MATERIAL HERE
Many opportunities to explore on your own.
(Don’t upload sensitive data.)
14. Note #3 of 11
YOU HAVE YOUR OWN INSTANCE
16 CPU, 104 GB RAM, 200GB SSD
(Each with a full Kubernetes Cluster.)
15. Note #4 of 11
DATASETS
Chicago Taxi Dataset
(and various others.)
16. Note #5 of 11
SOME NOTEBOOKS TAKE MINUTES
Please be patient.
(We are using large datasets)
17. Note #6 of 11
QUESTIONS?
Post questions to Zoom chat or Q&A.
(Antje and I will answer soon)
Antje >
18. Note #7 of 11
KUBEFLOW IS NOT A SILVER BULLET
There are still gaps in the pipeline.
(But gaps are getting smaller)
19. Note #8 of 11
THIS IS NOT CLOUD DEPENDENT*
*Except for 2 small exceptions…
(Patches are underway.)
20. Note #9 of 11
PRIMARILY TENSORFLOW 1.x
TF 2.x is not fully supported by TFX
(Until Mid-2020.)
21. Note #10 of 11
SHUTDOWN EACH NOTEBOOK AFTER
We are using complex browser voo-doo.
(Javascript is a mystery.)
22. Note #11 of 11
Retrieve 1 Single IP Address Here…
<INSERT HERE>
(Do not click refresh.)
23. 1 Setup Environment with Kubernetes
TensorFlow Extended (TFX)
ML Pipelines with Airflow and KubeFlow
Agenda – Part 1 of 2
Hyper-Parameter Tuning with KubeFlow
Deploy Notebook with Kubernetes
2
3
4
5
24. Agenda – Part 2 of 2
6 MLflow
7
8
9
10
TensorFlow Privacy
Model Serving & A/B Tests
Model Optimization
Papermill
29. System 6
System 5System 4
Training
At Scale
System 3
System 1
Data
Ingestion
Data
Analysis
Data
Transform
Data
Validation
System 2
Build
Model
Model
Validation
Serving Logging
Monitoring
Roll-out
Data
Splitting
Ad-Hoc
Training
Why TFX and Why KubeFlow?
Improve Training/Serving
Consistency
Unify Disparate Systems
Manage Pipeline Complexity
Improve Portability
Wrangle Large Datasets
Improve Model Quality
Manage Versions
Composability
Distributed
Training
Configure
30. 1.2 TensorFlow Extended (TFX)
Feature
Load
Feature
Analyze Feature
Transform
Model
Train
Model
Evaluate
Model
Deploy
Reproduce
Training
Jenkins?
36. 1 Setup Environment with Kubernetes
TensorFlow Extended (TFX)
ML Pipelines with Airflow and KubeFlow
Agenda – Part 1 of 2
Hyper-Parameter Tuning with KubeFlow
Deploy Notebook with Kubernetes
2
3
4
5
37. 2.1 TFX Internals
2.0 TFX Components
6
Feature
Load
Feature
Analyze Feature
Transform
Model
Train
Model
Evaluate
Model
Deploy Reproduce
Training
2.2 TFX Libraries
2.2 TFX Components
38. 2.1 TFX Internals
Driver/Publisher
Moves data to/from Metadata Store
Executor
Runs the Actual Processing Code
Metadata Store
Artifact, execution, and lineage Info
Track inputs & outputs of all components
Stores training run including inputs & outputs
Analysis, validation, and versioning results
Feature
Load
Feature
Analyze Feature
Transform
Model
Train
Model
Evaluate
Model
Deploy Reproduce
Training
39. 2.2 TFX Libraries
2.2.1
TFX Components Use These:
TensorFlow Data Validation (TFDV)
TensorFlow Transform (TFT)
TensorFlow Model Analysis (TFMA)
TensorFlow Metadata (TFMD) + ML Metadata (MLMD)
2.2.2
2.2.3
2.2.4
Feature
Load
Feature
Analyze Feature
Transform
Model
Train
Model
Evaluate
Model
Deploy Reproduce
Training
40. 2.2.1 TFX Libraries - TFDV
TensorFlow Data Validation (TFDV)
Find Missing, Redundant & Important Features
Identify Features with Unusually-Large Scale
`infer_schema()` Generates Schema
Describe Feature Ranges
Detect Data Drift
Feature
Load
Feature
Analyze Feature
Transform
Model
Train
Model
Evaluate
Model
Deploy Reproduce
Training
Uniformly
Distributed Data è
ç Non-Uniformly
Distributed Data
42. 2.2.2 TFX Libraries - TFT
TensorFlow Transform (TFT)
Preprocess `tf.Example` data with TensorFlow
Useful for data that requires a full pass
Normalize all inputs by mean and std dev
Create vocabulary of strings è integers over all data
Bucketize features based on entire data distribution
Outputs a TensorFlow graph
Re-used across both training and serving
Uses Apache Beam (local mode) for Parallel Analysis
Can also use distributed mode
`preprocessing_fn(inputs)`: Primary Fn to Implement
Feature
Load
Feature
Analyze Feature
Transform
Model
Train
Model
Evaluate
Model
Deploy Reproduce
Training
import tensorflow as tf
import tensorflow_transform as tft
def preprocessing_fn(inputs):
x = inputs['x']
y = inputs['y']
s = inputs['s']
x_centered = x - tft.mean(x)
y_normalized = tft.scale_to_0_1(y)
s_integerized = tft.compute_and_apply_vocabulary(s)
x_centered_times_y_normalized = x_centered * y_normalized
return {
'x_centered': x_centered,
'y_normalized': y_normalized,
'x_centered_times_y_normalized':x_centered_times_y_normalized,
's_integerized': s_integerized
}
45. 2.2.3 TFX Libraries - TFMA
TensorFlow Model Analysis (TFMA)
Analyze Model on Different Slices of Dataset
Track Metrics Over Time (“Next Day Eval”)
`EvalSavedModel` Contains Slicing Info
TFMA Pipeline: Read, Extract, Evaluate, Write
ie. Ensure Model Works Fairly Across All Users
Feature
Load
Feature
Analyze Feature
Transform
Model
Train
Model
Evaluate
Model
Deploy Reproduce
Training
47. 2.2.4 TFX Libraries – Metadata
TensorFlow Metadata (TFMD)
ML Metadata (MLMD)
Record and Retrieve Experiment Metadata
Artifact, Execution, and Lineage Info
Track Inputs / Outputs of All TFX Components
Stores Training Run Info
Analysis and Validation Results
Model Versioning Info
Feature
Load
Feature
Analyze Feature
Transform
Model
Train
Model
Evaluate
Model
Deploy Reproduce
Training
49. 2.3.1 ExampleGen
Feature
Load
Feature
Analyze Feature
Transform
Model
Train
Model
Evaluate
Model
Deploy Reproduce
Training
Load Training Data Into TFX Pipeline
Supports External Data Sources
Supports CSV and TFRecord Formats
Converts Data to tf.Example
Note: TFX Pipelines require tf.Example (?!)
Difficult to use non-TF models like XGBoost
from tfx.utils.dsl_utils import csv_input
from
tfx.components.example_gen.csv_example_gen.component
import CsvExampleGen
examples = csv_input(os.path.join(base_dir, 'data/simple'))
example_gen = CsvExampleGen(input_base=examples)
51. 2.3.3 SchemaGen
Feature
Load
Feature
Analyze Feature
Transform
Model
Train
Model
Evaluate
Model
Deploy Reproduce
Training
Schema Needed by Some TFX Components
Data Types, Value Ranges, Optional, Required
Consumes Data from StatisticsGen
Schema used by TFDV, TFT, TFMA Libraries
Uses TFDV Library to infer schema
Best effort and basic
Human should verify
feature {
name: "age"
value_count {
min: 1
max: 1
}
type: FLOAT
presence {
min_fraction: 1
min_count: 1
}
}
from tfx import components
infer_schema = components.SchemaGen(
stats=compute_training_stats.outputs.output)
53. 2.3.5 Transform
Feature
Load
Feature
Analyze Feature
Transform
Model
Train
Model
Evaluate
Model
Deploy Reproduce
Training
Uses Data from ExampleGen & SchemaGen
Transformations Become Part of TF Graph (!!)
Helps Avoid Training/Serving Skew
Uses TFT Library for Transformations
Transformations Require Full Pass Thru Dataset
Global Reduction Across All Batches
Create Word Embeddings, Normalize, PCA
def preprocessing_fn(inputs):
# inputs: map from feature keys
# to raw not-yet-transformed features
# outputs: map from string feature key
# to transformed feature operations
54. 2.3.6 Trainer
Feature
Load
Feature
Analyze Feature
Transform
Model
Train
Model
Evaluate
Model
Deploy Reproduce
Training
Trains / Validates tf.Examples from Transform
Uses schema.proto from SchemaGen
Produces SavedModel and EvalSavedModel
Uses Core TensorFlow Python API
Works with TensorFlow 1.x Estimator API
TensorFlow 2.0 Keras Support Coming Soon
from tfx import components
trainer = components.Trainer(
module_file=taxi_pipeline_utils,
train_files=transform_training.outputs.output,
eval_files=transform_eval.outputs.output,
schema=infer_schema.outputs.output,
tf_transform_dir=transform_training.outputs.output,
train_steps=10000,
eval_steps=5000)
55. 2.3.7 Evaluator
Feature
Load
Feature
Analyze Feature
Transform
Model
Train
Model
Evaluate
Model
Deploy Reproduce
Training
Uses EvalSavedModel from Trainer
Writes Analysis Results to ML Metadata Store
Uses TFMA Library for Analysis
TFMA Uses Apache Beam to Scale Analysis
from tfx import components
import tensorflow_model_analysis as tfma
taxi_eval_spec = [
tfma.SingleSliceSpec(),
tfma.SingleSliceSpec(columns=['trip_start_hour'])
]
model_analyzer = components.Evaluator(
examples=examples_gen.outputs.eval_examples,
eval_spec=taxi_eval_spec,
model_exports=trainer.outputs.output)
56. 2.3.8 ModelValidator
Feature
Load
Feature
Analyze Feature
Transform
Model
Train
Model
Evaluate
Model
Deploy Reproduce
Training
Validate Models from Trainer
Uses Data from SchemaGen & StatisticsGen
Compares New Models to Baseline
Baseline == current model in production
New Model is Good if Meets/Exceeds Metrics
If Good, Notify Pusher to Deploy New Model
Simulate “Next Day Evaluation” On New Data
import tensorflow_model_analysis as tfma
taxi_mv_spec = [tfma.SingleSliceSpec()]
model_validator = components.ModelValidator(
examples=examples_gen.outputs.output,
model=trainer.outputs.output)
57. 2.3.9 Model Pusher (Deployer)
Feature
Load
Feature
Analyze Feature
Transform
Model
Train
Model
Evaluate
Model
Deploy Reproduce
Training
Push Good Model to Deployment Target
Uses Trained SavedModel
Writes Version Data to Metadata Store
Write to FileSystem or TensorFlow Hub
from tfx import components
pusher = components.Pusher(
model_export=trainer.outputs.output,
model_blessing=model_validator.outputs.blessing,
serving_model_dir=serving_model_dir)
58. 2.3.10 Slack Component (!!)
Feature
Load
Feature
Analyze Feature
Transform
Model
Train
Model
Evaluate
Model
Deploy Reproduce
Training
Runs After ModelValidator
Adds Human-in-the-Loop Step to Pipeline
TFX Sends Message to Slack with Model URI
Asks Human to Review the New Model
Respond ‘LGTM’, ‘approve’, ‘decline’, ‘reject’
Requires Slack API Setup / Integration
export SLACK_BOT_TOKEN={your_token}
_channel_id = 'my-channel-id'
_slack_token = os.environ['SLACK_BOT_TOKEN’]
slack_validator = SlackComponent(
model_export=trainer.outputs.output,
model_blessing=model_validator.outputs.blessing,
slack_token=_slack_token, channel_id=_channel_id,
timeout_sec=3600, )
https://github.com/tensorflow/tfx/tree/master
/tfx/examples/custom_components/slack/slack_component
59. 1 Setup Environment with Kubernetes
TensorFlow Extended (TFX)
ML Pipelines with Airflow and KubeFlow
Agenda – Part 1 of 2
Hyper-Parameter Tuning with KubeFlow
Deploy Notebook with Kubernetes
2
3
4
5
60. 3.0 ML Pipelines with Airflow and KubeFlow
Feature
Load
Feature
Analyze Feature
Transform
Model
Train
Model
Evaluate
Model
Deploy
3.1 Airflow
KubeFlow3.2
70. 1 Setup Environment with Kubernetes
TensorFlow Extended (TFX)
ML Pipelines with Airflow and KubeFlow
Agenda – Part 1 of 2
Hyper-Parameter Tuning with KubeFlow
Deploy Notebook with Kubernetes
2
3
4
5
71. 4.0 Hyper-Parameter Tuning
6
Experiment
Single Optimization Run
Single Objective Function Across Runs
Contains Many Trials
Trial
List of Param Values
Suggestion
Optimization Algorithm
Job
Evaluates a Trial
Calculates Objective
73. 1 Setup Environment with Kubernetes
TensorFlow Extended (TFX)
ML Pipelines with Airflow and KubeFlow
Agenda – Part 1 of 2
Hyper-Parameter Tuning with KubeFlow
Deploy Notebook with Kubernetes
2
3
4
5
74. 5.0 Deploy Notebook as Job
Feature
Load
Feature
Analyze Feature
Transform
Model
Train
Model
Evaluate
Model
Deploy Reproduce
Training
6
5.1 Wrap Model in a Docker Image
Deploy Job to Kubernetes5.2
75. 5.1 Create Docker Image
Feature
Load
Feature
Analyze Feature
Transform
Model
Train
Model
Evaluate
Model
Deploy Reproduce
Training
76. 5.2 Deploy Notebook as Job
Feature
Load
Feature
Analyze Feature
Transform
Model
Train
Model
Evaluate
Model
Deploy Reproduce
Training
79. 1 Setup Environment with Kubernetes
TensorFlow Extended (TFX)
ML Pipelines with TFX, Airflow, and KubeFlow
Agenda – Part 1 of 2
Hyper-Parameter Tuning with TFX and KubeFlow
Deploy Notebook with Kubernetes
2
3
4
5
80. Agenda – Part 2 of 2
6 MLflow
7
8
9
10
TensorFlow Privacy
Model Serving & A/B Tests
Model Optimization
Papermill
101. Agenda – Part 2 of 2
6 MLflow
7
8
9
10
TensorFlow Privacy
Model Serving & A/B Tests
Model Optimization
Papermill
102. 1 Setup Environment with Kubernetes
TensorFlow Extended (TFX)
ML Pipelines with Airflow and KubeFlow
Agenda – Part 1 of 2
Hyper-Parameter Tuning with KubeFlow
Deploy Notebook with Kubernetes
2
3
4
5
103. After Pushing Your Model to Production, Your Model is…
1 Already Out of Date – Need to Re-Train
Biased – Need to Validate Before Pushing
Broken – Need to A/B Test in Production
Hacked – Need to Train With Data Privacy
Slow – Need to Quantize and Speed Up Predictions
2
3
4
5