Discusses what to consider when writing a facial recognition application and how to scale it on multiple nodes using Spark. The approach discusses tools like OpenCV and dlib for traditional approaches and Tensorflow for inference to create embeddings\features.

1. 1© Cloudera, Inc. All rights reserved.
Presenting
Jason Hubbard
Facial Recognition at Scale

2. 2© Cloudera, Inc. All rights reserved.
Definitions
• Face Detection - Determine location of face within an image. Outputs
coordinates of detected face.
• Facial Landmarks - Determine location of landmarks (nose, chin, mouth),
typically given a single face.
• Face Recognition - Given a face classify based on previous labelled faces.

3. 3© Cloudera, Inc. All rights reserved.
Ingestion

4. 4© Cloudera, Inc. All rights reserved.
Hadoop Image Processing Interface - HIPI
• Native Image format
• Hooks into OpenCV
• Mostly dormant

5. 5© Cloudera, Inc. All rights reserved.
SequenceFile
• Well known solution, easy
• Key is filename, value is image blob
• Requires processing to extract image metadata

6. 6© Cloudera, Inc. All rights reserved.
AvroParquet
• Well known, a bit more work
• Can have multiple attributes including file name, image metadata, and blob of
image
• Can filter and project image metadata without processing image
• ImageSchema support added in Spark 2.3

7. 7© Cloudera, Inc. All rights reserved.
ToolsLibraries

8. 8© Cloudera, Inc. All rights reserved.
OpenCV
• Computer Vision Library from Intel
• C++, Java, Python
• Easy, mature product
• Multiple Models
• Train and Load Simple Models
• Load DNN, CaffeeTensorflow
• Image processing tools
• Loading, Saving, Converting
• Frame Grabber
• Image Translation

9. 9© Cloudera, Inc. All rights reserved.
JavaCV
• Project within large JavaCPP Project
• Add other components like ffmpeg, tesseract, mkl, etc
• Easy wrappers and helper methods
• Prebuilt & cross compiled (Maven Central)
• Windows, Linux, ARM, Android, etc
• May not be optimized

10. 10© Cloudera, Inc. All rights reserved.
dlib
• C++ Library
• Python bindings
• Not as easy
• Requires compiling
• Python requires native code
• Multiple Models
• Load and Train simple and DNN

11. 11© Cloudera, Inc. All rights reserved.
Deep Learning
• Caffe
• C++, Python bindings
• Originally focused on vision, but general deep learning now
• Train, Inference, export model and weights
• TensorFlow
• C++, Python and Java bindings
• General Deep learning library
• Train, Inference, export model and weights
• Generally missing image processing

12. 12© Cloudera, Inc. All rights reserved.
Preprocessing

13. 13© Cloudera, Inc. All rights reserved.
Normalize
• Detect Face
• Landmarks
• Resize
• Translate

14. 14© Cloudera, Inc. All rights reserved.
Cascade Classifier
• Boosted cascade of
weak classifiers
• Object Detection
• Haar, Local Binary
Patterns (LBP)
• Examples in OpenCV,
original models
• Hard to tune
• Can have poor results

15. 15© Cloudera, Inc. All rights reserved.
Histogram of Oriented Gradients (HOG)
• Provided in dlib
• Extension, discriminately trained
• Better accuracy
• SVM Model

16. 16© Cloudera, Inc. All rights reserved.
Multi-task CNN
• Caffe & Tensorflow models
• Examples exist, but complex
• Slower
• Good results
• Better at occlusion, silhouettes
• Combined Face Detection and Landmark

17. 17© Cloudera, Inc. All rights reserved.
Training

18. 18© Cloudera, Inc. All rights reserved.
Face Recognizer
• Eigenfaces – PCA to limit dimensionality, greatest variance
• Fisherfaces – LDA to increase discriminate features
• Local Binary Patterns Histograms (LBPH) – Local features instead of holistic
• Influenced by external sources (light, occlusion, etc)

19. 19© Cloudera, Inc. All rights reserved.
Facenet
• Inception-ResNet-v1 model
• Creates embeddings
• Discriminative features using center loss
• Train against embeddings
• Clustering
• SVM

20. 20© Cloudera, Inc. All rights reserved.
Embedding
[-1.234, -7.32, 9.323, ….]

21. 21© Cloudera, Inc. All rights reserved.
Flow
[3.23, -1.43, 5.23,
…][-3.432, 0.723, -2.34,
…][2.12, -1.43, 5.23,
…]
[-1.23, 1.43, 3.23,
…][-2.432, 1.723, 1.44,
…][2.432, -0.723, 2.14,
…]
[0.23, -1.73, 6.23,
…][-4.432, 2.32, 1.84,
…][3.12, -2.23, 7.64, …]
[-4.23, 0.13, 9.23,
…][-2.432, 1.723, 1.44,
…][1.432, 2.723, -3.44,
…]
Clustering
Or
Regression
MTCNN -> Face &
Landmark Locations
Crop, Scale,
Translate
Facenet ->
Embeddings
Decode

22. 22© Cloudera, Inc. All rights reserved.
TensorFlow Distributed Deep Learning
• Spark
• Tensorframes
• deep-learning-pipelines
• TensorflowOnSpark
• Import into alternative (dl4j, BigDL, etc)
• DIY

23. 23© Cloudera, Inc. All rights reserved.
Beer
Protection

24. 24© Cloudera, Inc. All rights reserved.
Kegbot
• Device to keep track of pours
• Associates poor to user
• Optionally takes photo from Android tablet
• Stored in MySQL database

25. 25© Cloudera, Inc. All rights reserved.
Case of missing pourers
• Claim Pours
• Unclaimed
• Wrong selection
• Tablet can go offline

26. 26© Cloudera, Inc. All rights reserved.
Solution
• Batch process photos
• Photos saved on filesystem
• Database stores photo to pour mapping
• Use external camera
• OpenCV frame grabber
• Spark receiver store frame/image

27. 27© Cloudera, Inc. All rights reserved.
Single Image

28. 28© Cloudera, Inc. All rights reserved.
Approaches
• Synthesis Data
• Jitter
• Shading
• Translate, Flip
• Take multiple of unknown subject
• Euclidean Distance

29. 29© Cloudera, Inc. All rights reserved.
Thank you
Jason Hubbard