SlideShare una empresa de Scribd logo
1 de 57
Descargar para leer sin conexión
Introduction Old Architecture New Architecture Decoupling Streaming Conclusion
1
2
Introduction Old Architecture New Architecture Decoupling Streaming Conclusion
● Introduction
● Old Architecture
● New Architecture
● Decoupling
● Streaming
● Conclusion
3
Introduction Old Architecture New Architecture Decoupling Streaming Conclusion
● Legacy Java Process
○ “Crunches” data
○ Sends data downstream to our own datastores and to 3rd party
analytics
○ Runs every hour
● Growth
○ Process can run over an hour
○ 12 GB -> 24GB heap in less than 1 year
○ Cron is a horrible job management system
○ A failure requires rerunning a job from the beginning
● 2.0
○ Horizontably scalable
○ Real Time ETL
○ Reuesable
4
Introduction Old Architecture New Architecture Decoupling Streaming Conclusion
ETL @ Vungle
● ~1 Billion Events / Day
● Deduplication
● Calculating $$$
● Outputting data to various destinations
5
Introduction Old Architecture New Architecture Decoupling Streaming Conclusion
Old Architecture
6
Introduction Old Architecture New Architecture Decoupling Streaming Conclusion
7
Introduction Old Architecture New Architecture Decoupling Streaming Conclusion
8
Introduction Old Architecture New Architecture Decoupling Streaming Conclusion
9
Introduction Old Architecture New Architecture Decoupling Streaming Conclusion
10
Introduction Old Architecture New Architecture Decoupling Streaming Conclusion
11
Introduction Old Architecture New Architecture Decoupling Streaming Conclusion
12
Introduction Old Architecture New Architecture Decoupling Streaming Conclusion
13
Introduction Old Architecture New Architecture Decoupling Streaming Conclusion
14
Introduction Old Architecture New Architecture Decoupling Streaming Conclusion
New Architecture
15
Introduction Old Architecture New Architecture Decoupling Streaming Conclusion
16
Introduction Old Architecture New Architecture Decoupling Streaming Conclusion
17
Introduction Old Architecture New Architecture Decoupling Streaming Conclusion
18
Introduction Old Architecture New Architecture Decoupling Streaming Conclusion
19
Introduction Old Architecture New Architecture Decoupling Streaming Conclusion
20
Introduction Old Architecture New Architecture Decoupling Streaming Conclusion
21
Introduction Old Architecture New Architecture Decoupling Streaming Conclusion
22
Introduction Old Architecture New Architecture Decoupling Streaming Conclusion
Decoupling
23
Introduction Old Architecture New Architecture Decoupling Streaming Conclusion
24
Introduction Old Architecture New Architecture Decoupling Streaming Conclusion
25
Introduction Old Architecture New Architecture Decoupling Streaming Conclusion
26
Introduction Old Architecture New Architecture Decoupling Streaming Conclusion
27
Introduction Old Architecture New Architecture Decoupling Streaming Conclusion
28
Introduction Old Architecture New Architecture Decoupling Streaming Conclusion
29
Introduction Old Architecture New Architecture Decoupling Streaming Conclusion
30
Introduction Old Architecture New Architecture Decoupling Streaming Conclusion
31
Introduction Problem Decoupling Streaming Conclusion
Setup connection and spark streams
Map each line of log into Mongo Objects
and insert into mongo
32
Introduction Old Architecture New Architecture Decoupling Streaming Conclusion
Setup connection and spark streams
33
Introduction Old Architecture New Architecture Decoupling Streaming Conclusion
Mapping to Mongo objects and insertions
34
Introduction Old Architecture New Architecture Decoupling Streaming Conclusion
Questions
35
Introduction Old Architecture New Architecture Decoupling Streaming Conclusion
Streaming
36
Introduction Old Architecture New Architecture Decoupling Streaming Conclusion
37
Introduction Old Architecture New Architecture Decoupling Streaming Conclusion
38
Introduction Old Architecture New Architecture Decoupling Streaming Conclusion
39
Introduction Old Architecture New Architecture Decoupling Streaming Conclusion
Ingestion
40
Introduction Old Architecture New Architecture Decoupling Streaming Conclusion
Event ID Request View Install ... Request
Added
View
Added
Install
Added
Value
Ingestion Table Schema
41
Introduction Old Architecture New Architecture Decoupling Streaming Conclusion
... Date Time Deliveries Views Installs Processed
Deliveries
Processed
Views
Processed
Installs
Fact Table Schema
42
Introduction Old Architecture New Architecture Decoupling Streaming Conclusion
Ingestion
43
Introduction Old Architecture New Architecture Decoupling Streaming Conclusion
44
Introduction Old Architecture New Architecture Decoupling Streaming Conclusion
45
Introduction Old Architecture New Architecture Decoupling Streaming Conclusion
46
Introduction Old Architecture New Architecture Decoupling Streaming Conclusion
47
Introduction Old Architecture New Architecture Decoupling Streaming Conclusion
48
Introduction Old Architecture New Architecture Decoupling Streaming Conclusion
49
Introduction Old Architecture New Architecture Decoupling Streaming Conclusion
Process
50
Introduction Old Architecture New Architecture Decoupling Streaming Conclusion
51
Introduction Old Architecture New Architecture Decoupling Streaming Conclusion
52
Introduction Old Architecture New Architecture Decoupling Streaming Conclusion
53
Introduction Old Architecture New Architecture Decoupling Streaming Conclusion
54
Introduction Old Architecture New Architecture Decoupling Streaming Conclusion
55
Introduction Old Architecture New Architecture Decoupling Streaming Conclusion
Next Steps
● Switching from JSON to ProtoBuf
● Using YARN to run multiple jobs on one cluster
● Data Science
● Who knows?
56
Introduction Old Architecture New Architecture Decoupling Streaming Conclusion
Questions
Thank you!
57

Más contenido relacionado

La actualidad más candente

Counting Elements in Streams
Counting Elements in StreamsCounting Elements in Streams
Counting Elements in StreamsJamie Grier
 
Hadoop summit - Scaling Uber’s Real-Time Infra for Trillion Events per Day
Hadoop summit - Scaling Uber’s Real-Time Infra for  Trillion Events per DayHadoop summit - Scaling Uber’s Real-Time Infra for  Trillion Events per Day
Hadoop summit - Scaling Uber’s Real-Time Infra for Trillion Events per DayAnkur Bansal
 
Flink Forward Berlin 2017: Mihail Vieru - A Materialization Engine for Data I...
Flink Forward Berlin 2017: Mihail Vieru - A Materialization Engine for Data I...Flink Forward Berlin 2017: Mihail Vieru - A Materialization Engine for Data I...
Flink Forward Berlin 2017: Mihail Vieru - A Materialization Engine for Data I...Flink Forward
 
Jamie Grier - Robust Stream Processing with Apache Flink
Jamie Grier - Robust Stream Processing with Apache FlinkJamie Grier - Robust Stream Processing with Apache Flink
Jamie Grier - Robust Stream Processing with Apache FlinkFlink Forward
 
Flink in Zalando's world of Microservices
Flink in Zalando's world of Microservices   Flink in Zalando's world of Microservices
Flink in Zalando's world of Microservices ZalandoHayley
 
Stream Processing using Apache Flink in Zalando's World of Microservices - Re...
Stream Processing using Apache Flink in Zalando's World of Microservices - Re...Stream Processing using Apache Flink in Zalando's World of Microservices - Re...
Stream Processing using Apache Flink in Zalando's World of Microservices - Re...Zalando Technology
 
Gyula Fóra - RBEA- Scalable Real-Time Analytics at King
Gyula Fóra - RBEA- Scalable Real-Time Analytics at KingGyula Fóra - RBEA- Scalable Real-Time Analytics at King
Gyula Fóra - RBEA- Scalable Real-Time Analytics at KingFlink Forward
 
Streaming in the Wild with Apache Flink
Streaming in the Wild with Apache FlinkStreaming in the Wild with Apache Flink
Streaming in the Wild with Apache FlinkKostas Tzoumas
 
Virtual Flink Forward 2020: Production-Ready Flink and Hive Integration - wha...
Virtual Flink Forward 2020: Production-Ready Flink and Hive Integration - wha...Virtual Flink Forward 2020: Production-Ready Flink and Hive Integration - wha...
Virtual Flink Forward 2020: Production-Ready Flink and Hive Integration - wha...Flink Forward
 
Apache Flink Community Updates November 2016 @ Berlin Meetup
Apache Flink Community Updates November 2016 @ Berlin MeetupApache Flink Community Updates November 2016 @ Berlin Meetup
Apache Flink Community Updates November 2016 @ Berlin MeetupRobert Metzger
 
Robust Stream Processing With Apache Flink
Robust Stream Processing With Apache FlinkRobust Stream Processing With Apache Flink
Robust Stream Processing With Apache FlinkJamie Grier
 
Flink Forward San Francisco 2018: Gregory Fee - "Bootstrapping State In Apach...
Flink Forward San Francisco 2018: Gregory Fee - "Bootstrapping State In Apach...Flink Forward San Francisco 2018: Gregory Fee - "Bootstrapping State In Apach...
Flink Forward San Francisco 2018: Gregory Fee - "Bootstrapping State In Apach...Flink Forward
 
Fabian Hueske_Till Rohrmann - Declarative stream processing with StreamSQL an...
Fabian Hueske_Till Rohrmann - Declarative stream processing with StreamSQL an...Fabian Hueske_Till Rohrmann - Declarative stream processing with StreamSQL an...
Fabian Hueske_Till Rohrmann - Declarative stream processing with StreamSQL an...Flink Forward
 
Apache Flink @ Alibaba - Seattle Apache Flink Meetup
Apache Flink @ Alibaba - Seattle Apache Flink MeetupApache Flink @ Alibaba - Seattle Apache Flink Meetup
Apache Flink @ Alibaba - Seattle Apache Flink MeetupBowen Li
 
Bitsy graph database
Bitsy graph databaseBitsy graph database
Bitsy graph databaseLambdaZen LLC
 
Thomas Lamirault_Mohamed Amine Abdessemed -A brief history of time with Apac...
Thomas Lamirault_Mohamed Amine Abdessemed  -A brief history of time with Apac...Thomas Lamirault_Mohamed Amine Abdessemed  -A brief history of time with Apac...
Thomas Lamirault_Mohamed Amine Abdessemed -A brief history of time with Apac...Flink Forward
 
The Past, Present, and Future of Apache Flink®
The Past, Present, and Future of Apache Flink®The Past, Present, and Future of Apache Flink®
The Past, Present, and Future of Apache Flink®Aljoscha Krettek
 
Flink Forward Berlin 2017: Zohar Mizrahi - Python Streaming API
Flink Forward Berlin 2017: Zohar Mizrahi - Python Streaming APIFlink Forward Berlin 2017: Zohar Mizrahi - Python Streaming API
Flink Forward Berlin 2017: Zohar Mizrahi - Python Streaming APIFlink Forward
 
Achieving end-to-end visibility into complex event-sourcing transactions usin...
Achieving end-to-end visibility into complex event-sourcing transactions usin...Achieving end-to-end visibility into complex event-sourcing transactions usin...
Achieving end-to-end visibility into complex event-sourcing transactions usin...HostedbyConfluent
 
Apache Beam @ GCPUG.TW Flink.TW 20161006
Apache Beam @ GCPUG.TW Flink.TW 20161006Apache Beam @ GCPUG.TW Flink.TW 20161006
Apache Beam @ GCPUG.TW Flink.TW 20161006Randy Huang
 

La actualidad más candente (20)

Counting Elements in Streams
Counting Elements in StreamsCounting Elements in Streams
Counting Elements in Streams
 
Hadoop summit - Scaling Uber’s Real-Time Infra for Trillion Events per Day
Hadoop summit - Scaling Uber’s Real-Time Infra for  Trillion Events per DayHadoop summit - Scaling Uber’s Real-Time Infra for  Trillion Events per Day
Hadoop summit - Scaling Uber’s Real-Time Infra for Trillion Events per Day
 
Flink Forward Berlin 2017: Mihail Vieru - A Materialization Engine for Data I...
Flink Forward Berlin 2017: Mihail Vieru - A Materialization Engine for Data I...Flink Forward Berlin 2017: Mihail Vieru - A Materialization Engine for Data I...
Flink Forward Berlin 2017: Mihail Vieru - A Materialization Engine for Data I...
 
Jamie Grier - Robust Stream Processing with Apache Flink
Jamie Grier - Robust Stream Processing with Apache FlinkJamie Grier - Robust Stream Processing with Apache Flink
Jamie Grier - Robust Stream Processing with Apache Flink
 
Flink in Zalando's world of Microservices
Flink in Zalando's world of Microservices   Flink in Zalando's world of Microservices
Flink in Zalando's world of Microservices
 
Stream Processing using Apache Flink in Zalando's World of Microservices - Re...
Stream Processing using Apache Flink in Zalando's World of Microservices - Re...Stream Processing using Apache Flink in Zalando's World of Microservices - Re...
Stream Processing using Apache Flink in Zalando's World of Microservices - Re...
 
Gyula Fóra - RBEA- Scalable Real-Time Analytics at King
Gyula Fóra - RBEA- Scalable Real-Time Analytics at KingGyula Fóra - RBEA- Scalable Real-Time Analytics at King
Gyula Fóra - RBEA- Scalable Real-Time Analytics at King
 
Streaming in the Wild with Apache Flink
Streaming in the Wild with Apache FlinkStreaming in the Wild with Apache Flink
Streaming in the Wild with Apache Flink
 
Virtual Flink Forward 2020: Production-Ready Flink and Hive Integration - wha...
Virtual Flink Forward 2020: Production-Ready Flink and Hive Integration - wha...Virtual Flink Forward 2020: Production-Ready Flink and Hive Integration - wha...
Virtual Flink Forward 2020: Production-Ready Flink and Hive Integration - wha...
 
Apache Flink Community Updates November 2016 @ Berlin Meetup
Apache Flink Community Updates November 2016 @ Berlin MeetupApache Flink Community Updates November 2016 @ Berlin Meetup
Apache Flink Community Updates November 2016 @ Berlin Meetup
 
Robust Stream Processing With Apache Flink
Robust Stream Processing With Apache FlinkRobust Stream Processing With Apache Flink
Robust Stream Processing With Apache Flink
 
Flink Forward San Francisco 2018: Gregory Fee - "Bootstrapping State In Apach...
Flink Forward San Francisco 2018: Gregory Fee - "Bootstrapping State In Apach...Flink Forward San Francisco 2018: Gregory Fee - "Bootstrapping State In Apach...
Flink Forward San Francisco 2018: Gregory Fee - "Bootstrapping State In Apach...
 
Fabian Hueske_Till Rohrmann - Declarative stream processing with StreamSQL an...
Fabian Hueske_Till Rohrmann - Declarative stream processing with StreamSQL an...Fabian Hueske_Till Rohrmann - Declarative stream processing with StreamSQL an...
Fabian Hueske_Till Rohrmann - Declarative stream processing with StreamSQL an...
 
Apache Flink @ Alibaba - Seattle Apache Flink Meetup
Apache Flink @ Alibaba - Seattle Apache Flink MeetupApache Flink @ Alibaba - Seattle Apache Flink Meetup
Apache Flink @ Alibaba - Seattle Apache Flink Meetup
 
Bitsy graph database
Bitsy graph databaseBitsy graph database
Bitsy graph database
 
Thomas Lamirault_Mohamed Amine Abdessemed -A brief history of time with Apac...
Thomas Lamirault_Mohamed Amine Abdessemed  -A brief history of time with Apac...Thomas Lamirault_Mohamed Amine Abdessemed  -A brief history of time with Apac...
Thomas Lamirault_Mohamed Amine Abdessemed -A brief history of time with Apac...
 
The Past, Present, and Future of Apache Flink®
The Past, Present, and Future of Apache Flink®The Past, Present, and Future of Apache Flink®
The Past, Present, and Future of Apache Flink®
 
Flink Forward Berlin 2017: Zohar Mizrahi - Python Streaming API
Flink Forward Berlin 2017: Zohar Mizrahi - Python Streaming APIFlink Forward Berlin 2017: Zohar Mizrahi - Python Streaming API
Flink Forward Berlin 2017: Zohar Mizrahi - Python Streaming API
 
Achieving end-to-end visibility into complex event-sourcing transactions usin...
Achieving end-to-end visibility into complex event-sourcing transactions usin...Achieving end-to-end visibility into complex event-sourcing transactions usin...
Achieving end-to-end visibility into complex event-sourcing transactions usin...
 
Apache Beam @ GCPUG.TW Flink.TW 20161006
Apache Beam @ GCPUG.TW Flink.TW 20161006Apache Beam @ GCPUG.TW Flink.TW 20161006
Apache Beam @ GCPUG.TW Flink.TW 20161006
 

Destacado

China for the Win! What Publishers Need to Know to Succeed in this Emerging M...
China for the Win! What Publishers Need to Know to Succeed in this Emerging M...China for the Win! What Publishers Need to Know to Succeed in this Emerging M...
China for the Win! What Publishers Need to Know to Succeed in this Emerging M...Vungle
 
Woundary 서비스 활용안 vine 130527_석혜윤
Woundary 서비스 활용안 vine 130527_석혜윤Woundary 서비스 활용안 vine 130527_석혜윤
Woundary 서비스 활용안 vine 130527_석혜윤Hyeyoon Seok
 
Going the extra mile on social media: moving from 1.0 to 2.0
Going the extra mile on social media: moving from 1.0 to 2.0Going the extra mile on social media: moving from 1.0 to 2.0
Going the extra mile on social media: moving from 1.0 to 2.0Scopernia
 
Designed to Win: How to Monetize Users and Enhance Experience in Your Game
Designed to Win: How to Monetize Users and Enhance Experience in Your Game Designed to Win: How to Monetize Users and Enhance Experience in Your Game
Designed to Win: How to Monetize Users and Enhance Experience in Your Game Vungle
 
Product (Experience) Management
Product (Experience) ManagementProduct (Experience) Management
Product (Experience) ManagementPeter John Marquez
 
Genius Strategies for Engaging Followers through Social Media
Genius Strategies for Engaging Followers through Social MediaGenius Strategies for Engaging Followers through Social Media
Genius Strategies for Engaging Followers through Social MediaJennifer Jones
 
Mobile Recruiting Best Practices
Mobile Recruiting Best PracticesMobile Recruiting Best Practices
Mobile Recruiting Best PracticesWorkology
 
Digital Transformation of the Channel
Digital Transformation of the ChannelDigital Transformation of the Channel
Digital Transformation of the ChannelScopernia
 
Social media around the world 2011
Social media around the world 2011Social media around the world 2011
Social media around the world 2011Steven Van Belleghem
 
Node.js and The Internet of Things
Node.js and The Internet of ThingsNode.js and The Internet of Things
Node.js and The Internet of ThingsLosant
 
Satyapriya rajguru: Every day, in one way or another.
Satyapriya  rajguru:  Every day, in one way or another.Satyapriya  rajguru:  Every day, in one way or another.
Satyapriya rajguru: Every day, in one way or another.Satyapriya Rajguru
 
Connecting With the Disconnected
Connecting With the DisconnectedConnecting With the Disconnected
Connecting With the DisconnectedChris Wejr
 
How to Become a Thought Leader in Your Niche
How to Become a Thought Leader in Your NicheHow to Become a Thought Leader in Your Niche
How to Become a Thought Leader in Your NicheLeslie Samuel
 

Destacado (16)

China for the Win! What Publishers Need to Know to Succeed in this Emerging M...
China for the Win! What Publishers Need to Know to Succeed in this Emerging M...China for the Win! What Publishers Need to Know to Succeed in this Emerging M...
China for the Win! What Publishers Need to Know to Succeed in this Emerging M...
 
Woundary 서비스 활용안 vine 130527_석혜윤
Woundary 서비스 활용안 vine 130527_석혜윤Woundary 서비스 활용안 vine 130527_석혜윤
Woundary 서비스 활용안 vine 130527_석혜윤
 
Going the extra mile on social media: moving from 1.0 to 2.0
Going the extra mile on social media: moving from 1.0 to 2.0Going the extra mile on social media: moving from 1.0 to 2.0
Going the extra mile on social media: moving from 1.0 to 2.0
 
Designed to Win: How to Monetize Users and Enhance Experience in Your Game
Designed to Win: How to Monetize Users and Enhance Experience in Your Game Designed to Win: How to Monetize Users and Enhance Experience in Your Game
Designed to Win: How to Monetize Users and Enhance Experience in Your Game
 
Product (Experience) Management
Product (Experience) ManagementProduct (Experience) Management
Product (Experience) Management
 
Genius Strategies for Engaging Followers through Social Media
Genius Strategies for Engaging Followers through Social MediaGenius Strategies for Engaging Followers through Social Media
Genius Strategies for Engaging Followers through Social Media
 
Mobile Recruiting Best Practices
Mobile Recruiting Best PracticesMobile Recruiting Best Practices
Mobile Recruiting Best Practices
 
Digital Transformation of the Channel
Digital Transformation of the ChannelDigital Transformation of the Channel
Digital Transformation of the Channel
 
Social media around the world 2011
Social media around the world 2011Social media around the world 2011
Social media around the world 2011
 
Node.js and The Internet of Things
Node.js and The Internet of ThingsNode.js and The Internet of Things
Node.js and The Internet of Things
 
Satyapriya rajguru: Every day, in one way or another.
Satyapriya  rajguru:  Every day, in one way or another.Satyapriya  rajguru:  Every day, in one way or another.
Satyapriya rajguru: Every day, in one way or another.
 
2015 US Global Mobile Consumer Survey
2015 US Global Mobile Consumer Survey2015 US Global Mobile Consumer Survey
2015 US Global Mobile Consumer Survey
 
THIRST
THIRSTTHIRST
THIRST
 
2016 Digital Yearbook
2016 Digital Yearbook2016 Digital Yearbook
2016 Digital Yearbook
 
Connecting With the Disconnected
Connecting With the DisconnectedConnecting With the Disconnected
Connecting With the Disconnected
 
How to Become a Thought Leader in Your Niche
How to Become a Thought Leader in Your NicheHow to Become a Thought Leader in Your Niche
How to Become a Thought Leader in Your Niche
 

Similar a Using Spark at Vungle

Etienne chauchot spark structured streaming runner
Etienne chauchot spark structured streaming runnerEtienne chauchot spark structured streaming runner
Etienne chauchot spark structured streaming runnerEtienne Chauchot
 
DesignCon-TF-THA2_final_21jan
DesignCon-TF-THA2_final_21janDesignCon-TF-THA2_final_21jan
DesignCon-TF-THA2_final_21janAshish Sirasao
 
LambdaFlow: Scala Functional Message Processing
LambdaFlow: Scala Functional Message Processing LambdaFlow: Scala Functional Message Processing
LambdaFlow: Scala Functional Message Processing John Nestor
 
Abstract Factory pattern application on multi-contract on-chain deployments
Abstract Factory pattern application on multi-contract on-chain deploymentsAbstract Factory pattern application on multi-contract on-chain deployments
Abstract Factory pattern application on multi-contract on-chain deploymentsDejan Radic
 
Continuous Delivery on Kubernetes Using Spinnaker
Continuous Delivery on Kubernetes Using SpinnakerContinuous Delivery on Kubernetes Using Spinnaker
Continuous Delivery on Kubernetes Using SpinnakerWSO2
 
OpenContrail Experience tcp cloud OpenStack Summit Tokyo
OpenContrail Experience tcp cloud OpenStack Summit TokyoOpenContrail Experience tcp cloud OpenStack Summit Tokyo
OpenContrail Experience tcp cloud OpenStack Summit TokyoJakub Pavlik
 
Flink Forward Berlin 2018: Paris Carbone - "Stream Loops on Flink: Reinventin...
Flink Forward Berlin 2018: Paris Carbone - "Stream Loops on Flink: Reinventin...Flink Forward Berlin 2018: Paris Carbone - "Stream Loops on Flink: Reinventin...
Flink Forward Berlin 2018: Paris Carbone - "Stream Loops on Flink: Reinventin...Flink Forward
 
Stream Loops on Flink - Reinventing the wheel for the streaming era
Stream Loops on Flink - Reinventing the wheel for the streaming eraStream Loops on Flink - Reinventing the wheel for the streaming era
Stream Loops on Flink - Reinventing the wheel for the streaming eraParis Carbone
 
Build a Bridge to Cloud with Apache Kafka® for Data Analytics Cloud Services
Build a Bridge to Cloud with Apache Kafka® for Data Analytics Cloud ServicesBuild a Bridge to Cloud with Apache Kafka® for Data Analytics Cloud Services
Build a Bridge to Cloud with Apache Kafka® for Data Analytics Cloud Servicesconfluent
 
Streaming SQL Foundations: Why I ❤ Streams+Tables
Streaming SQL Foundations: Why I ❤ Streams+TablesStreaming SQL Foundations: Why I ❤ Streams+Tables
Streaming SQL Foundations: Why I ❤ Streams+TablesC4Media
 
How to build an ETL pipeline with Apache Beam on Google Cloud Dataflow
How to build an ETL pipeline with Apache Beam on Google Cloud DataflowHow to build an ETL pipeline with Apache Beam on Google Cloud Dataflow
How to build an ETL pipeline with Apache Beam on Google Cloud DataflowLucas Arruda
 
TDC2017 | São Paulo - Trilha BigData How we figured out we had a SRE team at ...
TDC2017 | São Paulo - Trilha BigData How we figured out we had a SRE team at ...TDC2017 | São Paulo - Trilha BigData How we figured out we had a SRE team at ...
TDC2017 | São Paulo - Trilha BigData How we figured out we had a SRE team at ...tdc-globalcode
 
browserCloud.js - David Dias M.Sc Thesis Defense Deck
browserCloud.js - David Dias M.Sc Thesis Defense Deck browserCloud.js - David Dias M.Sc Thesis Defense Deck
browserCloud.js - David Dias M.Sc Thesis Defense Deck David Dias
 
2018 10-31 modern-http_routing-lisa18
2018 10-31 modern-http_routing-lisa182018 10-31 modern-http_routing-lisa18
2018 10-31 modern-http_routing-lisa18Sandor Szuecs
 
Cincom Roadmap ESUG2014
Cincom Roadmap ESUG2014Cincom Roadmap ESUG2014
Cincom Roadmap ESUG2014ESUG
 
Capella Days 2021 | A STEP towards Model-based: Case Study covering Conceptua...
Capella Days 2021 | A STEP towards Model-based: Case Study covering Conceptua...Capella Days 2021 | A STEP towards Model-based: Case Study covering Conceptua...
Capella Days 2021 | A STEP towards Model-based: Case Study covering Conceptua...Obeo
 
Evolution of AWS infrastructure for ML: from Zero to Hero
Evolution of AWS infrastructure for ML: from Zero to HeroEvolution of AWS infrastructure for ML: from Zero to Hero
Evolution of AWS infrastructure for ML: from Zero to HeroTaras Slipets
 

Similar a Using Spark at Vungle (20)

Etienne chauchot spark structured streaming runner
Etienne chauchot spark structured streaming runnerEtienne chauchot spark structured streaming runner
Etienne chauchot spark structured streaming runner
 
DesignCon-TF-THA2_final_21jan
DesignCon-TF-THA2_final_21janDesignCon-TF-THA2_final_21jan
DesignCon-TF-THA2_final_21jan
 
Undergraduate Portfolio
Undergraduate PortfolioUndergraduate Portfolio
Undergraduate Portfolio
 
LambdaFlow: Scala Functional Message Processing
LambdaFlow: Scala Functional Message Processing LambdaFlow: Scala Functional Message Processing
LambdaFlow: Scala Functional Message Processing
 
Abstract Factory pattern application on multi-contract on-chain deployments
Abstract Factory pattern application on multi-contract on-chain deploymentsAbstract Factory pattern application on multi-contract on-chain deployments
Abstract Factory pattern application on multi-contract on-chain deployments
 
Continuous Delivery on Kubernetes Using Spinnaker
Continuous Delivery on Kubernetes Using SpinnakerContinuous Delivery on Kubernetes Using Spinnaker
Continuous Delivery on Kubernetes Using Spinnaker
 
OpenContrail Experience tcp cloud OpenStack Summit Tokyo
OpenContrail Experience tcp cloud OpenStack Summit TokyoOpenContrail Experience tcp cloud OpenStack Summit Tokyo
OpenContrail Experience tcp cloud OpenStack Summit Tokyo
 
Flink Forward Berlin 2018: Paris Carbone - "Stream Loops on Flink: Reinventin...
Flink Forward Berlin 2018: Paris Carbone - "Stream Loops on Flink: Reinventin...Flink Forward Berlin 2018: Paris Carbone - "Stream Loops on Flink: Reinventin...
Flink Forward Berlin 2018: Paris Carbone - "Stream Loops on Flink: Reinventin...
 
Stream Loops on Flink - Reinventing the wheel for the streaming era
Stream Loops on Flink - Reinventing the wheel for the streaming eraStream Loops on Flink - Reinventing the wheel for the streaming era
Stream Loops on Flink - Reinventing the wheel for the streaming era
 
Build a Bridge to Cloud with Apache Kafka® for Data Analytics Cloud Services
Build a Bridge to Cloud with Apache Kafka® for Data Analytics Cloud ServicesBuild a Bridge to Cloud with Apache Kafka® for Data Analytics Cloud Services
Build a Bridge to Cloud with Apache Kafka® for Data Analytics Cloud Services
 
Streaming SQL Foundations: Why I ❤ Streams+Tables
Streaming SQL Foundations: Why I ❤ Streams+TablesStreaming SQL Foundations: Why I ❤ Streams+Tables
Streaming SQL Foundations: Why I ❤ Streams+Tables
 
How to build an ETL pipeline with Apache Beam on Google Cloud Dataflow
How to build an ETL pipeline with Apache Beam on Google Cloud DataflowHow to build an ETL pipeline with Apache Beam on Google Cloud Dataflow
How to build an ETL pipeline with Apache Beam on Google Cloud Dataflow
 
TDC2017 | São Paulo - Trilha BigData How we figured out we had a SRE team at ...
TDC2017 | São Paulo - Trilha BigData How we figured out we had a SRE team at ...TDC2017 | São Paulo - Trilha BigData How we figured out we had a SRE team at ...
TDC2017 | São Paulo - Trilha BigData How we figured out we had a SRE team at ...
 
Google Cloud Dataflow
Google Cloud DataflowGoogle Cloud Dataflow
Google Cloud Dataflow
 
browserCloud.js - David Dias M.Sc Thesis Defense Deck
browserCloud.js - David Dias M.Sc Thesis Defense Deck browserCloud.js - David Dias M.Sc Thesis Defense Deck
browserCloud.js - David Dias M.Sc Thesis Defense Deck
 
2018 10-31 modern-http_routing-lisa18
2018 10-31 modern-http_routing-lisa182018 10-31 modern-http_routing-lisa18
2018 10-31 modern-http_routing-lisa18
 
Cincom Roadmap ESUG2014
Cincom Roadmap ESUG2014Cincom Roadmap ESUG2014
Cincom Roadmap ESUG2014
 
Capella Days 2021 | A STEP towards Model-based: Case Study covering Conceptua...
Capella Days 2021 | A STEP towards Model-based: Case Study covering Conceptua...Capella Days 2021 | A STEP towards Model-based: Case Study covering Conceptua...
Capella Days 2021 | A STEP towards Model-based: Case Study covering Conceptua...
 
Evolution of AWS infrastructure for ML: from Zero to Hero
Evolution of AWS infrastructure for ML: from Zero to HeroEvolution of AWS infrastructure for ML: from Zero to Hero
Evolution of AWS infrastructure for ML: from Zero to Hero
 
Design a pipeline
Design a pipelineDesign a pipeline
Design a pipeline
 

Using Spark at Vungle