This document discusses challenges faced in implementing Presto, an open source distributed SQL query engine, for targeted audience delivery at TiVo. It describes choosing appropriate instance types for Presto worker nodes based on memory needs. It also addresses scaling the Presto cluster elastically to handle query concurrency and maturity issues with the Presto software. The document provides insights on testing Presto using Docker containers and connecting to mocked tables.

1. +
Lucas.Waye @ TiVo.com
April 5th, 2018

2. About Me

3. What’s using Presto:
Targeted Audience Delivery

4. TV networks, programmers,
and advertisers
What are my target
viewership segments?
Set-Top box data
Purchasing Behavior
Location-based Consumer Data
Targeted Audience Delivery
Program Metadata

5. TV networks, programmers,
and advertisers
What are my target
viewership segments?
Set-Top box data
Purchasing Behavior
Location-based Consumer Data
Targeted Audience Delivery
Program Metadata
brought to you (in part) by

6. looking to the past for inspiration for the future

7. Similar Products at TiVo
ETL
Amazon
S3 Java services
on EC2
ETL
Amazon
Redshift
MySQL
(RDS)

8. Similar Products at TiVo
ETL
Amazon
S3 Java services
on EC2
ETL
Amazon
Redshift
MySQL
(RDS)
transactional and customer-configurable data
semi-aggregated viewership data +
sets of households (e.g., “18-24 years old”, “owns minivan”)

9. New Product, New Challenges…
ETL
Amazon
S3 Java services
on EC2
ETL
Amazon
Redshift
MySQL
(RDS)
MySQL
MySQL
MySQL
Many new data marts
popping up in our tech stack

10. New Product, New Challenges…
ETL
Amazon
S3 Java services
on EC2
ETL
Amazon
Redshift
MySQL
(RDS)
more viewership data
OK,
storage is cheap

11. New Product, New Challenges…
ETL
Amazon
S3 Java services
on EC2
ETL
Amazon
Redshift
MySQL
(RDS)
more viewership data
storage is not cheap…

12. New Product, New Challenges…
ETL
Amazon
S3 Java services
on EC2
ETL
Amazon
Redshift
MySQL
(RDS)
storage is not cheap… Need finer
grain data!

13. New Product, New Challenges…
ETL
Amazon
S3 Java services
on EC2
ETL
Amazon
Redshift
MySQL
(RDS)
storage is not cheap… Need finer
grain data!
Can’t aggregate
as much

14. New Product, New Challenges…
ETL
Amazon
S3 Java services
on EC2
ETL
Amazon
Redshift
MySQL
(RDS)
static,
hard to scale

15. Wait, what about Redshift Spectrum ?

16. Redshift Spectrum
Redshift: Pay per
node-hour
Spectrum: Pay per
data access

17. How Does it Scale?

18. Experiment: join on two tables
• Small Joins: join small Redshift table with (filtered-down) large table on S3
• Join across ~1M rows
• Large Joins: join large Redshift table with (unfiltered) large table table on S3
• Join across ~10M rows
Compare to: both tables on Redshift
How Does it Scale?

19. Time
(sec)
Concurrent queries
Redshift Spectrum for “Simple" Queries
0
10
20
30
40
50
60
70
1 3 5 7 9 11 13 15
Latency (sec) vs. # Concurrent Requests
1 day 1 day (Spectrum)
Spectrum faster when cluster loaded
and can pre-filter/pre-aggregate data
small joins

20. Time
(sec)
Concurrent queries
Redshift Spectrum for “Simple" Queries
0
10
20
30
40
50
60
70
1 3 5 7 9 11 13 15
Latency (sec) vs. # Concurrent Requests
1 day 1 day (Spectrum)
Spectrum faster when cluster loaded
and can pre-filter/pre-aggregate data
small joins
Spectrum faster

21. Time
(sec)
Concurrent queries
Redshift Spectrum for Complex Queries

22. Time
(sec)
Concurrent queries
Redshift Spectrum for Complex Queries
Spectrum slower!

23. Memory for broadcast join on the cluster is a non-parallelizable resource in the cluster
Amdahl’s Law in Effect

24. Memory for broadcast join on the cluster is a non-parallelizable resource in the cluster
Amdahl’s Law in Effect
“Operations that can't be pushed to the Redshift Spectrum
layer include [JOIN], DISTINCT and ORDER BY. …
When large amounts of data are returned from Amazon S3,
the processing is limited by your cluster's resources.”
https://docs.aws.amazon.com/redshift/latest/dg/c-spectrum-external-performance.html

25. Wait, what about Redshift Spectrum ?
Our queries won’t work well on Spectrum.

26. Well, what about [X] ?

27. Our Choice:
• Storage/Compute Separation
• Easy to add and remove worker nodes
• Query many different data sources (inside our VPC)
without separate load
• Good performance for analytical queries.
Not so good for transactional and simple queries…
• Managed (e.g., Qubole, Starburst)

28. Coordinator
Worker Worker Worker
S3 / Hive
metastore
MySQL
Connector
Connector
SELECT SUM(v.seconds_viewed)
FROM hive.db.viewership v
JOIN mysql.db.audiences a ON a.hh_id = v.hh_id
WHERE audience_id = 42
mysql catalog à
hive catalog à
SELECT …
FROM db.audiences
WHERE audience_id = 42
DRAFT - TiVo Confidential 2018
How Presto Works
Data is streamed
back to the workers

29. First Challenge:
What instance types should we use?

30. Presto Worker Memory
System Memory
reserved-system-memory =
0.4 * JVM Max Memory
Reserved Memory
max-memory-per-node
General Memory
(the rest)
All Queries Start Using
Memory From Here

31. Presto Worker Memory
System Memory
reserved-system-memory =
0.4 * JVM Max Memory
Reserved Memory
max-memory-per-node
General Memory
(the rest)
All Queries Start Using
Memory From Here
Query

32. Presto Worker Memory
System Memory
reserved-system-memory =
0.4 * JVM Max Memory
Reserved Memory
max-memory-per-node
General Memory
(the rest)
Needs more memory than in
General Pool —> Switch to Reserved
Query

33. Presto Worker Memory
System Memory
reserved-system-memory =
0.4 * JVM Max Memory
Reserved Memory
max-memory-per-node
General Memory
(the rest)
Needs more memory than in
General Pool —> Switch to Reserved
Query

34. Presto Worker Memory
System Memory
reserved-system-memory =
0.4 * JVM Max Memory
Reserved Memory
max-memory-per-node
General Memory
(the rest)
Needs more memory than in
General Pool —> Switch to Reserved
Query
Only one query allowed!

35. Presto Worker Memory
System Memory
reserved-system-memory =
0.4 * JVM Max Memory
Reserved Memory
max-memory-per-node
General Memory
(the rest)
Needs more memory than in
Reserved Pool —> Fail
Query

36. Presto Worker Memory
System Memory
reserved-system-memory =
0.4 * JVM Max Memory
Reserved Memory
max-memory-per-node
General Memory
(the rest)
Needs more memory than in
Reserved Pool —> Fail
Query
But there’s available
memory??

37. Presto Worker Memory
System Memory
reserved-system-memory =
0.4 * JVM Max Memory
Reserved Memory
max-memory-per-node
General Memory
(the rest)
Needs more memory than in
Reserved Pool —> keep allocating
(resource overcommit)
Query

38. Presto Worker Memory
System Memory
reserved-system-memory =
0.4 * JVM Max Memory
Reserved Memory
max-memory-per-node
General Memory
(the rest)
Query
But now a single query can
hog the entire cluster!

39. Presto Worker Memory
Query

40. Presto Worker Memory
Query Query
Presto Worker Memory
Multiple Workers

41. Presto Worker Memory
Query Query
Presto Worker Memory
Multiple Workers
Total Memory
max-memory=
max-memory-per-node * number of nodes

42. Presto Worker Memory
QueryQuery
Presto Worker Memory
Multiple Workers
Total Memory
max-memory=
max-memory-per-node * number of nodes
QueryQuery

43. Total Memory
max-memory=
max-memory-per-node * number of nodes
Presto Worker Memory
Query
Query
Presto Worker Memory
Multiple Workers
Query
Query

44. • What if memory usage varies a lot between different queries?
• Use many inexpensive instances, or a few expensive instances?
• Compute optimized or memory optimized?
Working With Reserved Memory Pool
How do we achieve that?
Conceptually, reserved memory pool should be the “high water mark”
while most queries complete in the general pool.

45. • What if memory usage varies a lot between different queries?
• Use many inexpensive instances, or a few expensive instances?
• Compute optimized or memory optimized?
Working With Reserved Memory Pool
Conceptually, reserved memory pool should be the “high water mark”
while most queries complete in the general pool.
Solution: multiple clusters based on workload
Empiric testing found smaller cluster size was slightly faster
Solution: Cost/Benefit Analysis
How do we achieve that?

46. Choosing the Right Instance Type
r 4 . 4 x l a r g e
Instance
Class
Generation
Multiplier
For CPU and Mem
t 2 . 2 x l a r g e
c 5 . 16x l a r g e

47. Choosing the Right Instance Type
r 4 . 4 x l a r g e
Instance
Class
Generation
Multiplier
For CPU and Mem
t 2 . 2 x l a r g e
c 5 . 16x l a r g e
Over 100 to choose from!

48. Choosing the Right Instance Type
Credit: Willard Simmons (DataXu)

49. Choosing the Right Instance Type
Credit: Willard Simmons (DataXu)
Older generations
are inefficient

50. Choosing the Right Instance Type
Credit: Willard Simmons (DataXu)
Better for larger
memory clusters
Older generations
are inefficient

51. Choosing the Right Instance Type
Credit: Willard Simmons (DataXu)
Better for smaller
memory clusters
Older generations
are inefficient

52. Second Challenge:
Elastic Scaling

53. More Concurrency? Add More Nodes

54. More Concurrency? Add More Nodes
Presto
Worker
Presto
Worker
Presto
Coordinator
1 Query
When will queries complete
at current rate?

55. More Concurrency? Add More Nodes
Presto
Worker
Presto
Worker
Presto
Coordinator
10 Queries
When will queries complete
at current rate?
Not fast enough!

56. More Concurrency? Add More Nodes
Presto
Worker
Presto
Worker
Presto
Coordinator
10 Queries
When will queries complete
at current rate?
Qubole provisions more nodes up to a limit
(around 3 minutes)
Presto
Worker
Presto
Worker

57. More Concurrency? Add More Nodes
Presto
Worker
Presto
Worker
Presto
Coordinator
1 Query
When will queries complete
at current rate?
Presto
Worker
Presto
Worker
Too fast!

58. More Concurrency? Add More Nodes
Presto
Worker
Presto
Worker
Presto
Coordinator
1 Query
When will queries complete
at current rate?
Qubole decommissions more nodes up to a limit

59. Not so fast…
Presto
Worker
Presto
Worker
Presto
Coordinator
1 Query
When will queries complete
at current rate?
Not fast enough!
100% CPU 100% CPU

60. Presto
Worker
Presto
Worker
Presto
Coordinator
1 Query
When will queries complete
at current rate?
Upscaling only works for new queries
Presto
Worker
Presto
Worker
100% CPU 100% CPUIdle Idle
Not so fast…
Not fast enough!

61. Presto
Worker
Presto
Worker
Presto
Coordinator
1 Query
When will queries complete
at current rate?
Upscaling only works for new queries
Presto
Worker
Presto
Worker
100% CPU 100% CPUIdle Idle
Not so fast…
Not fast enough!
Maybe we should have sent this query
to a more powerful cluster?
Autoscaling is for concurrency

62. Third Challenge:
Maturity

63. Query History
Presto UI is nice for watching queries as they’re happening, but not for historical auditing

64. Service administration portal tracks Qubole commands
(Presto queries) and links to the Qubole web site
View and download intermediate queries and results
Presto Query Auditing

65. • Official Presto JDBC driver does not support Prepared Statements
• Worker loss not handled gracefully
(if one task fails, all tasks fail — we take that risk with retry logic)
• No support for upper-case table names in MySQL (Issue 2863)
• TIMESTAMP behavior does not match SQL standard (Issue 7122)
• Naïve query optimizer (talk to Starburst!)
Specific Technical Presto Issues

66. • Official Presto JDBC driver does not support Prepared Statements
• Worker loss not handled gracefully
(if one task fails, all tasks fail — we take that risk with retry logic)
• No support for upper-case table names in MySQL (Issue 2863)
• TIMESTAMP behavior does not match SQL standard (Issue 7122)
• Naïve query optimizer (talk to Starburst!)
Moral: you may need to get creative with workarounds
Specific Technical Presto Issues

67. Presto Docker container
using memory connectors
Testing

68. Presto Docker container
using memory connectors
Testing
Declarative syntax allows us to mock tables
in the Docker container

69. Presto Docker container
using memory connectors
Testing
Declarative syntax allows us to mock tables
in the Docker container
…so we can test our generated queries in isolation
using Behavior-Driven Development.

70. Final Takeaways

71. Setting expectations: Make sure everyone knows Presto is
not a full-fledged database.

72. Providing one logical view of the data model across many databases is great!
Favorite for many other workloads beyond its initial scope for this reason.
Presto’s simplicity resulted in widespread adoption.
Biggest (Positive) Surprise

73. Provocative Ending

74. Provocative Ending
Presto feels like an API gateway, but for data.
Behavioral Services Data Applications
Interface (REST, WSDL, Thrift, etc.) :: Data Definition Language (DDL)
Requests (HTTP, SOAP, etc.) :: Data Manipulation Language (DML)
Service implementation language :: Database technology
Publishing an endpoint :: Exposing a table or view
Service handler :: CREATE VIEW, CREATE TRIGGER
Service endpoint configuration :: Catalog/connector configuration

75. Provocative Ending
Presto feels like an API gateway, but for data.
Behavioral Services Data Applications
Interface (REST, WSDL, Thrift, etc.) :: Data Definition Language (DDL)
Requests (HTTP, SOAP, etc.) :: Data Manipulation Language (DML)
Service implementation language :: Database technology
Publishing an endpoint :: Exposing a table or view
Service handler :: CREATE VIEW, CREATE TRIGGER
Service endpoint configuration :: Catalog/connector configuration
What other engineering advancements can we push through the lens from
microservices (behaviors) to databases (state)?

76. Thanks!
Questions?