Speaker: Umayah Abdennabi Agenda * Intro Grammarly (Umayah Abdennabi, 5 mins) * Meetup Updates and Announcements (Chris, 5 mins) * Custom Functions in Spark SQL (30 mins) Speaker: Umayah Abdennabi Spark comes with a rich Expression library that can be extended to make custom expressions. We will look into custom expressions and why you would want to use them. * TF 2.0 + Keras (30 mins) Speaker: Francesco Mosconi Tensorflow 2.0 was announced at the March TF Dev Summit, and it brings many changes and upgrades. The most significant change is the inclusion of Keras as the default model building API. In this talk, we'll review the main changes introduced in TF 2.0 and highlight the differences between open source Keras and tf.keras * SQUAD Deep-Dive: Question & Answer with Context (45 mins) Speaker: Brett Koonce (https://quarkworks.co) SQuAD (Stanford Question Answer Dataset) is an NLP challenge based around answering questions by reading Wikipedia articles, designed to be a real-world machine learning benchmark. We will look at several different ways to tackle the SQuAD problem, building up to state of the art approaches in terms of time, complexity, and accuracy. https://rajpurkar.github.io/SQuAD-explorer/ https://dawn.cs.stanford.edu/benchmark/#squad Food and drinks will be provided. The event will be held at Grammarly's office at One Embarcadero Center on the 9th floor. When you arrive at One Embarcadero, take the escalator to the second floor where you will find the lobby and elevators to the office suites. Come on up to the 9th floor (no need to check in at security), and ring the Grammarly doorbell.

2. What is Grammarly?
A writing assistant that helps
make your communication clear
and effective, wherever you type.

3. Works Where You Do
Emails and Messages
Documents and Projects
Social Media

4. Our Mission

5. Grammarly’s evolution

6. Custom Expressions in Spark
Umayah Abdennabi
Software Engineer @ Grammarly

7. A lot of data produced
every second! How do
we understand it?

8. Gnar
Grammarly’s internal data
analytics platform.

9. Gnar
Grammarly’s internal data
analytics platform.

10. Gnar
● Goal: to understand

11. Gnar
● Goal: to understand
○ Who are our users

12. Gnar
● Goal: to understand
○ Who are our users
○ How do they interact with the product

13. Gnar
● Goal: to understand
○ Who are our users
○ How do they interact with the product
○ How do they sign-up, engage, pay, and how long do
they stay

14. Gnar
● Goal: to understand
○ Who are our users
○ How do they interact with the product
○ How do they sign-up, engage, pay, and how long do
they stay
● Allows us to make data driven decisions

15. Gnar

16. Gnar
segment “eventName”
where foo = “bar”
by browser
time from 2 months ago to today
User writes a query
using GQL using our
web application
*GQL stands for Gnar Query language, a SQL like language
built on top of Spark SQL

17. Gnar
Sent to our backend
which will run a
Spark job
segment “eventName”
where foo = “bar”
by browser
time from 2 months ago to today

18. Gnar
Results will be sent
back to the user and
displayed
segment “eventName”
where foo = “bar”
by browser
time from 2 months ago to today

19. Gnar
● When users write queries they use something
called expressions to describe what they want to
do

20. Gnar
● When users write queries they use something
called expressions to describe what they want to
do
○ The previous query had 2

21. Gnar
● When users write queries they use something
called expressions to describe what they want to
do
○ The previous query had 2
segment “eventName”
where foo = “bar”
by browser
time from 2 months ago to today

22. Gnar
● When users write queries they use something
called expressions to describe what they want to
do
● Hundreds of queries are run every day, and all of
them use expressions

23. Expressions

24. SELECT
weight,
(price - cost) * sold
FROM products
WHERE price > 100

25. SELECT
weight,
(price - cost) * sold
FROM products
WHERE price > 100

26. SELECT
weight,
(price - cost) * sold
FROM products
WHERE price > 100
These are expressions

27. (price - cost) * sold

28. (price - cost) * sold

29. sold
cost
*
-(price - cost) * sold
price

30. sold
cost
*
-
price

31. sold: 10 price: 2.99 cost: 0.50
Input Row
sold
cost
*
-
price

32. sold: 10 price: 2.99 cost: 0.50
Input Row
sold
cost
*
-
price

33. sold: 10 price: 2.99 cost: 0.50
Input Row
10
cost
*
-
price

34. sold: 10 price: 2.99 cost: 0.50
Input Row
10
0.50
*
-
2.99

35. sold: 10 price: 2.99 cost: 0.50
Input Row
10
0.50
*
-
2.99

36. sold: 10 price: 2.99 cost: 0.50
Input Row
102.49
*
2.99

37. sold: 10 price: 2.99 cost: 0.50
Input Row
102.49
*
2.99

38. sold: 10 price: 2.99 cost: 0.50
Input Row
24.90
2.99

39. Expressions
abstract class Expression extends Tree[Expression] {
...
}

40. Expressions
abstract class Expression extends Tree[Expression] {
def children: Seq[Expression]
...
}

41. Expressions
abstract class Expression extends Tree[Expression] {
def children: Seq[Expression]
def eval(row: Row): Any
...
}

42. Expressions
class Add(left: Expression, right: Expression) extends
Expression {
...
}

43. Expressions
class Add(left: Expression, right: Expression) extends
Expression {
def children: Seq[Expression] = Seq(left, right)
...
}

44. Expressions
class Add(left: Expression, right: Expression) extends
Expression {
def children: Seq[Expression] = Seq(left, right)
def eval(input: Row): Any =
left.eval(input) + right.eval(input)
...
}

45. Expressions
● How to generate a value given input values

46. Expressions
● How to generate a value given input values
● Used by Spark SQL to build around 300 SQL
functions

47. What if the expression
we want isn’t available?

48. UDF
User Defined Function

49. UDF
● Easy way to add new expressions

50. UDF
● Easy way to add new expressions
● Can be written in Scala, Java, Python, or R

51. UDF
val profit = udf((price: BigDecimal, cost: BigDecimal) =>
(price - cost) * sold
)

52. UDF
val profit = udf((price: BigDecimal, cost: BigDecimal) =>
(msrp - cost) * sold
)
SELECT weight, profit(price, cost, sold)
FROM products
WHERE msrp > 100

53. What are the limitations
with UDFs?

54. UDF
● You are using a closure which is opaque to Spark
SQL, preventing many optimizations

55. UDF
● You are using a closure which is opaque to Spark
SQL, preventing many optimizations
● Access to input types

56. UDF
● You are using a closure which is opaque to Spark
SQL, preventing many optimizations
● Access to input types
○ Spark SQL data types

57. UDF
● You are using a closure which is opaque to Spark
SQL, preventing many optimizations
● Access to input types
○ Spark SQL data types
● Hard to have a stateful implementation

58. Optimizations
Minimizing IO and computation

59. Constant Folding
hours * ( 60 * 60 * 1000)

60. Constant Folding
hours * ( 60 * 60 * 1000)
● Commonly done to get milliseconds in an hour

61. Constant Folding
hours * ( 60 * 60 * 1000)
● Commonly done to get milliseconds in an hour
● How do we reduce the time we spend
computing this largely static operation

62. Constant Folding
hours
60 *
*
*
60 1000

63. Constant Folding
hours
60 *
*
*
60 1000

64. Constant Folding
hours
60 *
*
*
60 1000

65. Constant Folding
60 *
*
*
60
hours
1000

66. Constant Folding
60 60,000
*
* hours

67. Constant Folding
60
*
*
60,000
hours

68. Constant Folding
*
hours3.6e6

69. Constant Folding
*
hours3.6e6
You can make your expressions a candidate for constant
folding by adding the following to your expression class
def foldable: Boolean = true

70. Constant Folding
1.6x Slower
1.96x Slower
1.97x Slower
One billion rows on single node machine
spark.range(1000000000l).withColumn("m", <expr>).rdd.count
1.68x Slower

71. Optimizations
Catalyst Optimizer

72. Optimizations
Catalyst Optimizer

73. Optimizations
Catalyst Optimizer

74. Boolean Simplification
a
||
&&
true true

75. Boolean Simplification
a
||
&&
true true

76. Boolean Simplification
a
||
true

77. Boolean Simplification
a
||
true

78. Boolean Simplification
true

79. Other Examples
● Pruning Filters

80. Other Examples
● Pruning Filters
● Predicate Pushdown

81. Other Examples
● Pruning Filters
● Predicate Pushdown
○ Pushing predicate within the query plan

82. Other Examples
● Pruning Filters
● Predicate Pushdown
○ Pushing predicate within the query plan
○ Pushing predicate down to data source

83. Other Examples
● Pruning Filters
● Predicate Pushdown

84. Other Examples
● Pruning Filters
● Predicate Pushdown
● Simplifying Casts

85. Optimizations
● All these optimizations are rules which are
implemented with pattern matching

86. Optimizations
● All these optimizations are rules which are
implemented with pattern matching
● If an expression matches the rule, it is applied

87. Optimizations
● All these optimizations are rules which are
implemented with pattern matching
● If an expression matches the rule, it is applied
● UDFs aren’t expressions so you cannot apply
many of these optimizations

88. Rule Example: Constant Folding
object ConstantFolding extends Rule {
def apply(plan: Plan): Plan = plan transformExpressions {
case l: Literal => l
case e if e.foldable =>
Literal.create(e.eval(EmptyRow), e.dataType)
}
}

89. Rule Example: Boolean Simplification
object BooleanSimplification extends Rule {
def apply(plan: Plan): Plan = plan transformExpressions {
case TrueLiteral And e => e
case FalseLiteral Or e => e
case e Or FalseLiteral => e
case FalseLiteral And _ => FalseLiteral
case TrueLiteral Or _ => TrueLiteral
case Not(TrueLiteral) => FalseLiteral
...
}
}

90. Custom
Expressions

91. Custom Expressions
● Don’t have the limitations of UDFs

92. Custom Expressions
● Don’t have the limitations of UDFs
○ Benefit fully from optimizations

93. Custom Expressions
● Don’t have the limitations of UDFs
○ Benefit fully from optimizations
○ Access to Spark data types

94. Custom Expressions
● Don’t have the limitations of UDFs
○ Benefit fully from optimizations
○ Access to Spark data types
○ Easy to maintain state

95. Custom Expressions
● Don’t have the limitations of UDFs
○ Benefit fully from optimizations
○ Access to Spark data types
○ Easy to maintain state
○ You can specify code generation

96. State
class TimestampToDate(ts: Expression) extends Expression {
...
}

97. State
class TimestampToDate(ts: Expression) extends Expression {
def inputTypes: DataType = LongType
...
}

98. State
class TimestampToDate(ts: Expression) extends Expression {
def inputTypes: DataType = LongType
def dataType: DataType = DateType
...
}

99. State
class TimestampToDate(ts: Expression) extends Expression {
def inputTypes: DataType = LongType
def dataType: DataType = DateType
var date = -1
var nextDaySts = -1
var prevSts = -1
...
}

100. State
class TimestampToDate(ts: Expression) extends Expression {
...
var date = -1
var nextDaySts = -1
var prevSts = -1
def eval(input: Row): {..}
...
}

101. State
def eval(input: Row) = {
val currentTs = ts.eval(input)
if (currentTs >= nextDayTs || currentTs < prevTs) {
date = DateUtils.millisToSQLDate(currentTs)
nextDaySts = DateUtils.nextDayToMillis(currentTs)
}
prevTs = currentTs
date
}

102. Type
class ToJsonExpression(sts: Expression) extends Expression {
...
}

103. Type
class ToJsonExpression(json: Expression) extends Expression {
def eval(input: InternalRow): Any = {
toJson(json.dataType, json.eval(input))
}
...
}

104. Type
def toJson(dataType: DataType, value: Any): JsValue = {
if (value == null) JsNull
else dataType match {
case BooleanType => JsBoolean(value)
case LongType => JsNumber(value)
case IntegerType => JsNumber(value)
case StringType => JsString(value)
case structType: StructType => toJsonStruct(structType, value)
case arrayType: ArrayType => toJsonArray(arrayType, value)
...
}
}

105. Code Generation
class Add(left: Expression, right) extends Expression {
...
}

106. Code Generation
class Add(left: Expression, right) extends Expression {
def doGenCode() = {
val l = left.doCodeGen()
val r = right.doCodeGen()
“””
${l}
${r}
{l.value + r.value}
“””
}...}

107. Code Generation
sold
cost
+
+
price
left.eval(input) +
right.eval(input)

108. Code Generation
sold
cost
+
+
price

109. Code Generation
sold
cost
+
+
price
Decimal p= price;
Decimal c = cost;
Decimal res = price + cost;
Decimal s = sold;
Decimal value = sold + res;

110. Code Generation
One billion rows on single node machine

111. Conclusion
● UDFs are great

112. Conclusion
● UDFs are great
○ Simple to write, compared to very involved
expressions, and they generally work well

113. Conclusion
● UDFs are great
○ Simple to write, compared to very involved
expressions, and they generally work well

114. Conclusion
● UDFs are great
○ Simple to write, compared to very involved
expressions, and they generally work well
● Custom Expressions are great

115. Conclusion
● UDFs are great
○ Simple to write, compared to very involved
expressions, and they generally work well
● Custom Expressions are great
○ Performance matters

116. Conclusion
● UDFs are great
○ Simple to write, compared to very involved
expressions, and they generally work well
● Custom Expressions are great
○ Performance matters
○ Complex operations which require lower level API

117. Conclusion
● UDFs are great
○ Simple to write, compared to very involved
expressions, and they generally work well
● Custom Expressions are great
○ Performance matters
○ Complex operations which require lower level API
● We use both of them to solve our complex problems

118. We are hiring!
www.grammarly.com/jobs

119. Questions
?
?
?
?
?
?
? ?
?
?