Cassandra nice use cases and worst anti patterns

1. Cassandra nice use-cases and worst anti-patterns
DuyHai DOAN, Technical Advocate
@doanduyhai

2. Agenda!
@doanduyhai
2
Anti-patterns
• Queue-like designs
• CQL null values
• Intensive update on same column
• Design around dynamic schema

3. Agenda!
@doanduyhai
3
Nice use-cases
• Rate-limiting
• Anti Fraud
• Account validation
• Sensor data timeseries

4. Worst anti-patterns!
Queue-like designs!
CQL null!
Intensive update on same column!
Design around dynamic schema!
!

5. Failure level!
@doanduyhai
5
☠
☠☠
☠☠☠
☠☠☠☠

6. Queue-like designs!
@doanduyhai
6
Adding new message ☞ 1 physical insert

7. Queue-like designs!
@doanduyhai
7
Adding new message ☞ 1 physical insert
Consuming message = deleting it ☞ 1 physical insert (tombstone)

8. Queue-like designs!
@doanduyhai
8
Adding new message ☞ 1 physical insert
Consuming message = deleting it ☞ 1 physical insert (tombstone)
Transactional queue = re-inserting messages ☞ physical insert * <many>

9. Queue-like designs!
FIFO queue
@doanduyhai
9
A
{ A }

10. Queue-like designs!
FIFO queue
@doanduyhai
10
A B
{ A, B }

11. Queue-like designs!
FIFO queue
@doanduyhai
11
A B C
{ A, B, C }

12. Queue-like designs!
FIFO queue
@doanduyhai
12
A B C A
{ B, C }

13. Queue-like designs!
FIFO queue
@doanduyhai
13
A B C A D
{ B, C, D }

14. Queue-like designs!
FIFO queue
@doanduyhai
14
A B C A D B
{ C, D }

15. Queue-like designs!
FIFO queue
@doanduyhai
15
A B C A D B C
{ D }

16. Queue-like designs!
FIFO queue, worst case
@doanduyhai
16
A A A A A A A A A A
{ }

17. Failure level!
@doanduyhai
17
☠☠☠

18. CQL null semantics!
@doanduyhai
18
Reading null value means
• value does not exist (has never bean created)
• value deleted (tombstone)
SELECT age FROM users WHERE login = ddoan; à NULL

19. CQL null semantics!
@doanduyhai
19
Writing null means
• delete value (creating tombstone)
• even though it does not exist
UPDATE users SET age = NULL WHERE login = ddoan;

20. CQL null semantics!
@doanduyhai
20
Seen in production: prepared statement
UPDATE users SET
age = ?,
…
geo_location = ?,
mood = ?,
…
WHERE login = ?;

21. CQL null semantics!
@doanduyhai
21
Seen in production: bound statement
preparedStatement.bind(33, …, null, null, null, …);
null ☞ tombstone creation on each update …
jdoe
age name geo_loc mood status
33 John DOE ý ý ý

22. Failure level!
@doanduyhai
22
☠

23. Intensive update!
@doanduyhai
23
Context
• small start-up
• cloud-based video recording & alarm
• internet of things (sensor)
• 10 updates/sec for some sensors

24. Intensive update on same column!
@doanduyhai
24
Data model
sensor_id
value
45.0034
CREATE TABLE sensor_data (
sensor_id long,
value double,
PRIMARY KEY(sensor_id));

25. Intensive update on same column!
UPDATE sensor_data SET value = 45.0034 WHERE sensor_id = …;
UPDATE sensor_data SET value = 47.4182 WHERE sensor_id = …;
UPDATE sensor_data SET value = 48.0300 WHERE sensor_id = …;
@doanduyhai
25
Updates
sensor_id
value (t1)
45.0034
sensor_id
value (t13)
47.4182
sensor_id
value (t36)
48.0300

26. Intensive update on same column!
@doanduyhai
26
Read
SELECT sensor_value from sensor_data WHERE sensor_id = …;
read N physical columns, only 1 useful …
sensor_id
value (t1)
45.0034
sensor_id
value (t13)
47.4182
sensor_id
value (t36)
48.0300

27. Intensive update on same column!
@doanduyhai
27
Solution 1: leveled compaction! (if your I/O can keep up)
sensor_id
value (t1)
45.0034
sensor_id
value (t13)
47.4182
sensor_id
value (t36)
48.0300
sensor_id
value (t36)
48.0300

28. Intensive update on same column!
@doanduyhai
28
Solution 2: reversed timeseries & DateTiered compaction strategy
CREATE TABLE sensor_data (
sensor_id long,
date timestamp,
sensor_value double,
PRIMARY KEY((sensor_id), date))
WITH CLUSTERING ORDER (date DESC);

29. Intensive update on same column!
SELECT sensor_value FROM sensor_data WHERE sensor_id = … LIMIT 1;
@doanduyhai
29
sensor_id
date3(t3)
date2(t2)
date1(t1)
Data cleaning by configuration (max_sstable_age_days)
...
48.0300 47.4182 45.0034 …

30. Failure level!
@doanduyhai
30
☠☠

31. Design around dynamic schema!
@doanduyhai
31
Customer emergency call
• 3 nodes cluster almost full
• impossible to scale out
• 4th node in JOINING state for 1 week
• disk space is filling up, production at risk!

32. Design around dynamic schema!
@doanduyhai
32
After investigation
• 4th node in JOINING state because streaming is stalled
• NPE in logs

33. Design around dynamic schema!
@doanduyhai
33
After investigation
• 4th node in JOINING state because streaming is stalled
• NPE in logs
Cassandra source-code to the rescue

34. Design around dynamic schema!
@doanduyhai
34
public class CompressedStreamReader extends StreamReader
{
…
@Override
public SSTableWriter read(ReadableByteChannel channel) throws IOException
{
…
Pair<String, String> kscf = Schema.instance.getCF(cfId);
ColumnFamilyStore cfs = Keyspace.open(kscf.left).getColumnFamilyStore(kscf.right);
NPE here

35. Design around dynamic schema!
@doanduyhai
35
The truth is
• the devs dynamically drop & recreate table every day
• dynamic schema is in the core of their design
Example:
DROP TABLE catalog_127_20140613;
CREATE TABLE catalog_127_20140614( … );

36. Design around dynamic schema!
@doanduyhai
36
Failure sequence
n1
n2
n4
n3
catalog_x_y
catalog_x_y
catalog_x_y
catalog_x_y
4 1
2
3
5
6

37. Design around dynamic schema!
@doanduyhai
37
Failure sequence
n1
n2
n4
n3
catalog_x_y
catalog_x_y
catalog_x_y
catalog_x_y
4 1
2
3
5
6
catalog_x_z
catalog_x_z
catalog_x_z
catalog_x_z

38. Design around dynamic schema!
@doanduyhai
catalog_x_y ????
38
Failure sequence
n1
n2
n4
n3
4 1
2
3
5
6
catalog_x_z
catalog_x_z
catalog_x_z
catalog_x_z

39. Design around dynamic schema!
@doanduyhai
39
Consequences
• joining node got always stuck
• à cannot extend cluster
•
à changing code takes time
•
à production in danger (no space left)
•
à sacrify analytics data to survive

40. Design around dynamic schema!
@doanduyhai
40
Nutshell
• dynamic schema change as normal operations is not recommended
• concurrent schema AND topology change is an anti-pattern

41. Failure level!
@doanduyhai
41
☠☠☠☠

42. ! "
!
Q & R

43. Nice Examples!
Rate limiting!
Anti Fraud!
Account Validation!
Sensor Data Timeseries!

44. Rate limiting!
@doanduyhai
44
Start-up company, reset password feature
1) /password/reset
2) SMS with token A0F83E63DB935465CE73DFE….
Phone number Random token
3) /password/new/<token>/<password>

45. Rate limiting!
@doanduyhai
45
Problem 1
• account created with premium phone number

46. Rate limiting!
@doanduyhai
46
Problem 1
• account created with premium phone number
• /password/reset x 100

47. Rate limiting!
@doanduyhai
47
« money, money, money, give money, in the richman’s world » $$$

48. Rate limiting!
@doanduyhai
48
Problem 2
• massive hack

49. Rate limiting!
@doanduyhai
49
Problem 2
• massive hack
• 106 /password/reset calls from few accounts

50. Rate limiting!
@doanduyhai
50
Problem 2
• massive hack
• 106 /password/reset calls from few accounts
• SMS messages are cheap

51. Rate limiting!
@doanduyhai
51
Problem 2
• ☞ but not at the 106/per user/per day scale

52. Rate limiting!
@doanduyhai
52
Solution
• premium phone number ☞ Google libphonenumber

53. Rate limiting!
@doanduyhai
53
Solution
• premium phone number ☞ Google libphonenumber
• massive hack ☞ rate limiting with Cassandra

54. Cassandra Time To Live!
@doanduyhai
54
Time to live
• built-in feature
• insert data with a TTL in sec
• expires server-side automatically
• ☞ use as sliding-window

55. Rate limiting in action!
@doanduyhai
55
Implementation
• threshold = max 3 reset password per sliding 24h

56. Rate limiting in action!
@doanduyhai
56
Implementation
• when /password/reset called
• check threshold
• reached ☞ error message/ignore
• not reached ☞ log the attempt with TTL = 86400

57. Rate limiting
demo

58. Anti Fraud!
@doanduyhai
58
Real story
• many special offers available
• 30 mins international calls (50 countries)
• unlimited land-line calls to 5 countries
• …

59. Anti Fraud!
@doanduyhai
59
Real story
• each offer has a duration (week/month/year)
• only one offer active at a time

60. Anti Fraud!
@doanduyhai
60
Cassandra TTL
• check for existing offer before
SELECT count(*) FROM user_special_offer WHERE login = ‘jdoe’;

61. Anti Fraud!
@doanduyhai
61
Cassandra TTL
• then grant new offer
INSERT INTO user_special_offer(login, offer_code, …)
VALUES(‘jdoe’, ’30_mins_international’,…)
USING TTL <offer_duration>;

62. Account Validation!
@doanduyhai
62
Requirement
• user creates new account
• sends sms/email link with token to validate account
• 10 days to validate

63. Account Validation!
@doanduyhai
63
How to ?
• create account with 10 days TTL
INSERT INTO users(login, name, age)
VALUES(‘jdoe’, ‘John DOE’, 33)
USING TTL 864000;

64. Account Validation!
@doanduyhai
64
How to ?
• create random token for validation with 10 days TTL
INSERT INTO account_validation(token, login, name, age)
VALUES(‘A0F83E63DB935465CE73DFE…’, ‘jdoe’, ‘John DOE’, 33)
USING TTL 864000;

65. Account Validation!
@doanduyhai
65
On token validation
• check token exist & retrieve user details
SELECT login, name, age FROM account_validation
WHERE token = ‘A0F83E63DB935465CE73DFE…’;
• re-insert durably user details without TTL
INSERT INTO users(login, name, age) VALUES(‘jdoe’, ‘John DOE’, 33);

66. Sensor Data Timeseries!
@doanduyhai
66
Requirements
• lots of sensors (103 – 106)
• medium to high insertion rate (0.1 – 10/secs)
• keep good load balancing
• fast read & write

67. Bucketing!
@doanduyhai
67
CREATE TABLE sensor_data (
sensor_id text,
date timestamp,
raw_data blob,
PRIMARY KEY(sensor_id, date));
sensor_id
date1 date2 date3 date4 …
blob1 blob2 blob3 blob4 …

68. Bucketing!
@doanduyhai
68
Problems:
• limit of 2.109 physical columns
• bad load balancing (1 sensor = 1 node)
• wide row spans over many files
sensor_id
date1 date2 date3 date4 …
blob1 blob2 blob3 blob4 …

69. Bucketing!
@doanduyhai
69
Idea:
• composite partition key: sensor_id:date_bucket
• tunable date granularity: per hour/per day/per month …
CREATE TABLE sensor_data (
sensor_id text,
date_bucket int, //format YYYYMMdd
date timestamp,
raw_data blob,
PRIMARY KEY((sensor_id, date_bucket), date));

70. Bucketing!
Idea:
• composite partition key: sensor_id:date_bucket
• tunable date granularity: per hour/per day/per month …
@doanduyhai
70
sensor_id:2014091014
date1 date2 date3 date4 …
blob1 blob2 blob3 blob4 …
sensor_id:2014091015
date11 date12 date13 date14 …
blob11 blob12 blob13 blob14 …
Buckets

71. Bucketing!
@doanduyhai
71
Advantage:
• distribute load: 1 bucket = 1 node
• limit partition width (max x columns per bucket)
Buckets
sensor_id:2014091014
date1 date2 date3 date4 …
blob1 blob2 blob3 blob4 …
sensor_id:2014091015
date11 date12 date13 date14 …
blob11 blob12 blob13 blob14 …

72. Bucketing!
@doanduyhai
72
But how can I select raw data between 14:45 and 15:10 ?
14:45 à ?
15:00 à 15:10
sensor_id:2014091014
date1 date2 date3 date4 …
blob1 blob2 blob3 blob4 …
sensor_id:2014091015
date11 date12 date13 date14 …
blob11 blob12 blob13 blob14 …

73. Bucketing!
Solution
• use IN clause on partition key component
• with range condition on date column
☞ date column should be monotonic function (increasing/decreasing)
@doanduyhai
73
SELECT * FROM sensor_data WHERE sensor_id = xxx
AND date_bucket IN (2014091014 , 2014091015)
AND date >= ‘2014-09-10 14:45:00.000‘
AND date <= ‘2014-09-10 15:10:00.000‘

74. Bucketing Caveats!
@doanduyhai
74
IN clause for #partition is not silver bullet !
• use scarcely
• keep cardinality low (≤ 5)
n1
n2
n3
n4
n5
n6
n7
coordinator
n8
sensor_id:2014091014
sensor_id:2014091015

75. Bucketing Caveats!
@doanduyhai
75
IN clause for #partition is not silver bullet !
• use scarcely
• keep cardinality low (≤ 5)
• prefer // async queries
• ease of query vs perf
n1
n2
n3
n4
n5
n6
n7
n8
Async client
sensor_id:2014091014
sensor_id:2014091015

76. ! "
!
Q & R

77. Cassandra developers!
@doanduyhai
77
Rule n°1
If you don’t know, ask for help
(me, Cassandra ML, PlanetCassandra, stackoverflow, …)
!

78. Cassandra developers!
@doanduyhai
78
Rule n°2
Do not blind-guess troubleshooting
alone in production
(ask for help, see rule n°1)
!

79. Cassandra developers!
@doanduyhai
79
Rule n°3
Share with the community
(your best use-cases … and worst failures)
!
http://planetcassandra.org/

80. Thank You
@doanduyhai
duy_hai.doan@datastax.com