Utilizing Kafka Connect to Integrate Classic Monoliths into Modern Microservices | Zachary Lark, LogRhythm
•
0 recomendaciones•447 vistas
Having started with classic monolith applications in the late 90s and adopting a new microservice architecture in 2015, our organization needed a convenient, reliable, and low-cost way to push changes back and forth between them. One that preferably utilized technology already on hand and could exchange information between multiple data stores. In this session we will explore how Kafka Connect and its various connectors satisfied this need. We will review the two disparate tech stacks we needed to integrate, and the strategies and connectors we used to exchange information. Finally, we will cover some enhancements we made to our own processes including integrating Kafka Connect and its connectors into our CI/CD pipeline and writing tools to monitor connectors in our production environment.
Recomendados
Recomendados
Más contenido relacionado
La actualidad más candente
La actualidad más candente (20)
Similar a Utilizing Kafka Connect to Integrate Classic Monoliths into Modern Microservices | Zachary Lark, LogRhythm
Similar a Utilizing Kafka Connect to Integrate Classic Monoliths into Modern Microservices | Zachary Lark, LogRhythm (20)
Más de HostedbyConfluent
Más de HostedbyConfluent (20)
Último
Último (20)
Utilizing Kafka Connect to Integrate Classic Monoliths into Modern Microservices | Zachary Lark, LogRhythm
- 1. Utilizing Kafka Connect to Integrate Classic Monoliths into Modern Microservices @Zach_Lark linkedin.com/in/zachary-lark
- 2. Who am I? •Started in software development in 2003 •Professional Services -> Developer -> Dev Manager -> Architect •Using Kafka since 2016 •Started at LogRhythm in August of 2021
- 3. Customer Hosted History of the Monolith PowerBuilder 1990s Logic Logic
- 4. Customer Hosted History of the Monolith PowerBuilder 1990s Logic Logic 2000s
- 5. Data Center History of the Monolith PowerBuilder 1990s Logic Logic 2000s
- 6. Data Center History of the Monolith Java 1990s Logic Logic 2000s
- 7. Data Center Application Server History of the Monolith Java 1990s Logic Logic 2000s
- 8. Data Center Application Server History of the Monolith Java 1990s Logic Logic 2000s JSPs Logic
- 9. Data Center Application Server History of the Monolith Java 1990s Logic Logic 2000s JSPs Logic
- 10. Data Center Application Server History of the Monolith Database Server Java 1990s Logic Logic 2000s JSPs Logic
- 11. Data Center Application Server History of the Monolith Database Server Java 1990s Logic Logic 2000s JSPs Logic
- 12. Data Center Application Server History of the Monolith Database Server Java 1990s Logic Logic 2000s JSPs Logic FTP Server
- 13. Data Center Application Server History of the Monolith Database Server Java 1990s Logic Logic 2000s JSPs Logic FTP Server
- 14. Data Center Application Server History of the Monolith Database Server Java 1990s Logic Logic 2000s JSPs Logic FTP Server
- 16. 1990s 2000s Microservices The Age Of 2010s Service 1 Service 2
- 17. 1990s 2000s Microservices The Age Of 2010s Service 1 Service 2 Service 3
- 18. 1990s 2000s Microservices The Age Of 2010s
- 19. 1990s 2000s Microservices The Age Of 2010s Present
- 20. 1990s 2000s Microservices The Age Of 2010s Present
- 21. 1990s 2000s Microservices The Age Of 2010s Present
- 31. JDBC Connector Config The type of connector being configured
- 32. JDBC Connector Config Call out specific jdbc dialect to use Allows for type-specific functionality for the connected database
- 33. JDBC Connector Config Connector name
- 34. JDBC Connector Config Max number of tasks to allocate to the connector For JDBC Connectors, this is governed by the number of tables being monitored
- 35. JDBC Connector Config Topic to which records are published
- 37. JDBC Connector Config Standard JDBC Connection info
- 39. JDBC Connector Config Query used when poll executes Structured to accommodate for JDBC connector to append where clause appropriate for mode
- 40. JDBC Connector Config Mode: * bulk: All the data every time * incrementing: Use an incrementing column to detect changes * timestamp: Use a timestamp column to detect changes * timestamp+incrementing: Use a combination of both to uniquely identify changes with time
- 41. JDBC Connector Config Used to identify an incrementing and/or timestamp column depending on mode used.
- 42. JDBC Connector Config Attempt to map numeric values by precision to integral types Has been replaced by numeric.mapping
- 43. JDBC Connector Config Interval to execute connector against that data store
- 44. JDBC Connector Config Interval which the connector polls schema for new tables
- 46. JDBC Connector Config Specifies the name for avro schema used which matches that referenced in the code
- 47. JDBC Connector Config Transforms used in conjunction to extract the integer portion of the field and use its value as the message's partition key
- 48. CI/CD Pipeline with Chef
- 49. CI/CD Pipeline with Chef 1. Manage connector configurations
- 50. CI/CD Pipeline with Chef
- 51. CI/CD Pipeline with Chef
- 52. CI/CD Pipeline with Chef 1. Manage connector configurations 2. Deploy Connectors to Kafka Connect
- 53. CI/CD Pipeline with Chef End Start Changes? Issue PUT Issue DELETE New? Delete? Issue DELETE
- 54. Monitoring
- 55. Monitoring
- 56. Monitoring
- 57. Monitoring
- 58. Monitoring
- 59. Monitoring
- 60. Monitoring
- 61. Kafka Connect Bridging the Divide