Over the last decade, the 3Vs of data - Volume, Velocity & Variety has grown massively. The Big Data revolution has completely changed the way companies collect, analyze & store data. Advancements in cloud-based data warehousing technologies have empowered companies to fully leverage big data without heavy investments both in terms of time and resources. But, that doesn’t mean building and managing a cloud data warehouse isn’t accompanied by any challenges. From deciding on a service provider to the design architecture, deploying a data warehouse tailored to your business needs is a strenuous undertaking. Looking to deploy a data warehouse to scale your company’s data infrastructure or still on the fence? In this presentation you will gain insights into the current Data Warehousing trends, best practices, and future outlook. Learn how to build your data warehouse with the help of real-life use-cases and discussion on commonly faced challenges. In this session you will learn:
- Choosing the best solution - Data Lake vs. Data Warehouse vs. Data Mart
- Choosing the best Data Warehouse design methodologies: Data Vault vs. Kimball vs. Inmon
- Step by step approach to building an effective data warehouse architecture
- Common reasons for the failure of data warehouse implementations and how to avoid them
Data Warehousing Trends, Best Practices, and Future Outlook
1. Data Warehousing Trends,
Best Practices, and Future
Outlook
James Serra
Data Platform Architecture Lead
EY
JamesSerra3@gmail.com
Blog: JamesSerra.com
2. About Me
EY, Data Platform Architecture Lead
Was previously a Data & AI Architect at Microsoft for seven years
In IT for 35 years, worked on many BI and DW projects
Worked as desktop/web/database developer, DBA, BI and DW architect and developer, MDM
architect, PDW/APS developer
Been perm employee, contractor, consultant, business owner
Presenter at PASS Summit, SQLBits, Enterprise Data World conference, Big Data Conference
Europe, SQL Saturdays
Blog at JamesSerra.com
Former SQL Server MVP
Author of book “Reporting with Microsoft SQL Server 2012”
3. Agenda
Why Data Warehouse
Data Warehouse vs Data Mart
Kimball vs Inmon
Data Vault
Data Lake
Why Data Warehouses fail
How Data Warehouses succeed
Data Warehouse concepts
Modern Data Warehouse
Data Fabric
Data Lakehouse
Data Mesh
4. I tried to build a data warehouse on my own…
And ended up passed-out drunk in a Denny’s
parking lot
Let’s prevent that from happening…
5. What a Data Warehouse is not
• A data warehouse is not a copy of a source database with the name prefixed with “DW”
• It is not a copy of multiple tables (i.e. customer) from various sources systems unioned
together in a view
• It is not a dumping ground for tables from various sources with not much design put into it
6. What is a Data Warehouse and why use one?
A data warehouse is where you store data from multiple data sources to be used for historical and
trend analysis reporting. It acts as a central repository for many subject areas and contains the
"single version of truth". It is NOT to be used for OLTP applications.
Reasons for a data warehouse:
Reduce stress on production system
Optimized for read access, sequential disk scans
Integrate many sources of data
Keep historical records (no need to save hardcopy reports)
Restructure/rename tables and fields, model data
Protect against source system upgrades
Use Master Data Management, including hierarchies
No IT involvement needed for users to create reports
Improve data quality and plugs holes in source systems
One version of the truth
Easy to create BI solutions on top of it (i.e. Azure Analysis Services Cubes)
7. Why use a Data Warehouse?
Legacy applications + databases = chaos
Production
Control
MRP
Inventory
Control
Parts
Management
Logistics
Shipping
Raw Goods
Order Control
Purchasing
Marketing
Finance
Sales
Accounting
Management
Reporting
Engineering
Actuarial
Human
Resources
Continuity
Consolidation
Control
Compliance
Collaboration
Enterprise data warehouse = order
Single version
of the truth
Enterprise Data
Warehouse
Every question = decision
Two purposes of data warehouse: 1) save time building reports; 2) slice in dice in ways you could not do before
8. Enterprise Data Maturity Stages
Structured data is
transacted and
locally managed.
Data used
reactively
STAGE 2:
Informative
STAGE 1:
Reactive
Structured data is
managed and
analyzed centrally
and informs the
business
Data capture is
comprehensive and
scalable and leads
business decisions
based on advanced
analytics
STAGE 4:
Transformative
STAGE 3:
Predictive Data transforms
business to drive
desired outcomes.
Real-time
intelligence
Rear-view
mirror
Any data, any
source, anywhere at
scale
9. Data Warehouse vs Data Mart
Data Warehouse: A single organizational repository of enterprise wide data
across many or all subject areas
Holds multiple subject areas
Holds very detailed information
Works to integrate all data sources
Feeds data mart
Data Mart: Subset of the data warehouse that is usually oriented to specific
subject (finance, marketing, sales)
• The logical combination of all the data marts is a data warehouse
In short, a data warehouse as contains many subject areas, and a data mart
contains just one of those subject areas
10. Kimball and Inmon Methodologies
Two approaches for building data warehouses
11. Kimball and Inmon Myths
Myth: Kimball is a bottom-up approach without enterprise focus
Really top-down: BUS matrix (choose business processes/data sources), conformed
dimensions, MDM
Myth: Inmon requires a ton of up-front design that takes a long time
Inmon says to build DW iteratively, not big bang approach (p. 91 BDW, p. 21 Imhoff)
Myth: Star schema data marts are not allowed in Inmon’s model
Inmon says they are good for direct end-user access of data (p. 365 BDW), good for data
marts (p. 12 TTA)
Myth: Very few companies use the Inmon method
Survey had 39% Inmon vs 26% Kimball. Many have a EDW
Myth: The Kimball and Inmon architectures are incompatible
They can work together to provide a better solution
12. Kimball and Inmon Methodologies
Relational (Inmon) vs Dimensional (Kimball)
Relational Modeling:
Entity-Relationship (ER) model
Normalization rules
Many tables using joins
History tables, natural keys
Good for indirect end-user access of data
Dimensional Modeling:
Facts and dimensions, star schema
Less tables but have duplicate data (de-normalized)
Easier for user to understand (but strange for IT people used to relational)
Slowly changing dimensions, surrogate keys
Good for direct end-user access of data
13. Relational Model vs Dimensional Model
Relational Model Dimensional Model
If you are a business user, which model is easier to use?
14. Kimball and Inmon Methodologies
• Kimball:
• Logical data warehouse (BUS), made up of subject areas (data marts)
• Business driven, users have active participation
• Decentralized data marts (not required to be a separate physical data store)
• Independent dimensional data marts optimized for reporting/analytics
• Integrated via Conformed Dimensions (provides consistency across data sources)
• 2-tier (data mart, cube), less ETL, no data duplication
• Inmon:
• Enterprise data model (CIF) that is a enterprise data warehouse (EDW)
• IT Driven, users have passive participation
• Centralized atomic normalized tables (off limit to end users)
• Later create dependent data marts that are separate physical subsets of data and can
be used for multiple purposes
• Integration via enterprise data model
• 3-tier (data warehouse, data mart, cube), duplication of data
17. Reasons to add a Enterprise Data Warehouse
Single version of the truth
May make building dimensions easier using lightly denormalized tables in EDW instead of
going directly from the OLTP source
Normalized EDW results in enterprise-wide consistency which makes it easier to spawn-off
the data marts at the expense of duplicated data
Less daily ETL refresh and reconciliation if have many sources and many data marts in
multiple databases
One place to control data (no duplication of effort and data)
Reason not to: If have a few sources that need reporting quickly
18. Which model to use?
Models are not that different, having become similar over the years, and can compliment
each other
Boils down to Inmon creates a normalized DW before creating a dimensional data mart and
Kimball skips the normalized DW
With tweaks to each model, they look very similar (adding a normalized EDW to Kimball,
dimensionally structured data marts to Inmon)
Bottom line: Understand both approaches and pick parts from both for your situation – no
need to just choose just one approach
BUT, no solution will be effective unless you possess soft skills (leadership, communication,
planning, and interpersonal relationships)
19. Hybrid Model
Advice: Use SQL Server Views to interface between each level in the model (default values, computations, rename fields, etc)
In the DW Bus Architecture, each data mart could be a schema (broken out by business process subject areas), all in one
database. Another option is to have each data mart in its own database with all databases on one server or spread among
multiple servers. Also, the staging areas, CIF, and DW Bus can all be on the same powerful server (MPP)
20. Kimball Methodology
From: Kimball’s The Microsoft Data Warehouse Toolkit
Kimball defines a development lifecycle, where Inmon is just about the data warehouse (not “how” used)
21. Data Vault
• Daniel Linstedt, author of Building Scalable Data Warehouse with Data Vault 2.0
• Not just a modeling technique, but entire methodology for DW projects
• Layer between Star Schema and Staging (use in combination with Kimball model, modeled
somewhere between the 3NF and star schema)
• Improves audit and historical tracking
• Hubs (business entities)
• Links (relationships)
• Satellites (descriptive attributes)
https://www.indellient.com/blog/data-vault-what-is-it-and-when-should-it-be-used/
22. Observation
Pattern
Theory
Hypothesis
What will
happen?
How can we
make it happen?
Predictive
Analytics
Prescriptive
Analytics
What
happened?
Why did
it happen?
Descriptive
Analytics
Diagnostic
Analytics
Confirmation
Theory
Hypothesis
Observation
Two Approaches to getting value out of data: Top-Down +
Bottoms-Up
23. Implement Data Warehouse
Physical Design
ETL
Development
Reporting &
Analytics
Development
Install and Tune
Reporting &
Analytics Design
Dimension Modelling
ETL Design
Setup Infrastructure
Understand
Corporate
Strategy
Data Warehousing Uses A Top-Down Approach
Data sources
Gather
Requirements
Business
Requirements
Technical
Requirements
24. The “data lake” Uses A Bottoms-Up Approach
Ingest all data
regardless of requirements
Store all data
in native format without
schema definition
Do analysis
Using analytic engines
like Hadoop
Interactive queries
Batch queries
Machine Learning
Data warehouse
Real-time analytics
Devices
25. Data Lake + Data Warehouse Better Together
Data sources
What happened?
Descriptive
Analytics
Diagnostic
Analytics
Why did it happen?
What will happen?
Predictive
Analytics
Prescriptive
Analytics
How can we make it happen?
26. Exactly what is a data lake?
A storage repository, usually Hadoop, that holds a vast amount of raw data in its native format until
it is needed.
Reasons to use a data lake:
• Inexpensively store unlimited data
• Centralized place for multiple subjects (single version of the truth)
• Collect all data “just in case” (data hoarding). The data lake is a good place for data that you “might” use down the road
• Easy integration of differently-structured data
• Store data with no modeling – “Schema on read”
• Complements enterprise data warehouse (EDW)
• Frees up expensive EDW resources for queries instead of using EDW resources for transformations (avoiding user contention)
• Wanting to use technologies/tools (i.e Databricks) to refine/filter data that do the refinement quicker/better than your EDW
• Quick user access to data for power users/data scientists (allowing for faster ROI)
• Data exploration to see if data valuable before writing ETL and schema for relational database, or use for one-time report
• Allows use of Hadoop tools such as ETL and extreme analytics
• Place to land IoT streaming data
• On-line archive or backup for data warehouse data
• With Hadoop/ADLS, high availability and disaster recovery built in
• It can ingest large files quickly and provide data redundancy
• ELT jobs on EDW are taking too long because of increasing data volumes and increasing rate of ingesting (velocity), so offload some of them to the Hadoop data lake
• Have a backup of the raw data in case you need to load it again due to an ETL error (and not have to go back to the source). You can keep a long history of raw data
• Allows for data to be used many times for different analytic needs and use cases
• Cost savings and faster transformations: storage tiers with lifecycle management; separation of storage and compute resources allowing multiple instances of different
sizes working with the same data simultaneously vs scaling data warehouse; low-cost storage for raw data saving space on the EDW
• Extreme performance for transformations by having multiple compute options each accessing different folders containing data
• The ability for an end-user or product to easily access the data from any location
27. Data Warehouse
Serving, Security & Compliance
• Business people
• Low latency
• Complex joins
• Interactive ad-hoc query
• High number of users
• Additional security
• Large support for tools
• Dashboards
• Easily create reports (Self-service BI)
• Know questions
28. Why Data Warehouses fail
• 70-80% corporate BI projects fail (Gartner)
• The executive who thinks data warehousing is “easy”
• Starting with an end date and working backwards
• Lack of user involvement
• Insufficient business requirements or too much
• Insufficient architect/design or too much
• Not enough planning for data governance
• Chose the wrong technology
29. Why Data Warehouses fail
• Poor communication between IT and the business
• Not setting up time to “validate the numbers”
• Performance issues (slow response times)
• Inexperience and lack of technical knowledge
• Modeling the data warehouse to reflect the source data rather than the
business
30. Why Data Warehouses fail
• The wrong consulting company is hired
• Lack of experience
• Using offshore developers
• Owning the whole project
• No mentoring or knowledge transfer
• Budget runs out
• Developers
• Testers
• DBA
• PM
• Cloud costs
• Software licenses
• Training
31. How DW project succeed
• Choosing the right technology
• Understand all aspects of data warehouse architectures
• Kimball vs Inmon
• Star Schema vs Relational
• Modern data warehouse vs data fabric vs data lakehouse vs data mesh
• Find a project champion/sponsor
• Have DW director who keeps everyone running at 80% efficiency
• Interative/Agile approach to delivery with frequent releases
• Break project into phases
• Start with a high-priority subject but one that is small and easy
• Get a quick win that shows lot’s of benefits and market it
• Build a solid foundation with a repeatable process
• Get people excited about DW thru training
• User Power BI to prototype/requirements
32. Modern Data Warehouse
Data Fabric adds: data access, data policies, data catalog, MDM, data virtualization,
data scientist tools, APIs, building blocks, products
36. Q & A ?
James Serra, Big Data Evangelist
Email me at: JamesSerra3@gmail.com
Follow me at: @JamesSerra
Link to me at: www.linkedin.com/in/JamesSerra
Visit my blog at: JamesSerra.com