Professional Documents
Culture Documents
on BI Projects
Mark Rittman, CTO, Rittman Mead
Dutch Oracle Users Group, Jan 14th 2015
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or
+61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : info@rittmanmead.com
W : www.rittmanmead.com
E : info@rittmanmead.com
W : www.rittmanmead.com
E : info@rittmanmead.com
W : www.rittmanmead.com
Agenda
Part 1 : The Hadoop (BDA) technical stack for Oracle BI/DW projects
Why are Oracle BI/DW customers adopting Hadoop (BDA) technologies?
What are the Oracle and Cloudera products being used?
New Oracle products on the roadmap - Big Data Discovery, Big Data SQL futures
Where does OBIEE, ODI etc fit in with these new products
Rittman Meads development platform
Part 2 : Rittman Mead Hadoop (BDA) + Oracle BI Project Experiences
What is Cloudera CDH, and the BDA, like to work with?
How do we approach projects and PoCs?
What architecture and approach do we actually take, now?
How well do OBIEE and ODI work with Hadoop and BDA?
What are the emerging techs, products and architectures we see for 2015+?
E : info@rittmanmead.com
W : www.rittmanmead.com
Part 1 :
The Hadoop (BDA) technical stack for Oracle BI/DW projects
or How did we get here?
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or
+61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : info@rittmanmead.com
W : www.rittmanmead.com
E : info@rittmanmead.com
W : www.rittmanmead.com
E : info@rittmanmead.com
W : www.rittmanmead.com
E : info@rittmanmead.com
W : www.rittmanmead.com
E : info@rittmanmead.com
W : www.rittmanmead.com
E : info@rittmanmead.com
W : www.rittmanmead.com
E : info@rittmanmead.com
W : www.rittmanmead.com
and now
this happened
E : info@rittmanmead.com
W : www.rittmanmead.com
E : info@rittmanmead.com
W : www.rittmanmead.com
Actionable
Insights
Actionable
Information
Structured
Enterprise
Data
Input
Events
Data
Reservoir
Event Engine
Data Factory
Enterprise
Information Store
Execution
Innovation
Events
& Data
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or
+61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
Discovery Lab
E : info@rittmanmead.com
W : www.rittmanmead.com
Discovery
Output
Reporting
Other
Data
Enterprise
Performance
Management
Access & Performance Layer
Data Ingestion
Operational Data
COTS Data
Master & Ref. Data
Streaming & BAM
SMS
Docs
Pre-built &
Ad-hoc
BI Assets
Information
Services
Information Interpretation
Virtualization &
Query Federation
E : info@rittmanmead.com
W : www.rittmanmead.com
Data
Science
E : info@rittmanmead.com
W : www.rittmanmead.com
E : info@rittmanmead.com
W : www.rittmanmead.com
E : info@rittmanmead.com
W : www.rittmanmead.com
Exadata Database
Server
SmartScan
Exadata
Storage Servers
E : info@rittmanmead.com
W : www.rittmanmead.com
SmartScan
Hadoop
Cluster
Oracle Big
Data SQL
E : info@rittmanmead.com
W : www.rittmanmead.com
E : info@rittmanmead.com
W : www.rittmanmead.com
E : info@rittmanmead.com
W : www.rittmanmead.com
E : info@rittmanmead.com
W : www.rittmanmead.com
E : info@rittmanmead.com
W : www.rittmanmead.com
Business Presentation
Layer (Reports, Dashboards)
Enterprise Semantic
Business Model
Hadoop /
NoSQL
Sources
Application
Sources
DW / OLAP Sources
E : info@rittmanmead.com
W : www.rittmanmead.com
E : info@rittmanmead.com
W : www.rittmanmead.com
E : info@rittmanmead.com
W : www.rittmanmead.com
E : info@rittmanmead.com
W : www.rittmanmead.com
E : info@rittmanmead.com
W : www.rittmanmead.com
E : info@rittmanmead.com
W : www.rittmanmead.com
E : info@rittmanmead.com
W : www.rittmanmead.com
E : info@rittmanmead.com
W : www.rittmanmead.com
E : info@rittmanmead.com
W : www.rittmanmead.com
E : info@rittmanmead.com
W : www.rittmanmead.com
vmhost3
vmhost2
ODI12c
OBIEE 11g
BI Apps 11g
LDAP
VCenter
vmhost4
vmhost5
OS X Server
DNS etc
EM 12c R4
KDC
E : info@rittmanmead.com
W : www.rittmanmead.com
Cluster Management
VMWare VSphere 5 + VCenter Server
Oracle Enterprise Manager 12cR4 Cloud Control
OSX Server Yosemite
E : info@rittmanmead.com
W : www.rittmanmead.com
E : info@rittmanmead.com
W : www.rittmanmead.com
Part 1 :
Rittman Mead Hadoop (BDA) + Oracle BI
Project Experiences
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or
+61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : info@rittmanmead.com
W : www.rittmanmead.com
E : info@rittmanmead.com
W : www.rittmanmead.com
E : info@rittmanmead.com
W : www.rittmanmead.com
E : info@rittmanmead.com
W : www.rittmanmead.com
E : info@rittmanmead.com
W : www.rittmanmead.com
E : info@rittmanmead.com
W : www.rittmanmead.com
E : info@rittmanmead.com
W : www.rittmanmead.com
Hive Metastore /
HCatalog
RDBMS
Imports
Real-Time
Logs / Events
File /
Unstructured
Imports
E : info@rittmanmead.com
W : www.rittmanmead.com
E : info@rittmanmead.com
W : www.rittmanmead.com
E : info@rittmanmead.com
W : www.rittmanmead.com
E : info@rittmanmead.com
W : www.rittmanmead.com
E : info@rittmanmead.com
W : www.rittmanmead.com
Actionable
Insights
Actionable
Information
Structured
Enterprise
Data
Input
Events
Data
Reservoir
Event Engine
Data Factory
Enterprise
Information Store
Execution
Innovation
Events
& Data
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or
+61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
Discovery Lab
E : info@rittmanmead.com
W : www.rittmanmead.com
Discovery
Output
Reporting
Other
Data
E : info@rittmanmead.com
W : www.rittmanmead.com
E : info@rittmanmead.com
W : www.rittmanmead.com
E : info@rittmanmead.com
W : www.rittmanmead.com
E : info@rittmanmead.com
W : www.rittmanmead.com
Discovery phase
Exploitation phase
Data Export
Batch via Sqoop
cmd-line exec
Sharing output via Hive
tables, Impala tables,
HDFS files etc
E : info@rittmanmead.com
W : www.rittmanmead.com
E : info@rittmanmead.com
W : www.rittmanmead.com
Loading
Stage
Real-Time
Logs / Events
File /
Unstructured
Imports
E : info@rittmanmead.com
W : www.rittmanmead.com
E : info@rittmanmead.com
W : www.rittmanmead.com
E : info@rittmanmead.com
W : www.rittmanmead.com
NoSQL Databases
Family of database types that reject tabular storage,
SQL access and ACID compliance
Useful as a way of landing data quickly + supporting
random cell-level access by ETL process
Focus is on scalability, speed and schema-on-read
Oracle NoSQL Database - speed and scalability
Apache HBase - speed, scalability and Hadoop
MongoDB - native storage of JSON documents
May or may not run on Hadoop, but associated with it
Great choice for high-velocity data capture
CRUD approach vs write-once/read many in HDFS
E : info@rittmanmead.com
W : www.rittmanmead.com
E : info@rittmanmead.com
W : www.rittmanmead.com
E : info@rittmanmead.com
W : www.rittmanmead.com
E : info@rittmanmead.com
W : www.rittmanmead.com
E : info@rittmanmead.com
W : www.rittmanmead.com
E : info@rittmanmead.com
W : www.rittmanmead.com
1
3
E : info@rittmanmead.com
W : www.rittmanmead.com
E : info@rittmanmead.com
W : www.rittmanmead.com
E : info@rittmanmead.com
W : www.rittmanmead.com
E : info@rittmanmead.com
W : www.rittmanmead.com
Hadoop
HDFS etc
Impala
Hadoop
HDFS etc
E : info@rittmanmead.com
W : www.rittmanmead.com
BI Server
Presentation Svr
Cloudera Impala
ODBC Driver
Impala
Impala
Hadoop
Hadoop
HDFS etc
HDFS etc
Impala
Hadoop
HDFS etc
Multi-Node
Hadoop Cluster
E : info@rittmanmead.com
W : www.rittmanmead.com
E : info@rittmanmead.com
W : www.rittmanmead.com
Loading
Stage
Processing
Stage
Real-Time
Logs / Events
Store / Export
Stage
File
Exports
RDBMS
Exports
File /
Unstructured
Imports
E : info@rittmanmead.com
W : www.rittmanmead.com
E : info@rittmanmead.com
W : www.rittmanmead.com
E : info@rittmanmead.com
W : www.rittmanmead.com
SmartScan
Oracle Big
Data SQL
DATABASE_NAME
-----------------------------default
default
default
default
default
default
default
default
TABLE_NAME
-----------------------------access_per_post
access_per_post_categories
access_per_post_full
apachelog
categories
countries
cust
hive_raw_apache_access_log
E : info@rittmanmead.com
W : www.rittmanmead.com
E : info@rittmanmead.com
W : www.rittmanmead.com
E : info@rittmanmead.com
W : www.rittmanmead.com
E : info@rittmanmead.com
W : www.rittmanmead.com
E : info@rittmanmead.com
W : www.rittmanmead.com
Combined output
in report form
E : info@rittmanmead.com
W : www.rittmanmead.com
E : info@rittmanmead.com
W : www.rittmanmead.com
E : info@rittmanmead.com
W : www.rittmanmead.com
E : info@rittmanmead.com
W : www.rittmanmead.com
E : info@rittmanmead.com
W : www.rittmanmead.com
E : info@rittmanmead.com
W : www.rittmanmead.com
E : info@rittmanmead.com
W : www.rittmanmead.com
E : info@rittmanmead.com
W : www.rittmanmead.com
E : info@rittmanmead.com
W : www.rittmanmead.com
E : info@rittmanmead.com
W : www.rittmanmead.com
E : info@rittmanmead.com
W : www.rittmanmead.com
movieapp_log_odistage.custid = CUSTOMER.CUSTID
E : info@rittmanmead.com
W : www.rittmanmead.com
E : info@rittmanmead.com
W : www.rittmanmead.com
Sqoop extract
movieapp_log_odistage.custid =
customer.custid
E : info@rittmanmead.com
W : www.rittmanmead.com
E : info@rittmanmead.com
W : www.rittmanmead.com
2
Register in ODI Model
E : info@rittmanmead.com
W : www.rittmanmead.com
E : info@rittmanmead.com
W : www.rittmanmead.com
1
Register in ODI Model
4
Hive table AP uses LKM Hive to Oracle (Big Data SQL)
2
IKM Oracle Insert
E : info@rittmanmead.com
W : www.rittmanmead.com
E : info@rittmanmead.com
W : www.rittmanmead.com
E : info@rittmanmead.com
W : www.rittmanmead.com
E : info@rittmanmead.com
W : www.rittmanmead.com
E : info@rittmanmead.com
W : www.rittmanmead.com
Summary
Hadoop and Oracle Big Data Appliance are increasingly appearing in BI+DW Projects
Gives DW projects the ability to store more data, cheaper and more flexibly than before
Enables non-relational (SQL) query tools and analysis techniques (R, Spark etc)
Extends BIs capability to report and analyze across wider data sources
Maturity varies widely in terms of tool maturity, and Oracle integration with Hadoop
Trend is for Oracle to productize big data, creating tools + products around Oracle BDA
We are probably at early stages - but very interesting times to be an Oracle BI+DW dev!
E : info@rittmanmead.com
W : www.rittmanmead.com
E : info@rittmanmead.com
W : www.rittmanmead.com
E : info@rittmanmead.com
W : www.rittmanmead.com