Copyright 2010, Oracle. All rights reserved.

Oracle Data Integrator

Overview

<Insert Picture Here>

PTS Workshop: ODI 11g

Copyright 2010, Oracle. All rights reserved.

Data Integration Overview

ODI: Key Benefits
ODI at Oracle
Q&A

Agenda
<Insert Picture Here>

Copyright 2010, Oracle. All rights reserved.

Oracle Data Integration

Orchestrate processes using SOA

Expose data services for reuse

Actively govern most valuable asset

Ensure the quality of your data

Access data in real time

Ensure continuous uptime

Leverage skills & dictate patterns

Design metadata-driven integration

Copyright 2010, Oracle. All rights reserved.

4. Automate and Adapt

Business Processes

3. Use Data for Competitive

Advantage

2. Compete Globally 24X7

1. Do More with Less

Add Value to the Core Business Lines

Business Drivers for Data Integration

Copyright 2010, Oracle. All rights reserved.

Migrations and Master Data Management

SOA, Enterprise Integration & Modernization

Key Data Integration Use Cases

BI, DW, and OLTP Data Integration & Replication

Central point for data governance, availability and controls

Common technical foundation platform across data silos

Strategic Value of Data Integration

Consistency for major enterprise initiatives like BI, DW, & MDM

Essential Ingredient for Information Agility

Project Drivers for Data Integration

Business
Intelligence

Flat Files

SOA
Platforms

Web 2.0

Web and Event

Services, SOA

Match and Merge

Data Cleansing

Data Parsing

Data Profiling

Oracle Data Quality

Data Federation

Activity
Monitoring

Copyright 2010, Oracle. All rights reserved.

OLAP Cube

Data Verification

Data Lineage

OLTP
System

Bi-directional Replication

Bulk Data Movement

Data Warehouse/
Data Mart

Log-based CDC

Data Transformation

Storage

Real-time Data

ELT/ETL

Data Services

Oracle GoldenGate

Service Bus

Oracle Data Integrator

Process Manager

MDM
Applications

Comprehensive Data Integration Solution

Custom
Applications

SOA Abstraction Layer

Oracle
Applications

Best-in-class Heterogeneous Platform for Data Integration

Oracle Data Integration Solution

OLTP DB
Sources

Application
Sources

Legacy
Sources

Copyright 2010, Oracle. All rights reserved.

Heterogeneous, Open and

Standards-based

Faster time-to-value through

Declarative Design

Lightweight Deployment for

Lowering TCO

Best in class for Performance,

Scalability, Availability

E-LT Transformation
vs. E-T-L

Any
Planning
System

Any Data
Warehouse

Optimized E-LT for improved Performance, Productivity and Lower TCO

Oracle Data Integrator Enterprise Edition

Heterogeneous
Source Systems

OGG

Log Based, RealTime Change Data

Capture

Highly Flexible
Fast Deployments
Lower TCO & Improved ROI

Standardize on Single
Technology for Multiple Needs
Deploy for Continuous
Availability and Real-time Data
Access for Reporting / BI

Copyright 2010, Oracle. All rights reserved.

Data Distribution

EDW

EDW

Database

Reporting

Query Offloading

Real-time BI

ETL

ODS

ETL

Operational
Reporting

Zero Downtime
Migration and
Upgrades

Standby

(Open & Active)

Disaster Recovery,
Data Protection

Enterprise-wide Solution for Real Time Data Needs

Oracle GoldenGate Overview

Data Profiling

Statistical Analysis
Rule-based Validation
Monitoring & Timeslice
Fine-grained Auditing

Data Profiling

E-LT & ETL

Data Transformation
Change Data Capture
Data Access
Data Services

Data Validation during ETL

Data Standardization
Address Matching & Dedup
Error Hospital / Workflow

Data Cleansing

Data Cleansing

Copyright 2010, Oracle. All rights reserved.

Data Integration

Data Movement

Data Movement

Govern Data Better with Data Quality

Shorten Time to Value

Lower TCO

Improve Performance
and Scalability

Design Themes

Copyright 2010, Oracle. All rights reserved.

6. Embedded API for applications

5. New connectivity to OLAP, SAP

4. Enterprise Manager

3. Unified JDev environment

2. Datasource connection pooling

1. Integrated with Application Grid

Whats New in Oracle Data Integrator 11g

OWB
10gR1

OWB
10gR2

CY
2009

ODI
10gR3

OWB
11gR1

Unified
Team

ODI
11gR2

CY
2011

Natural Upgrade Path

No Forced Migrations

OWB/ODI Investments are

Fully Protected

Unified Platform aims to be

a Superset of Existing
Products no regression

Unified
Platform

Copyright 2010, Oracle. All rights reserved.

CY
2010

ODI
11gR1

OWB
11gR2

CY2009

Natural Upgrade Path for Existing Solutions

High-Level Data Integration Roadmap

 KM Enhancements

Copyright 2010, Oracle. All rights reserved.

Monitor an Enterprise Architecture

EM Integration

Manage the erroneous rows

Error Table Management

Help troubleshooting/fixing

Better Exceptions

Externalized Authentication + SSO support

JPS Integration

Parallelism, Restartability, Concurrent Task Management, Task

Killing

Execution Management: Load Plans

Major Features

Directions for 11g

Copyright 2010, Oracle. All rights reserved.

Why is ODI Unique?

Proprietary ETL Engine

Poor Performance
High Costs for Separate Standalone Server
IBM & Informaticas approach
Transform

Load

Transform
Extract

Load

Transform

E-LT

Next Generation Architecture

Extract

Conventional ETL Architecture

Copyright 2010, Oracle. All rights reserved.

Optimal Performance & Scalability

Better Hardware Leverage

Easier to Manage & Lower Cost

Benefits

Lower Cost: Leverage Compute Resources &

Partition Workload efficiently
Efficient: Exploits Database Optimizer
Fast: Exploits Native Bulk Load & Other
Database Interfaces
Scalable: Scales as you add Processors to
Source or Target

Oracle: No New Servers

Conventional: Separate ETL Server

High Performance

Differentiator: E-LT Architecture

Automatically
Generate
Dataflow

Define How: Built-in Templates

You Want

What

Define

ODI Declarative Design

Conventional ETL Design

Copyright 2010, Oracle. All rights reserved.

Significantly reduce the learning curve

Shorter implementation times

Streamline access to non-IT pros

Benefits

Simplifies the number of steps

Automatically generates the Data Flow whatever
the sources and target DB

Oracle: Declarative Set-based Design

Developer must define every step of Complex

ETL Flow Logic
Traditional approach requires specialized ETL
skills
And significant development and maintenance
efforts

Conventional: Specify ETL Data Flow

Developer Productivity

Differentiator: Declarative Design

Journalize

Journalize
Read from CDC
Source

Load

Load
From Sources to
Staging

Benefits

TPump/
Multiload
Type II SCD

Check MS
Excel
Check
Sybase

JMS Queues

Oracle
SQL*Loader

Oracle
DBLink

DB2 Exp/Imp

SQL Server
Triggers

DB2 Journals

Log Miner

Error Tables

Siebel EIM
Schema

W
S

Oracle Web
Services

Services

W
W S
S

Service
Expose Data and
Transformation
Services

DB2 Web
Services

Oracle Merge

Target Tables

Integrate

Integrate
Transform and Move
to Targets

Copyright 2010, Oracle. All rights reserved.

Leverage Database Optimizations:

Native SQL; Native Functions; Native Loads; Native Journaling / CDC
Tailor to an organizations existing best practices
Ease administration work
Reduce cost of ownership

Siebel

SAP/R3

Check

Staging Tables

Check
Constraints before
Load

Sample out-of-the-box Knowledge Modules

Sources

CDC

Reverse

Reverse
Engineer Metadata

Pluggable Knowledge Modules Architecture

Hot-Pluggable, Modular, Extensible

Differentiator: Knowledge Modules

Copyright 2010, Oracle. All rights reserved.

(continuously refreshed throughout the day)

Key Use Case: Realtime Data Warehouse

Change Data Capture (CDC) allows for real time data to be available
for any type of data integration solution. Change Data Capture is
accomplished by capturing just the changed records and not the
full data set, dramatically reducing time and resources over the life
of the data integration solution.

Event-Driven Framework for Realtime Data Integration

Differentiator: Changed Data Capture

Application

Analytic BI
(Facts & Dims)

(Mini-Batch + Transforms)

E-LT

Consistency
Window

Copyright 2010, Oracle. All rights reserved.

Real Time BI
(using Replica)

Non-invasive Capture + E-LT Processing

Supply Real Time Business Intelligence

 Performance Management
Database & OLAP

Application Integration

Middleware Servers

ERP Migration, Replication / Loading

Query Offloading & Zero Downtime

Unified Data Steward Workflow

(ETL Error Hospital w/BPEL PM)

SOA-driven Business Intelligence

Load DW from SOA

Bulk Data Transformation (any2any)

XML/EDI Large File Handling

Copyright 2010, Oracle. All rights reserved.

Business Intelligence

E-LT Frameworks are optimal architectures for:

Data Warehouse
& OLAP

Embedded Applications

Any Data Source

High Performance
ETL & Replication

Common Runtime 100% Java

Common Monitoring

Example Use Cases

Unified Management + Monitoring

Best of Breed Data Integration as a Shared SOA Service

Unify E-LT Agent with SOA Runtime

Message
Duplicated Record
Duplicated Record
Invalid City
Reference

Id
001
022
230

City
New York
Boston
Maris

Declare constraints at table level

Design mappings and check flow
integrity
Audit, cleanse or recycle rejected
records

1.
2.
3.

Data Integrity Firewall

Auditing, cleansing and recycling

Copyright 2010, Oracle. All rights reserved.

Name
John Doe
John Doe
Albert Fresh

Inline E-LT Data Integrity Framework

Differentiator: Data Quality

Transformer

(*) Joint Development with Trillium

Global Data
Router
Parser

Relationship
Linker

Target

Best of breed Quality

Joint development with Trillium
Proven, scalable DQ engines
Rich global content for cleansing,
standardization, validation
Extensible by customers with
domain-specific Data Quality
functions
Out-of-box integration
ODI integrates with Quality functions
via pre-built Knowledge Modules
ODI Model metadata passed to
ODQ at design time
ODQ source and target Metadata
passed to ODI at Design Time
Out-of-box ODI Tool for runtime
invocation of ODQ processes
Packaged Quality Rules
Delivered Out-of-the-Box by Oracle
For 35 Countries & Domains
Copyright 2010, Oracle. All rights reserved.

Postal
Matcher

Parsing, Cleansing, Standardization,

Matching

Oracle Data Quality for Data Integrator (*)

Sources

Integration Process

Oracle Data Integrator

Oracle Data Quality for Data Integrator

Optional: Advanced Governance

Sources

Sales

BPEL Process Manager

Human
Workflow

Static
Check

Errors

Target

Error Recycling

Flow Check
During Integration

Integration
Process

Copyright 2010, Oracle. All rights reserved.

Manage Long Running Transactions

Manage Complex Approval and Overriding Patterns

Integrated with Human Workflow

BPEL Process Manager / BPEL4People

Reuse Standardized Error Handling Procedures

Standards-Based Process Flow

Static Constraint Checking and Validation

Flow-based Constraint Checking and Validation
Error Recycling

Flexible Data Correction Workflow

Oracle: Open, Flexible Workflow

Proprietary workflow baked-in

No workflow / 3rd party solution
Weak integration, non-standard

Conventional: Rigid/Non-Existent Workflow

Error Recycling, Native Integration with BPEL & Human Workflow

Differentiator: Flexible Error Workflow

Understand your data flows

Follow the path of data
Drill-down to transformations

Copyright 2010, Oracle. All rights reserved.

1.
2.
3.

Web-based end-to-end data lineage

Large number of data flows in a complex

environment
How to get an overview?

Differentiator: Data Lineage

Copyright 2010, Oracle. All rights reserved.

ODI at Oracle

Pre-built
Content

ODI
Embedded
Pre-built
Content

ODI
Embedded

BI Apps

Identity
Analytics

ODI
Embedded

ODI
Embedded

Fusion AIA
ODI
Embedded
Pre-built
Content

Fusion Pillar
Replication

OGG
Embedded

Pre-built
Content

Pre-built
Content

Planning

ODI Integrated

MDM/DRM

Pre-built
Content

ODI
Embedded

Spend
Analyzer

Pre-built
Content

ODI
Embedded

Fusion MDM

Pre-built
Content

ODI
Embedded

Fusion BI

Copyright 2010, Oracle. All rights reserved.

Pre-built
Content

ODI
Embedded

Fusion CDH

Oracle Fusion ERP Applications

ODI Integrated

Essbase

ODI Integrated

BIEE Suite

Business Intelligence

ID Mgmt

Enterprise Performance Management and Financials

ODI Integrated

BPEL &
OSB

HFM

Pre-built
Content

ODI
Embedded

BAM & CEP

SOA Suite

Pre-built
Content

Pre-built
Content

Pre-built
Content

ODI
Embedded

Fusion
Financials

Pre-built
Content

ODI
Embedded

Pre-built
Content

ODI / OGG
Embedded

Fusion
Migration

GoldenGate
Integrated

Siebel CRM

ODI
Embedded

ODI
Embedded

Governance
& Risk

Retek

Agile PLM

Other Apps (+more)

Pervasive Data Integration in the Oracle Ecosystem

ODI at Oracle

Oracle
GoldenGate

DIM

DW Schema

DIM

FACT

DIM

Copyright 2010, Oracle. All rights reserved.

DEPT

ODS Schema

EMP

DIM

Oracle Data
Integrator

Oracle Database

Extreme Exadata performance can handle huge data volumes that come with Real Time DW

New Exadata platform optimized for OLTP and Data Warehousing suits Real Time DW

Load Exadata with GoldenGate; Generate Transforms from ODI-EE

Exadata V2 is the Platform for Real Time Data Warehousing

Extreme Performance for Data Warehousing

7
1
3

10
3
7

Number of Setup Steps

Number of Servers
Number of connections

Number of Setup Steps

Number of Servers
Number of connections

33-50% Less Complex

Copyright 2010, Oracle. All rights reserved.

40% Efficiency Gains

E-LT uses Declarative SQL-style Design + Simple Runtime

Development Productivity
Environment Setup (ex: BI Apps)

Speed Project Delivery/Time to Market

Load

Heterogeneous

Bulk

Trickle

No mid-tier; transformations use

database (most efficient)
Mini-batches throughout the day orbulk processing nightly
Non-invasive data capture
Continuous low latency feeds from
operational systems

Modern E-LT + Real-time

Agent

Copyright 2010, Oracle. All rights reserved.

Lookup
Data

Lookup
Data

Staging

Xform

Xform

Extract

Transformations in ETL engine on

expensive middle tier servers
Bulk load to the data warehouse with
large nightly/daily batch
Invasive Capture on OLTP systems
using complex Adapters
Streaming requires several hops

Traditional ETL + CDC

Heterogeneous, Real-time, Non-Invasive, High Performance E-LT

Modern Data Integration Approach

Agent

Copyright 2010, Oracle. All rights reserved.

Copyright 2010, Oracle. All rights reserved.