Data Transfer Process

Agenda

1. Data Transfer Process

2. Monitoring
3. Delta
4. Error-Handling
5. Manual Update
6. Roadmap and Useful Transactions

© SAP 2008 / Page 2

 Data Transfer Process: Simple Example 1

BW

Process Chain

DataTransfer Process TRANSFORMATION

DataSource (PSA)

InfoPackage

SourceSystem 1

© SAP 2008 / Page 3

 Data Transfer Process: Simple Example 2

BW

Process Chain
TRANSFORMATION 1 TRANSFORMATION 3

DataTransfer Process InfoSource (7.0) DataTransfer Process

TRANSFORMATION 2

DataSource (PSA)

InfoPackage

SourceSystem 1

© SAP 2008 / Page 4

 Data Transfer Process: Complex Example

TR
BW
InfoSource (7.0)
DTP
TR DTP
TR
DataStore Object 3

DTP TR TR DTP

DataStore Object 1 DataStore Object 2

DTP DTP
TR TR

DataSource (PSA) DataSource (PSA)

IP IP

SourceSystem 1 SourceSystem 2
© SAP 2008 / Page 5
 Benefits of DTP

Data Transfer Process (DTP): data ‘distribution’ within SAP BI

 Improved transparency of staging processes across data warehouse layers

(PSA, DWH layer, ODS layer, Architected Data Marts)
 Separation of delta mechanism for different data targets
 Improved performance: Optimized parallelization
 Enhanced filtering in dataflow
 Enhanced error handling via error stack and temporary data storage is
possible.

© SAP 2008 / Page 6

 Creating DTP

Once the transformation is defined, it is possible to create the Data

Transfer Process.
Header-Table for DTP: RSBKDTP

© SAP 2008 / Page 7

 Filter in DTP

Different data selections are

possible for one DataSource via
different DTPs
For the same DataTarget
For different DataTargets.

Extraction mode:
Delta or Full

© SAP 2008 / Page 8

 DTP for Open Hub Destination

Open Hub Destination also

possible as DTP Data
Target  InfoSpokes are
obsolete

© SAP 2008 / Page 9

 Agenda

1. Data Transfer Process

2. Monitoring
3. Delta
4. Error-Handling
5. Manual Update
6. Roadmap and Useful Transactions

© SAP 2008 / Page 10

 Monitor in general

 To run the Monitor Transaction RSMO or use the Button

 You can find both InfoPackage- and DTP-Monitor here

Or use Transaction
RSRQ to enter a DTP-
ID or DTP-SID directly

© SAP 2008 / Page 11

 The different tabs – Header

Shows
 Technical status
 source and target of DTP

Possibility to jump to
 Job overview
 Error stack
 Data Target Administration
 DataSource Administration
 Different analysis tools
which use the time frame
of the given request

Request-ID

© SAP 2008 / Page 12

 The different tabs – Details

Important for
runtime
analysis

© SAP 2008 / Page 13

 Temporary Storage for DTP

The Details-Tab also shows the temporary

storage for the different sub-steps of the
DTP.
 The DTP can be set up from the
temporary storage in case of a problem
(see „manual update“).
 You can view and verify the data in the
temporary storage in case of problems.

© SAP 2008 / Page 14

 Temporary Storage for DTP

Identifying
the valid time
for temporary
storage

Identifying
the detail
levels of
temporary
storage

Here you can

switch on / off the
temporary storage
for each substep of
the DTP process.

© SAP 2008 / Page 15

 Debugging possibilities

Switch Processing Mode to „Serially in the Dialog Process (for Debugging)“

allows to simulate the DTP execution
 Always the next set of data is simulated
 Various possibilities for breakpoints
 Even DataPackage number can be specified for single breakpoints

© SAP 2008 / Page 16

 Request Simulation along the time line

You can „rerun“ the protocol creation of a DTP request

 Shows only the monitor-steps which already happened at that time, allows
scrolling back and forward.
 Helpful for analysis of lock situations or parallel processing with different data
packages.

Detail-screen – Fixed –
Scroll back and forward
with the arrows

© SAP 2008 / Page 17

 Customizing

Customizing setting for

warning-messages is the
only one which is used in
DTP monitor.

Most customizing in SPRO is not used for DTP:

 DTP is executed via batch processes  No dialog timeouts anymore
 DTP is scheduled via process chain  Monitoring via CCMS, no monitor
assistant.
 Traffic light for „no data available“ is always green in DTP monitor.

© SAP 2008 / Page 18

 Agenda

1. Data Transfer Process

2. Monitoring
 Delta
 Error-Handling
 Manual Update
 Roadmap and Useful Transactions

© SAP 2008 / Page 19

 DTP Delta

Delta is determined by the updating DTP

 Delta is NOT determined anymore by the combination DataSource /
Sourcesystem
  Separation of delta mechanism for different data targets

DTP 1 DTP 2
Delta Delta

DataSource (PSA) Delta DataSource (PSA)

OLD Datamart-Handling: NEW DTP-Delta:

One Delta-Handling for each Separate Delta-Handling
DataSource / Sourcesystem for each DTP
combination
© SAP 2008 / Page 20
 Delta Determination

Delta Determination:
 Check table RSBKREQUEST for all green requests of a DTP
 Table RSBKREQUEST: Shows all DTP-Requests
USTATE (User-set status) = “2” means “processed successfully”

 Check table RSSTATMANREQMAP to find which of the Source-

Requests were already updated
 Table RSSTATMANREQMAP: Mapping Target-Request – Source-Request

  Next delta-request updates all Source-Requests which were not yet

updated before

© SAP 2008 / Page 21

 Reset Delta-Status

Reset Delta-Status:
 Delete Request(s) from DataTarget
 Deleted requests disappear from mapping-
table RSSTATMANREQMAP and therefore are
picked up with next delta-upload

 No other status has to be reset (like “DM-

Status” in BW 3.x)
 No „repeat“-request necessary, just normal
delta-request
DataTarget

Table RSSTATMANREQMAP

© SAP 2008 / Page 22

 Multiple DTPs for one transformation (I)

Multiple DTPs can be defined for one transformation:

 Full-Mode: No constraints
 Delta-Mode: Disjunctive selection criteria
 Up to now, separate Delta-Handling for each DTP
 Even if source, target and transformation are the same!

 In future, it will be possible to “merge” delta-DTPs with disjunctive selection

criteria, so that only one DTP is necessary for delta-processing of both DTPs.

Delta-DTP 1
Delta-DTP 1 Delta-DTP 2 Merged Selection:
Selection: Selection: Customer A,
Customer A Customer B customer B

DataSource (PSA)
DataSource (PSA)

© SAP 2008 / Page 23

 Multiple DTPs for one transformation (II)

 You can upload mixed full- / delta-requests to any DataTarget

 Even to DataStore-Objects
 “Full-Repair”-Request is not necessary anymore
  User has more responsibility which data is uploaded in which order!

 No “Reconstruct” possible anymore with DTP

 “Reconstruct”-tab does not show DTP-requests
 Not necessary due to separate delta-handling for each DataTarget

© SAP 2008 / Page 24

 Where-Used List

Available with SP07, the Source-Object offers a Where-Used List about

the requests updated to the connected DataTargets.

Source-Object:

The Where-Used List shows:

 To which DataTargets the
request was updated.
 Which DataMart-requests have
to be deleted before this
request can be deleted.
 Attention: deletes the
request itself, not just a status.
© SAP 2008 / Page 25
 Data-Consistency and Delta Invalidation (I)

To guarantee Data-Consistency and prevent Delta Invalidation:

 It is not possible to delete a source-request without deleting its DataMart-
requests before.
 PSA-requests (which often are the “source-request” of a DTP) can of course
be deleted.
 This is guaranteed with table RSMDATASTATE:
 When a Delta-DTP is started, pointer DMEXIST is increased to prevent the
deletion of the source-request.

Cube B
Request 1 Cube C
Request 2 Request 1

RSMDATASTATE:
InfoCube DMEXIST DMALL
Cube A 2 1

Cube A
Request 1
Request 2
© SAP 2008 / Page 26
 Data-Consistency and Delta Invalidation (II)

To guarantee Data-Consistency and prevent Delta Invalidation:

 When an InfoCube is source for a DTP, you can only compress requests which
are already updated in all connected DataMarts.
 This is guaranteed with table RSMDATASTATE:
 As long as there is still a Delta-DTP or Export-DataSource that has NOT yet
picked up the request, DMALL is not updated and the request cannot be
compressed.
 When ALL Delta-DTPs AND Export-DataSources have picked up the source-
request, the pointer DMALL is increased.

Cube B
Request 1 Cube C
Request 2 Request 1

RSMDATASTATE:
InfoCube DMEXIST DMALL

Cube A 2 1

Cube A
Request 1
Request 2
© SAP 2008 / Page 27
 Delta without Data Transfer

The delta-status for a DTP can be set to „already picked up“ for all
records in the Source-Object without transferring data.

© SAP 2008 / Page 28

 How to switch Delta from InfoPackage to
DTP?
The already-existing Delta data flow using InfoPackages can be switched
easily to the new Delta data flow using DTP without data loss.

© SAP 2008 / Page 29

 Agenda

1. Data Transfer Process

2. Monitoring
 Delta
 Error-Handling
 Manual Update
 Roadmap and Useful Transactions

© SAP 2008 / Page 30

 Error Handling Overview

DTP Scheduler

IP DTP
DataSource (PSA)

Source System

No error handling for

InfoPackages. Error DTP
Error Stack
In case of invalid records,
data needs to be reloaded
from the sourcesystem.

Invalid records can be corrected in the error

stack and updated into the data target
© SAP 2008 / Page 31
 Captured errors

 Un-allowed characteristic values

 Lower case letters
 Arithmetic and conversion errors
 User built routine with return-code <> 0
 Master data read unsuccessful
 Currency translation or time conversion error
 Checks during Master data and Text update
 No SID for navigational attribute
 No language in text upload
 Double records concerning the key
 Overlapping or invalid time intervals
 Data does not map with the scheduler selection
 “Do not update, when no master data exists”
 Errors in Hierarchy structure
 Overlapping time intervals
 No SID for characteristic values

© SAP 2008 / Page 32

 Error Handling

Error Handling – Possibility to choose in the scheduler:

 1. option: Once errors occur, the whole data package is
terminated. The request is not released for reporting.
 2. option: Valid records are updated. After manual release of
the request, data is valid for reporting.
 3. option: Valid records are updated and available for
reporting.

© SAP 2008 / Page 33

 Error Handling

You can define the Number of wrong records which lead to a

wrong request.

If you select the indicator ‘No Aggregation Allowed’, the request is

regarded as incorrect if the number of records received in BW
does not match the number of updated records.

© SAP 2008 / Page 34

 Data Transfer Process Monitor

Data display in Error-

Stack

Data display in
temporary storage

© SAP 2008 / Page 35

 Temporary Storage and Error Stack

© SAP 2008 / Page 36

 How to reload from Error Stack

After the correction of the data in the Error Stack, the data can be
reloaded using the “Error DTP”.

The “Error DTP” has to be created once in the original DTP, can be used
like a normal DTP afterwards.

© SAP 2008 / Page 37

 Error Handling with DataStore Objects

Error-Handling is now also possible for DataStore Objects:

 In BW 3.x, this was not possible for ODS-Objects. Due to the
„Overwrite“-functionality, data consistency could not be guaranteed.
 Now new records with the same key as the erroneous ones are also
filtered out, when you define a key for the error stack.

© SAP 2008 / Page 38

 The Key of Error Stack

Choose key of error stack as detailed as possible

 Maximum number of key fields: 16
 A detailed key leads to a small amount of entries in the error stack.

Only the key fields of the DataStore Object can be defined as key for
the error stack (all key fields or only some of them).

© SAP 2008 / Page 39

 Example: Error Stack (I)

The following data records will be written into Error Stack:

 Incorrect data records
 Records with the same key of error stack within the same data package and
same request, that come after the error record.

d
recor
Request Data Order Calendar Day Quantity
r e c t Error Stack
Incor
number Rec. number
Key: Order Number
1 01 1000 01.07.2005 %&§
c o r d s with
Re s
key a
1 02 1000 03.07.2005 10 same ct ones
re
1 03 1000 02.07.2005 70 incor
1 04 1001 04.07.2005 20
Corre
1 05 1002 01.07.2005 30 ct rec
ords

© SAP 2008 / Page 40

 Example: Error Stack (Ia) – One request

Assumption
 One record is marked red in a custom-developed transformation routine
 The key of the error stack is order number

Result
 Records 2 and 3 will be written to the error stack

Error Stack
Request Data Order Calendar Day Qty
number Rec. number

109882 01 1000 01.07.2005 30

109882 02 20050703 1000 50
109882 02 1000 03.07.2005 50 109882 03 20050702 1000 70

109882 03 1000 02.07.2005 70

© SAP 2008 / Page 41

 Example: Error Stack (II)

The following data records will be written into Error Stack:

 Incorrect data records
 Records with the same key of error stack within the same data package and same
request, that come after the error record.
 Records with the same key of error stack within different requests

Request Data Order Calendar Quantity

number Rec. number Day
Incorrect record
1 01 1000 01.07.2005 %&§ Error Stack
e
s am nt Key: Order Number
h
Request Data Order Calendar Quantity
d wit iffere
number Rec. number Day r
co thin d
R e
wi
2 01 1000 03.07.2005 50
key uests
req
2 02 1001 01.08.2005 20

Correct records
2 03 1011 02.08.2005 10

Once the request in DataStore Object is deleted, the related

data records in error stack are automatically deleted.

© SAP 2008 / Page 42

 Example: Error Stack (IIb) – More requests

Result
 As the first request for order number 1000 contains an error, all records with the same
key of the same request and all records with the same key in subsequent requests will
be written to the error stack.

Request Data Order Calendar Day Qty

number Rec. number Error Stack

109882 01 1000 01.07.2005 30

109882 02 1000 03.07.2005 50 109882 02 20050703 1000 50

109883 01 20050702 1000 70
Request Data Order Calendar Day Qty
number Rec. number

109883 01 1000 02.07.2005 70

© SAP 2008 / Page 43

 Error Handling

Differences between error-handling variants:

Monitor Abort of Upd. valid Marked in Update Color of Request

entry update records tem. into Error
storage Stack

No Error handling
X X red
Error handling

No update, no
reporting X X X red

Update valid
records, no X X X X
reporting red

Update valid
records, X X X X
Reporting
possible
green

© SAP 2008 / Page 44

 Error Handling - Summary

Possibility to choose in the scheduler:

 To abort process when errors occur
 To process the correct records but do not allow reporting on them
 To process the correct records and allow reporting on them
 The Number of wrong records which lead to a wrong request

Error-Handling can be used for DataStore Object and Delta now!

Invalid records can be written into an error stack.

Keys should be defined for error stack to enable the error handling of
DataStore Object.

Temporary data storage can be switched on/off for each sub-step of the
loading process.

Invalid records can be updated into data targets after their correction.

© SAP 2008 / Page 45

 Agenda

1. Data Transfer Process

2. Monitoring
 Delta
 Error-Handling
 Manual Update
 Roadmap and Useful Transactions

© SAP 2008 / Page 46

 Manual Posting from Monitor

Possibility to reload a red request from the

monitor without deleting the request in the
DataTarget
 Reloads always the complete request

© SAP 2008 / Page 47

 Manual Posting from Monitor

The processing mode „Serially in Dialog Process“

indicates that the request was posted manually.

© SAP 2008 / Page 48

 When can a request be posted manually?

Two scenarios where a request can be posted manually:

Scenario 1:
 The Source is a PSA-request, which is still available.
 The DataPackages are not changed by the DTP, so that the system knows before
the extraction which DTP-package contains which source-package.
  Then the PSA-packages can be read selectively by the manual posting.
 This does NOT work with semantic groups (error stack, master data upload).

Scenario 2:
 The Source is an InfoCube or DataStore Object.
 The main process has finished and created a temporary storage of each
package.
  Then the manual posting is restarted from the temporary storage.

© SAP 2008 / Page 49

 Request Reverse Posting

Advantage Disadvantage

Delete Request  More flexible (also  Either deactivate

possible for ODS, no compression of
PSA necessary) aggregate after rollup or
deletion of aggregates
necessary

Request Reverse  Possible for already  Not possible for

Posting compressed requests in DTP-Requests
aggregates
 Restricted to InfoCubes
 Just for upload via PSA
 No further development
on this feature

© SAP 2008 / Page 50

 Agenda

1. Data Transfer Process

2. Monitoring
 Delta
 Error-Handling
 Manual Update
 Roadmap and Useful Transactions

© SAP 2008 / Page 51

 Steps in Loading Process

Infopackage
Scheduling 1
10
Update Rules

Communication structure

Transfer Rules 9
PSA
ALE ALE Transfer Structure
ALE SA P BW

tRFC 2 tRFC 8
3 ALE 7 ALE Transfer Structure SA P-
Sy ste
SAPI 4 5 SAPI
6 m
Extraction job

© SAP 2008 / Page 52

 Steps to analyze Loading Problems (1)

1. Load only to PSA / check whether data arrived in

PSA

YES NO

Conclusion: Conclusion: 1 5
9 Cause of problem Problem in source 2 6
10
is in BW-system system 3 7
or communication 4 8
between systems
Analyze monitor
messages, transfer
rules, update rules Analyze source system
(see simulation slides ) (next slides)

© SAP 2008 / Page 53

 Steps to analyze loading problems (2)

No data arrived in PSA :

 Check extraction job in SM37 (in case of SAP source system)
 Naming convention of job:
BI + <Requestid> or BI_PROCESS_LOADING

Monitor in BW-system SAP source system

© SAP 2008 / Page 54
 Extraction job

Possibilities
 There is no job:
1 2 3 4

 The job
cancelled:
5

 The job ran

successful, but
no data in PSA:
6 7 8

© SAP 2008 / Page 55

 Request error analysis (1)

1  Check for short-dumps in BW (ST22)

 Check selection routines / selection in
InfoPackage

2  Check tRFC queue (SM58) in BW

system for entries and errors

© SAP 2008 / Page 56

 Request error analysis (2)

3 4  Check source system connection in

RSA1
 Check that the right destinations are
entered in both systems (SM59)
 Check for not processed IDocs (BD87)
in both systems (SAP Note 555229)
 Check that remote-user in R/3 has
enough authorities for background job
processing and extraction.

© SAP 2008 / Page 57

 Request error analysis (3)

5  Check extraction job log (SM37) for

error messages ( shortdump, syslog)
 Extractor checker (RSA3)
6  Check entries in Delta Queue (RSA7),
see also SAP Note 380078
7  Check source system connection in
RSA1
 Check that the right destinations are
entered in both systems (SM59)
 Check for not processed IDocs (BD87)
in both systems (SAP Note 555229)

© SAP 2008 / Page 58

 Request error analysis (4)

8  Check tRFC queue (SM58) in source

system for entries and errors

9 10  Check in BW for dumps and syslog

entries
 Analyze BW Monitor messages
 Use simulation

© SAP 2008 / Page 59

 Thank you!

© SAP 2008 / Page 60

 Copyright 2008 SAP AG
All rights reserved

No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP AG. The information contained herein may be changed
without prior notice.
Some software products marketed by SAP AG and its distributors contain proprietary software components of other software vendors.
SAP, R/3, mySAP, mySAP.com, xApps, xApp, SAP NetWeaver, Duet, Business ByDesign, ByDesign, PartnerEdge and other SAP products and services mentioned herein as well as their
respective logos are trademarks or registered trademarks of SAP AG in Germany and in several other countries all over the world. All other product and service names mentioned and
associated logos displayed are the trademarks of their respective companies. Data contained in this document serves informational purposes only. National product specifications may vary.

The information in this document is proprietary to SAP. This document is a preliminary version and not subject to your license agreement or any other agreement with SAP. This document
contains only intended strategies, developments, and functionalities of the SAP® product and is not intended to be binding upon SAP to any particular course of business, product strategy,
and/or development. SAP assumes no responsibility for errors or omissions in this document. SAP does not warrant the accuracy or completeness of the information, text, graphics, links, or
other items contained within this material. This document is provided without a warranty of any kind, either express or implied, including but not limited to the implied warranties of
merchantability, fitness for a particular purpose, or non-infringement.
SAP shall have no liability for damages of any kind including without limitation direct, special, indirect, or consequential damages that may result from the use of these materials. This limitation
shall not apply in cases of intent or gross negligence.
The statutory liability for personal injury and defective products is not affected. SAP has no control over the information that you may access through the use of hot links contained in these
materials and does not endorse your use of third-party Web pages nor provide any warranty whatsoever relating to third-party Web pages

Weitergabe und Vervielfältigung dieser Publikation oder von Teilen daraus sind, zu welchem Zweck und in welcher Form auch immer, ohne die ausdrückliche schriftliche Genehmigung durch
SAP AG nicht gestattet. In dieser Publikation enthaltene Informationen können ohne vorherige Ankündigung geändert werden.
Einige von der SAP AG und deren Vertriebspartnern vertriebene Softwareprodukte können Softwarekomponenten umfassen, die Eigentum anderer Softwarehersteller sind.
SAP, R/3, mySAP, mySAP.com, xApps, xApp, SAP NetWeaver, Duet, Business ByDesign, ByDesign, PartnerEdge und andere in diesem Dokument erwähnte SAP-Produkte und Services
sowie die dazugehörigen Logos sind Marken oder eingetragene Marken der SAP AG in Deutschland und in mehreren anderen Ländern weltweit. Alle anderen in diesem Dokument erwähnten
Namen von Produkten und Services sowie die damit verbundenen Firmenlogos sind Marken der jeweiligen Unternehmen. Die Angaben im Text sind unverbindlich und dienen lediglich zu
Informationszwecken. Produkte können länderspezifische Unterschiede aufweisen.

Die in diesem Dokument enthaltenen Informationen sind Eigentum von SAP. Dieses Dokument ist eine Vorabversion und unterliegt nicht Ihrer Lizenzvereinbarung oder einer anderen
Vereinbarung mit SAP. Dieses Dokument enthält nur vorgesehene Strategien, Entwicklungen und Funktionen des SAP®-Produkts und ist für SAP nicht bindend, einen bestimmten
Geschäftsweg, eine Produktstrategie bzw. -entwicklung einzuschlagen. SAP übernimmt keine Verantwortung für Fehler oder Auslassungen in diesen Materialien. SAP garantiert nicht die
Richtigkeit oder Vollständigkeit der Informationen, Texte, Grafiken, Links oder anderer in diesen Materialien enthaltenen Elemente. Diese Publikation wird ohne jegliche Gewähr, weder
ausdrücklich noch stillschweigend, bereitgestellt. Dies gilt u. a., aber nicht ausschließlich, hinsichtlich der Gewährleistung der Marktgängigkeit und der Eignung für einen bestimmten Zweck
sowie für die Gewährleistung der Nichtverletzung geltenden Rechts.
SAP übernimmt keine Haftung für Schäden jeglicher Art, einschließlich und ohne Einschränkung für direkte, spezielle, indirekte oder Folgeschäden im Zusammenhang mit der Verwendung
dieser Unterlagen. Diese Einschränkung gilt nicht bei Vorsatz oder grober Fahrlässigkeit.
Die gesetzliche Haftung bei Personenschäden oder die Produkthaftung bleibt unberührt. Die Informationen, auf die Sie möglicherweise über die in diesem Material enthaltenen Hotlinks
zugreifen, unterliegen nicht dem Einfluss von SAP, und SAP unterstützt nicht die Nutzung von Internetseiten Dritter durch Sie und gibt keinerlei Gewährleistungen oder Zusagen über
Internetseiten Dritter ab.
Alle Rechte vorbehalten.

© SAP 2008 / Page 61