Realtimedata Acess

Demonstration of Real-Time Data Acquisition (RDA) for SAP BI 7.
0 using Web Services API
Applies to:
SAP BI 7.0. For SAP 4.6C with plug-in 2004.1, this option has been available since Service pack 10.For more information, visit the Business Intelligence Homepage.
Summary
Real time data acquisition brings real time data to BI system from R/3 or Web services. RDA uses a program called daemon that controls data flow in BI 7.0 and takes care of extraction from source system. We can do reporting on large volume of data & large number of users. The data here is stored physically. This article mainly focuses on Real-time data acquisition of data from web services, and using WSADMIN to monitor the usage of daemons in handling them (via RPC).
Author:
Arun Bala Ganesan
Company: Wipro Technologies Created on: 16 January 2010
Author Bio
Arun Bala Ganesan currently works for Wipro Technologies as Associate SAP BI Developer. His expertise includes SAP Business Intelligence 7.0 and ABAP.
SAP COMMUNITY NETWORK 2010 SAP AG
SDN - sdn.sap.com | BPX - bpx.sap.com | BOC - boc.sap.com | UAC - uac.sap.com 1
Demonstration of Real-Time Data Acquisition (RDA) for SAP BI 7.0 using Web Services API
Table of Contents
Business Scenario .............................................................................................................................................. 3 RDA Vs Direct Access ........................................................................................................................................ 3 Basics of RDA ..................................................................................................................................................... 3 RDA Basics ......................................................................................................................................................... 3 Example: RDA using SAP Service API .......................................................................................................... 3 Existing Data Flow Concept Example ............................................................................................................. 4 New Data Flow Concept RDA using SAP Web Services API ......................................................................... 4 Constraints .......................................................................................................................................................... 5 Modeling View .................................................................................................................................................... 5 RDA Example Using Service API Administration View ................................................................................... 5 Types of RDA Mechanisms ................................................................................................................................ 6 RDA Implementation Scenario ........................................................................................................................ 7 Why use Delta? .................................................................................................................................................. 7 RDA Processing ................................................................................................................................................. 7 Daemons ......................................................................................................................................................... 7 Monitoring Daemons ....................................................................................................................................... 8 RDA in Extraction Layer ..................................................................................................................................... 8 RDA in BI Staging ............................................................................................................................................... 9 3 Requests in RDA ............................................................................................................................................. 9 Data source for RDA ......................................................................................................................................... 10 Info Package for RDA ....................................................................................................................................... 10 DTP for RDA ..................................................................................................................................................... 10 Controlling the RDA .......................................................................................................................................... 10 Daemon for RDA .............................................................................................................................................. 10 Step-by-Step Guide for RDA Implementation................................................................................................... 10 Steps: SOAP using WSA .................................................................................................................................. 11 Process Steps ................................................................................................................................................... 11 Implementation Results .................................................................................................................................... 28 Related Content ................................................................................................................................................ 29 Disclaimer and Liability Notice .......................................................................................................................... 30
Business Scenario
Some organizations require making decision based on real time up to-date details regarding certain process or product information. For Example: Stock markets, Communications, Defense, Aeronautics, Supply chain management etc. Here each of and every transaction data matters the most in making a decision at a priority level, and its very crucial to transfer data to the reporting side at shorter & regular intervals.
RDA Vs Direct Access

RDA: Extracts data from BW with a high frequency from source systems & migrates it to BI system. Direct Access: DA to source systems. Data is not physically migrated into BI, but read rights for data analysis are directly transferred to respective Data source
Basics of RDA
Real time data acquisition which supports tactical decision-making is a framework for deriving information from data as data becomes more complex. Time factors: Lower time scale than for scheduled/batch data acquisition Scalability: Stream oriented Frequency: Immediate availability for reporting
RDA Basics
Frequency is data transfer is quite high. Follows regular type of data transfer, where extracted data runs through extraction & transformation process. Although not true real-time, data transferred is physical data Example: RDA using SAP Service API We can also set the features for SOAP Runtime & publish a Web service (as a business service in the UDDI)
Fig. A. Existing Example: RDA using SAP Service API
Existing Data Flow Concept Example

The Fig. B illustrates the regular data flow example which we regularly follow to use.
Fig. B. Existing Data Flow Concept Example
New Data Flow Concept RDA using SAP Web Services API
Using new functionality of Real-time Data Acquisition (RDA) in BI 7.0 system we can now load transactional data into SAP BI system every single minute. The source system for RDA could be SAP System or it could be any non-SAP system, as most of the Standard Data Sources are real-time enabled.
Fig. C. New Data Flow Concept RDA using SAP Web Services API
Constraints
Real time enabled Data Sources could only be defined since the release of BI Service API of SAP Net Weaver 2004s. For SAP 4.6C with plug-in 2004.1, this option has been available since Service pack 10.
Modeling View
We must start the modeling by creating an Info package & DTP for Real time data acquisition, wherein the type of adapter used for both of them must be Web Service (Push). Refer Fig. D
Fig. D. Modeling View
RDA Example Using Service API Administration View

The following Fig. E illustrates the administration view, also better known as Real-time data acquisition using the transaction code RSRDA. Here we must first create a daemon in the RDA monitor, which will generate an open request waiting for transactional data from the SAP system. We must further enter data through a proper channel like say Web service API to trigger the data acquisition process.
Fig. E. Administration View

Note: Info packages for real time data acquisition can be defined with SAP source systems only if the delta procedure was initiated and data transfer process was created.
Types of RDA Mechanisms

There are 2 types of Real time data acquisition which are discussed as under, Pull: Its a request oriented, for long tern planning, scheduled at nights & consumes less resources. Push: For tactical decision making on a daily basis, information is processed every 1 hour or 1 min, its based on data availability, and resource needed is very high as its an active background job.
Note: Enhance established Data Flow with RDA capabilities [+] Implement additional DSO for operational reporting [+] Replace standard delta IP by RDA IP [+] Regular data loads can be scheduled after closing RDA IP Request using appropriate Process Chain feature [+] Typically data is deleted regularly from DSO supplied using RDA [+] Standard reporting can be enhanced by operational reporting using the report-report interface
RDA Implementation Scenario
Fig. F. RDA Implementation Scenario
Why use Delta?

In Extraction part of RDA, data sources are probably not ready to immediately supply data in small quantities, but in very large quantities instead. If several million records are transferred subsequently, latency exists. Ex: What if it takes several minutes to load data. So, in RDA, the concept of request generation or logging in BI needs to be changed. Data sources in source systems must be able to react as quickly as possible for the new requirements. Therefore data sources in RDA are delta enabled. Situations: Answer: If the data source is small, which one will you use? RDA or direct access? Direct Access
RDA Processing
Processing of RDA is done by daemons. (Refer Fig. H) Daemons Daemons originally stand for Disk And Execution MONitor in the UNIX. Each step is tracked in a control table, as and when its successfully executed. This also, allows restarting which can be initiated such that it starts at the next step after last successfully executed step. The BI Daemon data load includes three steps: Initiate BI Service-API data pull using Info Package for RDA into PSA (SAP source systems) Track status of data transfer from source system
Initiate update of Data Store Object using DTP
Daemons are system process to initiate data loads at regular intervals: from one minute to hourly basis. Monitoring Daemons In RSRDA, we can monitor status of daemons, check if a batch job is still active, running, or not. Sometimes, it might be stopped by an error or user itself. This monitor is useful for displaying runtime information about the daemons.
Fig. G. Types of Icons for daemon processes
Fig. H. Real Time Data Acquisitions monitoring using RSRDA
RDA in Extraction Layer

The core of RDA is the SAP Source systems. Other source systems can be linked to the concept of RDA only if they act as a Web service and transfer data via SOAP messages to SAP BI. (Refer Fig. I)
Figure I: RDA in Extraction Layer For SAP systems, the data sources that need to be connect by RDA need to be explicitly marked as 'Real time enabled'. For Generic Data sources, you can specify if a data source is able to handle accesses in realtime or not. The decision if a Generic Data source needs to be marked as real time enabled depends, how generic deltas are created. Condition 1: The delta defining fields should enable a high performance access to new/modified data. I.e. delta defining field must be selective & must be max. Affected by different values Condition 2: If there is a recent extraction, identification of the relevant data records from delta creation must contain only those recent data records.
Via the SOAP services, the SOAP messages are directly written to the PSA level of the data source. Hence, the messages don't find a way into BI through extraction process. So a BI system does not require executing the extraction, but only the PSA data, which is further processed in the delta procedure.
RDA in BI Staging
Here in RDA, the data is extracted from the PSA in delta mode and is written to the DSOs. For Web service source systems, the data is already available in the PSA. While for SAP systems, extraction from PSA must also be considered in the RDA design.
3 Requests in RDA
The PSA Request - Contains raw data of the source system. The DTP request - Describes transformation into DSO. The Change Log Request - Contains activated data of the DSO
With regular staging these 3 requests are not often generated. Instead, they are created synchronously & are closed if they contain a defined amount of data records. Control is usually transferred completely to a daemon. So, RDA is not only a technology but also a particular form of automation in standard operations.
Data source for RDA

In SAP systems, the controlling of extraction needs to be explicitly defined by the RDA. But this is one optional form of usage. Only with the selection of suitable adapter will be the execution affected in real time.
Info Package for RDA

IP for real time data acquisition can be defined with SAP Source systems only if the delta procedure was initiated & the DTP was created. For a Data source for a real time data acquisition, only one info package for RDA can be defined. However for Web services, the IP must also be provided, if data is transferred to BI via Web Services. (Even if data is written into PSA & is not extracted)
DTP for RDA

As a precondition for the creation of IP for RDA, we must choose DTP as 'DTP for Real-time data acquisition' with the definition of DTP. As with the definition of IP, the qualification of DTP mostly serves the purpose of modifying request handling. A precondition for having a DTP for RDA is that, we must have at least one DSO as data target & one data source to provide data
Controlling the RDA

The processing of the RDA, through the staging architecture involves 2 scenarios: Initialization of Delta procedure & Data transfer during RDA. The monitoring scenarios are as explained below: Data Initialization: The initialization of delta procedure is necessary at first to create the IP for RDA. If the initialization is made with DTP, then initialization process can't be handled by the DTP. Switching DTP types: The handling of the initialization of the DTP cannot be done, and must be reverted back to initial state. The request needs to be switched to its default behavior, as the initialization can be only started via a daemon that controls & monitors the RDA. The DTP definition contains a key called 'Change to Standard DTP', which when clicked can switch DTP to initial state.
Daemon for RDA

If the delta initialization was processed, & respective request was activated in DSO, controlling IP, DTP & activation of new data in DSO can be transferred to a daemon that controls & monitors the staging for RDA. A daemon is a continuous running back ground job that switches into sleep mode between single extraction processes. However, to improve the main memory usage, it ends the job & reschedules itself again.
Step-by-Step Guide for RDA Implementation

The other alternative for RDA is Web Services, even though Web Services are referred for non-SAP systems. For a sample scenario we will consider implementing a Web Service in XML Data Source system
Steps: SOAP using WSA

The following are the steps for creating SOAP Runtime Web services. We must load data into the DSO by creating DS, IP of the same type. Go to WSADMIN, the Web Service Administration using the Info package details From here navigate to the Web service homepage, where you can login to view your WSDL details. The corresponding Web service is listed under the technical name /BIC/**NAME*000, where we can view the XML type structure Now, we can schedule IP for a Real Time Data Acquisition using Push service, assigning interval. Back in RSA1, We create transformation, map source/targets & create DTP, where care must be taken to choose it as 'DTP for Real Time Data Acquisition' Even after Activation, 'Execute' option would not be enabled, because its to be handled by daemon. So, we go to RSRDA, where we create a daemon. Further, both the IP & DTP are assigned to this daemon & once done its execution is started. Back in WSADMIN, We initiate the Web Service Homepage & login to view Web services. Here, as we enter data via the portal, using RPC the daemon will initiate a process to load the data directly at regular intervals into the Info provider. In this case, we use a DSO, where we can view the data present in it using RSA1/LISTCUBE.
Process Steps
Go to RSA1, and follow the steps as below.
Fig.1.1 - This is a DSO which has been copied from the cube. The name for this source DSO: DSO_SLS. The target DSO has a name: DSO SOAP (Technical Name: C_DSO_SO)
Fig.1.2 - Now, In Info provider section we can view the DSO's structure. It has got the following fields. Primary Key fields - Customer ID (TCS CUSTID; Technical Name is TC_CUSTID) & Material ID (TCS MATID; Technical Name is TC_MATID). Data Fields is Quantity (TCS_QTY; Technical Name is TC_QTY) & its Unit of measure (0UNIT)
Fig.1.3 Go to DSO Settings & choose 'Select Type'. There are 3 options in the DSO level to choose its type. We choose to play safe with Standard DSO.
Fig.1.4 - It should be noted that this DSO will have unique fields. That means no data must be repeated. Go to Data sources & we choose 'XML'. This option helps us choose the type of Data source we intend to use. It could be 'FLAT FILE', 'SAP...' or 'XML' etc. Since we are using WSDL demo, we'll choose XML for now.
Fig.1.5 - In 'Data Sources', we need to create the Application Component Area. We create an AC Area & provide its long description also in this step.
Fig.1.6 - In this step, under the newly created Application Component Area, we create a 'Data Source'. In short all DS are just objects & are collected under this Application Comp. Area, which is a sort of 'folder' for these objects.
Fig.1.7 - Here, as illustrated, we name the data source using a name, describe its Source system as XML_DATA & its to be noted that we are using a data source which utilizes 'Transaction Data' perfect for reporting purposes. XML_DATA is the source system, as XML is very light weight & is currently used widely in Web Services.
Fig.1.8 - Here, in the 'Extraction' field, we have an adapter option, where we must choose to use a push service. So we must choose 'Web Service (Push)'. As for as extraction is concerned for this data source its never a problem specifying the file names, header rows, columns to be ignored etc. When using Web service its tad simpler as compared to 'Load Data from Flat Files' following these issues. But by any means, XML can handle only limited data & has its own pros & cons.
Fig.1.9 - This is the next tab, Proposal tab where we must specify the fields for use.
Fig.1.10 - So we need to copy these from the DSO structure of previous source DSO, which is DS_SLS. Now that we have copied, they are pasted as in order. Save & Activate
Fig.1.11 - Now, Save and activate the new structure. There's a dialog box displaying an information 'R3TR generated changes possible'. Now that the Data source has been successfully created & activated, we can refresh & find if its activated or not
Fig.1.12 - After a refresh, we can notice the 'Active' Version of the Data source.
Fig.1.13 - Creation of IP: We now create an IP. Right click DS & choose option "Create Info Package"
Fig.1.14 - Enter a description for this IP & Click Save.
Fig.1.15 - Click 'Yes'
Fig.1.16 - Finally the Info package has been created. We can notice the technical name of the IP starts with "ZPAK...... Now, we need to migrate to the reach the Web administration. From here we can directly go to 'Web Service Administration' or use TCODE WSADMIN. But for now we move on to create the entire data flow structure, further which we can transfer data from portal to Info providers (like DSO) via IP & DTP
Fig.1.17 - This is an optional step. Go to WSADMIN to view the URL, Web service Key, Release Text, WS Definition etc. Also dont assign IP at this stage.
Fig.1.18 - Creation of Transformation: Right click the DSO & choose the option "Create Transformation". This will now create a transformation to map source(s) & target(s).
Fig.1.19 - Create Transformation: This has the Source & Targets. Here the source object is the Data source (DS_XML_CSK) & the source system is XML_DATA. Now, the target object is of type "Data Source Object" (DSO SOAP; Technical Name: C_DSO_SO)
Fig.1.20 - In the transformation level, we have mappings between the source Data source & the target DSO.
Fig.1.21 - Creation of DTP: At this step we create a DTP. Right click on & choose the option "Create Data Transfer Process..."
Fig.1.22 - As earlier, we need to choose setting of the options - DTP Type: "DTP for Real Time Acquisition" & Source is Data source (DS_XML_CSK); Target is DSO (DSO SOAP; Technical Name: C_DSO_SO)
Fig.1.23 - Notice the 'Execute' option is disabled. The DTP is now in 'Modified' state, so it must be activated.
Fig.1.24 - Now, we can notice the Version is now 'Active' & Mode: 'Processing Mode for Real-Time Data Package'. Even after Activation, the 'Execute' option would not be enabled, because it must be handled by a daemon. So, we go to RSRDA, where we create a daemon.
Fig.1.25 - Create a daemon in RSRDA, using key F5 or clicking 'Daemon' command button
Fig.1.26 - Now, enter the Daemon #, Short Description and Interval Period [1 min]. After this click on 'Save'
Fig.1.27 - We can notice the nodes which are all daemons. Here we our Data sources (DS_XML.... etc) under the category 'Unassigned Nodes. Now right click the corresponding DS & choose an option to 'Assign Daemon' to a particular daemon 'node'.
Fig.1.28 - Here judiciously we assign the daemon ID [For instance: Daemon ID #07 has been assigned for this data source]
Fig.1.29 - Now, that we have assigned the DS to this daemon 'node', we cannot start unless we have DTP also. Note: Info package is visible; its technical name comes in handy as it starts with 'ZPAK.......'.
Fig.1.30 - At this point, we have to assign the DTP also for data transfer. Here we have right clicked on the DS & choose the option 'Assign DTP'.
Fig.1.31 - Notice the message 'DTP assigned to Info Package', which means its a successful procedure.
Fig.1.32 - Now, its time to begin the daemon process: Here we right click the daemon & choose the option 'Start Daemon with all Info packages...'
Fig.1.33 - The batch job has been planned for the IP & the job is started when batch job is free. Notice the change in the # of records in upload & the 'Last Upload' time.
Fig.1.34 Go to DS level & click on 'Web Service Administration'
Fig.1.35 - In this Web service administration page, Choose the corresponding web service (Here its /BIC/CQDS_XML_00003000)
Fig.1.36 - Choose the option 'Web Service Homepage' or use Ctrl + F8
Fig.1.37 - Choose the option - 'RPC Style' in the Settings for WSDL Generation & click OK.
Fig.1.38 - Login the web portal with user name & password
Fig.1.39 - Notice the details of the web service [As in example for /BIC/CQDS_XML_00003000]. We have the WSDL details, description, UDDI publications, Features etc.
Fig.1.40 - On viewing these details, we navigate to the page which handles the 'Operations' for the /BIC/CQDS_XML_00003000
Fig.1.41 - Clicking on the /BIC/CQDS_XML_00003000's operation would lead us to this page. Here we need to key in details of the Database via portal, which is formatted using XML. Here we can notice the details are entered as - [Mat.ID, Cust.ID, Quantity, Unit] E.g. for a sample data is like - [MX3330, CX3330, 343, KG]. Notice the time-out for this is 60 seconds as created for the daemon.
Fig.1.42 - After clicking 'Submit', the data is processed & sent to the recipient.
Fig.1.43 - Now in WSADMIN, we can notice the change in 'Records in Upload' as 1, and it took exactly the correct transfer as assumed. Hence its successful.
Fig.1.44 Go to RSA1, and we must right click the DSO & choose the option 'Display Data'
Fig.1.45 - Perform the 'Field selection Output' function for details necessary for user.
Fig.1.46 - Successfully all details have been updated to DSO from the portal via a web service.
Implementation Results
Thus, the demonstration of Real-Time Data Acquisition (RDA) for SAP BI 7.0 using Web Services API was successfully implemented.
Related Content
SAP Net Weaver BW-Administration Administration for the SOAP Runtime Real Time Data Acquisition (RDA) Overview and Step-by-Step Guide For more information, visit the Business Intelligence homepage.
Disclaimer and Liability Notice

This document may discuss sample coding or other information that does not include SAP official interfaces and therefore is not supported by SAP. Changes made based on this information are not supported and can be overwritten during an upgrade. SAP will not be held liable for any damages caused by using or misusing the information, code or methods suggested in this document, and anyone using these methods does so at his/her own risk. SAP offers no guarantees and assumes no responsibility or liability of any type with respect to the content of this technical article or code sample, including any liability resulting from incompatibility between the content within this document and the materials and services offered by SAP. You agree that you will not hold, or seek to hold, SAP responsible or liable with respect to the content of this document.

Realtimedata Acess

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Realtimedata Acess

Uploaded by

Copyright:

Available Formats

Demonstration of Real-Time Data Acquisition (RDA) for SAP BI 7.

0 using Web Services API

Arun Bala Ganesan

Company: Wipro Technologies Created on: 16 January 2010

SAP COMMUNITY NETWORK 2010 SAP AG

SDN - sdn.sap.com | BPX - bpx.sap.com | BOC - boc.sap.com | UAC - uac.sap.com 1

SAP COMMUNITY NETWORK 2010 SAP AG

SDN - sdn.sap.com | BPX - bpx.sap.com | BOC - boc.sap.com | UAC - uac.sap.com 2

RDA Vs Direct Access

Fig. A. Existing Example: RDA using SAP Service API

SAP COMMUNITY NETWORK 2010 SAP AG

SDN - sdn.sap.com | BPX - bpx.sap.com | BOC - boc.sap.com | UAC - uac.sap.com 3

Existing Data Flow Concept Example

Fig. B. Existing Data Flow Concept Example

SAP COMMUNITY NETWORK 2010 SAP AG

SDN - sdn.sap.com | BPX - bpx.sap.com | BOC - boc.sap.com | UAC - uac.sap.com 4

Fig. D. Modeling View

RDA Example Using Service API Administration View

SAP COMMUNITY NETWORK 2010 SAP AG

SDN - sdn.sap.com | BPX - bpx.sap.com | BOC - boc.sap.com | UAC - uac.sap.com 5

Fig. E. Administration View

Types of RDA Mechanisms

SAP COMMUNITY NETWORK 2010 SAP AG

SDN - sdn.sap.com | BPX - bpx.sap.com | BOC - boc.sap.com | UAC - uac.sap.com 6

RDA Implementation Scenario

Fig. F. RDA Implementation Scenario

Why use Delta?

SAP COMMUNITY NETWORK 2010 SAP AG

SDN - sdn.sap.com | BPX - bpx.sap.com | BOC - boc.sap.com | UAC - uac.sap.com 7

Initiate update of Data Store Object using DTP

Fig. G. Types of Icons for daemon processes

Fig. H. Real Time Data Acquisitions monitoring using RSRDA

RDA in Extraction Layer

SAP COMMUNITY NETWORK 2010 SAP AG

SDN - sdn.sap.com | BPX - bpx.sap.com | BOC - boc.sap.com | UAC - uac.sap.com 8

SAP COMMUNITY NETWORK 2010 SAP AG

SDN - sdn.sap.com | BPX - bpx.sap.com | BOC - boc.sap.com | UAC - uac.sap.com 9

Data source for RDA

Info Package for RDA

DTP for RDA

Controlling the RDA

Daemon for RDA

Step-by-Step Guide for RDA Implementation

SAP COMMUNITY NETWORK 2010 SAP AG

SDN - sdn.sap.com | BPX - bpx.sap.com | BOC - boc.sap.com | UAC - uac.sap.com 10

Steps: SOAP using WSA

SAP COMMUNITY NETWORK 2010 SAP AG

SDN - sdn.sap.com | BPX - bpx.sap.com | BOC - boc.sap.com | UAC - uac.sap.com 11

SAP COMMUNITY NETWORK 2010 SAP AG

SDN - sdn.sap.com | BPX - bpx.sap.com | BOC - boc.sap.com | UAC - uac.sap.com 12

SAP COMMUNITY NETWORK 2010 SAP AG

SDN - sdn.sap.com | BPX - bpx.sap.com | BOC - boc.sap.com | UAC - uac.sap.com 13

SAP COMMUNITY NETWORK 2010 SAP AG

SDN - sdn.sap.com | BPX - bpx.sap.com | BOC - boc.sap.com | UAC - uac.sap.com 14

SAP COMMUNITY NETWORK 2010 SAP AG

SDN - sdn.sap.com | BPX - bpx.sap.com | BOC - boc.sap.com | UAC - uac.sap.com 15

Fig.1.14 - Enter a description for this IP & Click Save.

Fig.1.15 - Click 'Yes'

SAP COMMUNITY NETWORK 2010 SAP AG

SDN - sdn.sap.com | BPX - bpx.sap.com | BOC - boc.sap.com | UAC - uac.sap.com 16

SAP COMMUNITY NETWORK 2010 SAP AG

SDN - sdn.sap.com | BPX - bpx.sap.com | BOC - boc.sap.com | UAC - uac.sap.com 17

SAP COMMUNITY NETWORK 2010 SAP AG