DataStage Version 8 on Information Server

Enterprise PACKs

SAP BW Pack o BAPI: (Staging Business API) loads from any source to BW. o OpenHub: extract data from BW.

SAP R/3 Pack o ABAP: (Advanced Business Application Processing) auto generate ABAP, Extraction Object Builder, SQL Builder, Load and execute ABAP from DataStage, CPI-C Data Transfer, FTP Data Transfer, ABAP syntax check, background execution of ABAP. o IDoc: create source system, IDoc listener for extract, receive IDocs, send IDocs. o BAPI: BAPI explorer, import export Tables Parameters Activation, call and commit BAPI.

Siebel Pack o EIM: (data integration manager) interface tables o Business Component: access business views via Siebel Java Data Bean o Direct Access: use a metadata browser to select data to extract o Hierarchy: for extracts from Siebel to SAP BW.

Oracle Applications Pack o Oracle flex fields: extract using enhanced processing techniques. o Oracle reference data structures: simplified access using the Hierarchy Access component. o Metadata browser and importer

DataStage Pack for PeopleSoft Enterprise o Import business metadata via a metadata browser. o Extract data from PeopleSoft tables and trees.

JD Edwards Pack o Standard ODBC calls o Pre-joined database tables via business views Database Connectivity The common connection objects functionality means the very wide range of DataStage database connections are now available across Information Server products. Latest supported databases for version 8:

DB2 8.1, 8.2 and 9.1

Oracle 9i, 10i, 10gR2 not Oracle 8

SQL Server 2005 plus stored procedures.

Teradata v2r5.1, v2r6.0, v2r6.1 (DB server) / 8.1 (TTU) plus Teradata Parallel Transport (TPT) and stored procedures and macro support, reject links for bulk loads, restart capability for parallel bulk loads.

Sybase ASE 15, Sybase IQ 11.5, 12.5, 12.7

Informix 10 (IDS)

SAS 612, 8.1, 9.1 and 9.1.3

IBM WS MQ 6.1, WS MB 5.1

Netezza v3.1

ODBC 3.5 standard and level 3 compliant

UniData 6 and UniVerse ?

Red Brick Source

New Stages in DataStage Version 8

New Stages in Datastage Version 8 A new stage from the IBM software family, new stages from new partners and the convergence of QualityStage functions into Datastage. Apart from the SCD stage these all come at an additional cost.

WebSphere Federation and Classic Federation

Netezza Enterprise Stage

SFTP Enterprise Stage

iWay Enterprise Stage

Slowly Changing Dimension: for type 1 and type 2 SCDs.

Six QualityStage stages New Functions in Existing Stages

Complex Flat File Stage: Multi Format File (MFF) in addition to existing cobol file support.

Surrogate Key Generator: now maintains the key source via integrated state file or DBMS sequence.

Lookup Stage: range lookups by defining checking high and low range fields on the input or reference data table. Updatable in memory lookups.

Transformer Stage: new surrogate key functions Initialize() and GetNextKey().

Enterprise FTP Stage: now choose between ftp and sftp transfer. Source :

When to choose Server or Parallel Data stage job

1. The choice of server or parallel depends upon time to implement, functionality and cost.
2. When we have lots of functionality to implement for lower volume and hardware is less and ease of implementation we can go for Server jobs.
3. Parallel jobs are costly due to high scale of hardware , difficult to implement, extreme processing capabilities for absurd volumes with vast array of operators for high-performance manipulation.
4. When the data volume is less it is better to go for Server job as parallel jobs can have a longer start up time.
5. When data volume is high, it is better to choose parallel job than server job. Parallel job will be a lot faster than server job even if it runs on single node. The obvious incentive for going parallel is data volume. Parallel jobs can remove bottlenecks and run across multiple nodes in a cluster for almost unlimited scalability. At this point parallel jobs become the faster and easier option. A parallel sort stage is lot faster than server stage. A Transformer stage in parallel job with the same transformations in server job is faster. Even on one node with a compiled transformer stage, the parallel version was three times faster. On 1 node configuration that does not have a lot of parallel processing also we can still get big performance improvements from an Enterprise Edition job. The improvements will be multiplied 10 or more than that if we work on 2CPU machines and two nodes in most stages.
6. Parallel jobs take advantage of both pipeline parallelism and partitioning parallelism.
7. We can improve the performance of server job by enabling inter process row buffering. This helps stages to exchange data as soon as it is available in the link. IPC stage also helps passive stage to read data from another as soon as data is available. In other words, stages do not have to wait for the entire set of records to be read first and then transferred to the next stage. Link partitioner and link collector stages can be used to achieve a certain degree of partitioning parallelism.
8. Look up with sequential file is possible in parallel jobs and not possible in server jobs.