Saturday

Introduction to WebSphere QualityStage

WebSphere QualityStage comprises a set of stages, a Match Designer, and related capabilities that provide a development environment for building data-cleansing tasks called jobs.
Using the stages and design components, you can quickly and easily process large stores of data, selectively transforming the data as needed.
WebSphere QualityStage provides a set of integrated modules for accomplishing data re-engineering tasks:
  • Investigating
  • Conditioning (standardizing)
  • Designing and running matches
  • Determining which data records survive
The probabilistic matching capability and dynamic weighting strategies of WebSphere QualityStage help you create high-quality, accurate data and consistently identify core business information such as customer, location, and product throughout the enterprise. WebSphere QualityStage standardizes and matches any type of information. By ensuring data quality, WebSphere QualityStage reduces the time and cost to implement CRM, business intelligence, ERP, and other strategic customer-related IT initiatives.
Scenarios for data cleansing
Organizations need to understand the complex relationships that they have with their customers, suppliers and distribution channels. They need to base decisions on accurate counts of parts and products to compete effectively, provide exceptional service, and meet increasing regulatory requirements. Consider the following scenarios:
  1. Banking: One view of households
To facilitate marketing and mail campaigns, a large retail bank needed a single dynamic view of its customers’ households from 60 million records in 50 source systems. The bank uses WebSphere QualityStage to automate the process. Consolidated views are matched for all 50 sources, yielding information for all marketing campaigns. The result is reduced costs and improved return on the bank's marketing investments. Householding is now a standard process at the bank, which has a better understanding of its customers and more effective customer relationship management. 2.Pharmaceutical: Operations information A large pharmaceutical company needed a data warehouse for marketing and sales information. The company had diverse legacy data with different standards and formats, information that was buried in free-form fields, incorrect data values, discrepancies between field metadata and actual data in the field, and duplicates. It was impossible to get a complete, consolidated view of an entity such as total quarterly sales from the prescriptions of one doctor. Reports were difficult and time-consuming to compile, and their accuracy was suspect. Most vendor tools lack the flexibility to find all the legacy data variants, different formats for business entities, and other data problems. The company chose WebSphere QualityStage because it goes beyond traditional data-cleansing techniques to investigate fragmented legacy data at the level of each data value. Analysts can now access complete and accurate online views of doctors, the prescriptions that they write, and their managed-care affiliations for better decision support, trend analysis, and targeted marketing. 3.Insurance: One real-time view of the customer A leading insurance company lacked a unique ID for each subscriber, many of whom participated in multiple health, dental, or benefit plans. Subscribers who visited customer portals could not get complete information on their account status, eligible services, and other details. Using WebSphere QualityStage, the company implemented a real-time, in-flight data quality check of all portal inquiries. WebSphere QualityStage and WebSphere MQ transactions were combined to retrieve customer data from multiple sources and return integrated customer views. The new process provides more than 25 million subscribers with a real-time, 360-degree view of their insurance services. A unique customer ID for each subscriber is also helping the insurer move toward a single customer database for improved customer service and marketing.

Tuesday

Data Stage Job Status Values

    The Data Stage Job Status values can be found in
$DSHOME/include/dsapi.h file


  These are the values defined in dsapi.h file for JOB STATUS:



  1. #define DSJS_RUNNING        0      /* Job running */
  2. #define DSJS_RUNOK          1      /* Job finished a normal run with no warnings */
  3. #define DSJS_RUNWARN        2      /* Job finished a normal run with warnings */
  4. #define DSJS_RUNFAILED      3      /* Job finished a normal run with a fatal error */
  5. #define DSJS_VALOK          11       /* Job finished a validation run with no warnings */
  6. #define DSJS_VALWARN        12      /* Job finished a validation run with warnings */
  7. #define DSJS_VALFAILED      13      /* Job failed a validation run */
  8. #define DSJS_RESET          21       /* Job finished a reset run */
  9. #define DSJS_CRASHED        96      /* Job was stopped by some indeterminate action */
  10. #define DSJS_STOPPED        97      /* Job was stopped by operator intervention (can't tell run type) */
  11. #define DSJS_NOTRUNNABLE    98      /* Job has not been compiled */
  12. #define DSJS_NOTRUNNING     99      /* Any other status */
JobStatus

Friday

DataStage Version 8 on Information Server

Enterprise PACKs
  • SAP BW Pack o BAPI: (Staging Business API) loads from any source to BW. o OpenHub: extract data from BW.
  • SAP R/3 Pack o ABAP: (Advanced Business Application Processing) auto generate ABAP, Extraction Object Builder, SQL Builder, Load and execute ABAP from DataStage, CPI-C Data Transfer, FTP Data Transfer, ABAP syntax check, background execution of ABAP. o IDoc: create source system, IDoc listener for extract, receive IDocs, send IDocs. o BAPI: BAPI explorer, import export Tables Parameters Activation, call and commit BAPI.
  • Siebel Pack o EIM: (data integration manager) interface tables o Business Component: access business views via Siebel Java Data Bean o Direct Access: use a metadata browser to select data to extract o Hierarchy: for extracts from Siebel to SAP BW.
  • Oracle Applications Pack o Oracle flex fields: extract using enhanced processing techniques. o Oracle reference data structures: simplified access using the Hierarchy Access component. o Metadata browser and importer
  • DataStage Pack for PeopleSoft Enterprise o Import business metadata via a metadata browser. o Extract data from PeopleSoft tables and trees.
  • JD Edwards Pack o Standard ODBC calls o Pre-joined database tables via business views Database Connectivity The common connection objects functionality means the very wide range of DataStage database connections are now available across Information Server products. Latest supported databases for version 8:
  • DB2 8.1, 8.2 and 9.1
  • Oracle 9i, 10i, 10gR2 not Oracle 8
  • SQL Server 2005 plus stored procedures.
  • Teradata v2r5.1, v2r6.0, v2r6.1 (DB server) / 8.1 (TTU) plus Teradata Parallel Transport (TPT) and stored procedures and macro support, reject links for bulk loads, restart capability for parallel bulk loads.
  • Sybase ASE 15, Sybase IQ 11.5, 12.5, 12.7
  • Informix 10 (IDS)
  • SAS 612, 8.1, 9.1 and 9.1.3
  • IBM WS MQ 6.1, WS MB 5.1
  • Netezza v3.1
  • ODBC 3.5 standard and level 3 compliant
  • UniData 6 and UniVerse ?
  • Red Brick Source