Build and test solution

Try Pentaho Data Integration and Analytics

Version
9.3.x
Audience
anonymous
Part Number
MK-95PDIA000-11
During this step, you develop transformations, jobs, and models, then test what you have developed. You will tune the transformations, jobs, and models for optimal performance.

Development occurs in the PDI client design tool. The PDI client's streamlined design tightly couples the build and test activities so that you can easily perform them iteratively. The PDI client has perspectives help you perform ETL and visualize data. The PDI client also provides a scheduling perspective that can be used to automate testing. Testing encompasses verifying the quality of transformations and jobs, reviewing visualizations, and debugging issues. One common method of testing is to include steps in a transformation or job that calculate hash totals, checksums, record counts, and so forth to determine whether data is being properly processed. You can also visualize your data in analyzer and report designer and review the results as you develop. This can not only help you find errors and issues with processing, but can help you get a jump on user acceptance testing if you show these reports to your customers or business analysts to get early feedback.

One basic question, is how to determine the numbers of transformations and jobs needed, as well as the order in which they should be executed. A good rule of thumb is to create one transformation for each combination of source system and target tables. You can often identify combinations in your mapping documents. Once you've identified the number of transformations that you need, you can use the same process to determine that number of jobs that you need. When considering the order of execution for transformations and jobs, consider how referential integrity is enforced. Run target table transformations that have no dependencies first, then run transformations that are depend on those tables next, and so forth.

Table. Table 2. Build and Test Checklist - PDI client
Task Do This Objective
Understand the Basics
  • Read the overview of the PDI client process in the Pentaho Data Integration document.
  • Review information about the process and perspectives.
Review most often used steps and entries
  • Review available transformations and determine how you can use them for your solution.
  • Review job step references to identify which steps can be used in your solution.
Create and Run Transformations
  • Create and run a transformation. See the Pentaho Data Integration document for details.
  • Identify the transformations needed for your job and implement them.
  • Save transformation.
  • Run transformations locally.
Create and Run a Job
  • Create and run a job. See the Pentaho Data Integration document for details.
  • Create a job.
  • Arrange transformations in a job so that they execute logically.
  • Run a job.