Data Subset Modernization Guide

Data Subset is a tool designed to extract smaller, representative portions of production data for use in testing environments. This allows organizations to test applications with realistic data without exposing sensitive information or overwhelming test systems with the full production dataset.

Data Subset is a specialized toolset focused on data extraction and manipulation for testing purposes. It is not a complete system, application, framework, or middleware, but rather a utility that supports those broader systems.

Organizations that require robust testing of applications against realistic data, particularly those dealing with large production databases, benefit from Data Subset. This includes companies in regulated industries like banking, insurance, and healthcare where data privacy and compliance are critical.

A company should consider Data Subset when the need arises to create smaller, manageable datasets for testing, development, or training purposes. This is especially relevant when dealing with sensitive data that needs to be masked or when full production datasets are too large to be efficiently used in non-production environments.

Alternatives to Data Subset include manual data extraction and masking processes, other commercial data subsetting tools, and custom-built scripts. Competing products include IBM Infosphere Optim Test Data Management Solution and Delphix.

Data Subset typically runs on z/OS and may require specific subsystems depending on the data sources being accessed. It is often deployed on-premise, close to the production data it needs to extract.

For mainframe products, Data Subset runs in an LPAR and is z/OS dependent. It may require specific subsystems like DB2 or IMS, depending on the data sources being accessed.

Data Subset often uses configuration files to define data extraction rules, masking policies, and connection parameters. These files specify which tables and columns to include in the subset, how to mask sensitive data, and how to connect to the source databases.

Data Subset may expose APIs for integration with other testing and development tools. These APIs could be used to automate the data subsetting process, trigger data refreshes, or integrate with CI/CD pipelines.

Data Subset solves the business problem of providing realistic test data without compromising sensitive information or overwhelming test environments. By creating smaller, representative subsets of production data, organizations can improve the efficiency and effectiveness of their testing processes.

If an organization did not use Data Subset, they would likely rely on manual data extraction and masking processes, which can be time-consuming, error-prone, and potentially expose sensitive data. Alternatively, they might use full production datasets in test environments, which can be inefficient and resource-intensive.

Data Subset helps organizations accelerate testing cycles, reduce the risk of data breaches, and comply with data privacy regulations. By providing realistic and secure test data, it enables faster and more reliable software releases.

Data Subset employs data masking techniques to protect sensitive information in the extracted subsets. This may include techniques like data encryption, data redaction, and data substitution.

Data Subset often integrates with existing security systems to control access to the extracted data subsets. This may involve integration with LDAP directories or other authentication providers.

Data Subset may provide audit logging capabilities to track data extraction and masking activities. This allows organizations to monitor compliance with data privacy regulations and identify potential security breaches.

Implementing Data Subset requires technical expertise in data extraction, data masking, and database administration. Ongoing operational requirements include monitoring data extraction jobs, maintaining data masking policies, and ensuring the security of the extracted data subsets.

Common implementation challenges include identifying sensitive data, defining appropriate data masking rules, and ensuring the consistency and accuracy of the extracted data subsets.

Data Subset may provide administrative interfaces, such as a command-line interface (CLI) or a web console, for managing data extraction jobs, defining data masking policies, and monitoring system performance.