We need to talk about test data 'strategy'

For many organisations, test data 'best practices' start and end with compliance. This reflects a tendency to focus on the problem immediately in front of us. 'The business' or legislation have called for the removal of sensitive data from non-production environments; so, that’s the fire that organisations strive to put out first.

Though typically necessary, removing sensitive data from non-production environments overlooks two of the biggest challenges associated with test data today. Firstly, it does not help with the immense time that testers and developers spend waiting for, finding, and making test data. Secondly, it overlooks the impact that low-variety production data has on overall test coverage. To solve all three test data challenges – speed, quality, and compliance – a new strategy is needed.

I recently joined Paul Hammersley of EPI-USE Labs to discuss how organisations can target all three of these test data challenges. You can watch the recorded webinar when it suits you.

This blog highlights some of the test data pressures that we resolved in the webinar, and indicates the solution.

Test data: A problem that isn’t going away by itself

Many organisations today must re-think their strategy for test data 'management'. Relying on a central team to anonymise and copy large production data sets will always be a game of catch-up. Meanwhile, it does nothing to improve the quality of the data for testing, and nor does it reduce the time teams spend wading through large data sets or making missing combinations by hand.

A range of factors have increased the demand for test data, adding to the urgency of a strategy re-think. These related trends have made it harder than ever for manual data provisioning to provide data of sufficient variety, at the speed demanded by parallel teams and frameworks. They include:

1. “Agile”, DevOps and iterative delivery: With rapid, iterative development, changes and new release candidates arrive faster than ever. This demands continuous access to ever-changing data sets.

2. Automated testing and CI/CD: Automated test execution has increased the volume and variety of tests being run, each requiring up-to-date data. Automated tests are also less forgiving than manual testers. If they are provisioned with inaccurate or inconsistent data, they simply fail, wasting time as those failures must be investigated.

3. Parallelisation of teams and frameworks: Today, there are usually more teams and frameworks than ever trying to work in parallel. These parallel testers cannot rely on a limited number of production data copies. They need parallel data, as otherwise they use up or edit one another’s data.

4. Parallelisation of tests: While executing faster than manual tests, automated tests are also capable of running in parallel. Often, two or more tests in a test suite will require similar data combinations. This increases the demand for data, as time-consuming test failures will mount if one test consumes another test’s data.

5. System complexity and new technologies: As developers adopt new technologies and systems grow increasingly complex, it’s becoming harder than ever to fulfil all the requisite dependencies in test environments. Data masking and generation, for instance, must anonymise data consistently across a range of databases and files. Otherwise, integrated and end-to-end tests will fail.

Modernising test data

Test data practices today need to be brought into line with the 'best practices' found across DevOps and CI/CD pipelines. 'Provisioning' data must be automated and parallelised, as well as capable of responding to changing requests on-the-fly. Both automated and manual data requesters must further be capable of triggering the reusable processes on demand, easing the pressure on an overworked data provisioning team.

Today, there are many effective tools and techniques that address different problems associated with test data. They include data masking to support compliance, generation to boost data variety, and data cloning to make data available to parallel tests, testers, and environments. Database virtualisation has further minimised the time and costs associated with copying data, while data comparisons and analysis engines help testers and developers understand data.

You probably already have some of these solutions at your organisation, either built in-house or using commercial tools. The missing piece in many test data strategies is the process by which the different tools can be combined, reused, and made available on demand to manual and automated data requesters. Instead, responsibility is pushed back onto an over-worked provisioning team, who adjust and slowly run a set of linear processes for each data request.

A two-stage modernisation strategy for test data therefore looks as follows:

In other words, a complete test data strategy must comprise all the technologies needed to create complete and compliant data in parallel and on demand. These techniques must furthermore be standardised and automated, as well as being exposed to parallel teams, automation frameworks and CI/CD pipelines. Manual and automated data requesters must be capable of parameterising and triggering the reusable test data processes on demand, receiving the data they need on-the-fly.

Want to see this strategy in action?

In a recent webinar, we explored how organisations can move from supporting test data compliance to implementing a modern test data strategy. To see how complete and compliant data can be made available on-the-fly, watch the webinar 'Testing across SAP and non-SAP systems: From test data compliance to continuous innovation.'