Idea
The Software Factory already supports smart upgrades, which upgrade only what has changed. I'd like the same principle for unit tests: when running an upgrade or a creation, only execute the unit tests of objects that were affected by the changes, and run the full set separately.
Why
We now generate a large share of our unit tests with AI, which makes our test suites many times bigger than when we wrote them by hand. On one application with about 125 tables we have around 1,500 unit tests, and a full run takes 30 to 40 minutes. While it runs, the branch is occupied: we can't generate the definition, so development on that branch stops for the duration.
Most of that time is spent for nothing. In a typical change only a small part of the model is touched, yet every test is executed, including the tests for the large majority of objects that did not change at all.
Proposal
- Smart run during upgrade/creation. Only run the unit tests whose object is affected by the changes since the last successful run. Ideally this includes dependencies: a test on a task that calls a changed subroutine should also run.
- Full run on schedule. The ability to schedule a run of all unit tests, for example nightly, so the full regression still happens without blocking development during the day.
- Full run on demand. Keep the option to start a full run manually, for example before a delivery or release.
Result
Short feedback loops during development, no idle time on the branch, and the full safety net still in place where it matters: nightly and before delivery.