Introducing Productboard Pulse. Exec-level insights into what your customers need, powered by AI.
We recently decided it was time to move from Cypress 9 to Cypress 12 on the platform team. Our testing framework hadn’t been scaling as fast as our engineering had over the last couple of months, and we were quickly finding ourselves in an increasingly sticky situation. We wanted to use this opportunity to restructure the test framework.
We had a lot of unresolved technical debt. The old Cypress version we were using had become a bottleneck for other tooling upgrades, and our product teams were creating new test projects using the old framework setup, essentially creating even more technical debt.
We wanted to use new Cypress features – including testing in isolation, improved test setup performance and more debugging possibilities – so we could move faster and have less friction during test automation.
It was time for Cypress 12! We knew that such a large-scale upgrade might possibly give us (and the product teams) a massive collective headache. But we also knew that in the long-run the change would be worth the effort.
Here’s a brief look at what went on behind the scenes for all those who are about to embark on a similar journey.
We have about 130 spec files across 25 test projects. Each test project is a separate Nx application that defines its own dependencies. This allows us to run tests only for affected parts of our application by code changes. We also have custom generators to quickly bootstrap new test projects with custom configurations.
Our test framework consists of a custom Nx Cypress executor that executes our tests. Imagine this as an orchestration tool that takes care of space setup, resolves project configurations and calls either `cypress open` or `cypress run`, depending on the test environment. The framework also has custom helpers that consist of test data, command overwrites, custom commands, and shared functions for multiple domains.
Our structure looks something like this:
Each new Cypress version has introduced some challenges for our test framework. Version 10 introduced breaking changes in project configuration logic. Version 11 caused breaking changes for our component tests. Version 12, meanwhile, completely changed the way we handle authentication and execution in Nx Cypress executor.
Imagine reviewing all of these changes in one pull request 🤯.
After an initial discussion, we decided to migrate the changes version by version and include some refactoring along the way.
We started by reading the Cypres migration guides. These guides are available for each major Cypress version and identify main changes needed for the upgrade.
Our discovery was based on the Cypress migration guide document and a quick showcase migration on a simple test project. Based on this, we identified tasks that needed to be done for each test project and understood how much of the work was required.
Our tasks were split into five main areas:
Using this as a guideline, we were able to split our work into smaller tasks that could be easily groomed and estimated.
We started by deleting all unused or skipped tests of more than four months. In some cases, this involved removing the whole test project setup. Our goal was to migrate only projects that were actively used, that saved us time during the migration.
Next, we continued refactoring the old framework code related to the migration. As part of this work, we revisited current logic and identified parts of the test framework that were obsolete, hard to understand, and slow in execution.
Finally, we upgraded Cypress. With the exception of Cypress 10 changes, which were difficult to split per project, we handled migration project by project. This made it faster for collecting approvals and easier for others to review code changes.
Before the merge, we ran each migration multiple times on local and CI environments to make sure there was no regression. We focused on test stability and the CI resources needed for the execution. After the merge, monitoring tools were put in place to observe stability and trigger alerts in case there was a large amount of test failures or a degradation in test performance.
In total, we removed around 4,500 lines of code (⬆️3,513 lines, ⬇️7,979 lines) during this initiative.
Does that mean our technical debt is gone?
No, but this migration helped us to address some of the problematic parts of our code base. and we are closer to a more robust and scalable test framework.
While we wait for Version 13 to be released, there are certain steps we would definitely follow again.
Here are our recommendations for your next Cypress upgrade:
Fancy joining us on our mission to make products that matter? We’re hiring across the board. Head over to our careers page to see our available positions. We’d love to hear from you!
