Migrating from Cypress to Capybara — a reluctant tale

Many months ago, our team had to have a hard conversation about Cypress. Cypress was the new kid in our CI pipeline, a browser integration testing framework. We had thought it would replace Capybara, our older way of driving a browser manically around our website. Capybara was slow, we thought. It had bad developer feedback, we thought. The tests needed to be written in Ruby and wouldn't it be far more comfortable writing them in JavaScript, we thought?

Cypress was having problems. Sometimes the tests passed, sometimes the tests failed. What we did in between those times had little effect. Cypress was flaky.

Now, we had already had flakiness problems with Capybara. A large portion of our engineering team (1 dev) had put in a significant amount of effort (a week sipping slightly stronger coffee and retiring early every day complaining of a faint headache) to solve the flakiness problems. We had it down to a tee. And so we entered into the Cypress world with confidence that the problem would be the same. We must have merely been writing our tests in the wrong way. "It's not you, Cypress — it's us!" we exclaimed.

It was Cypress

Unfortunately Cypress was intractable. It appeared we were not the only troubled fools. When your test framework needs a flaky test manager, your own efforts look pitiful by comparison. After one engineer ejected their laptop from a third-story window when Cypress couldn't be persuaded (nicely or harshly with sticks) to wait for a checkbox to be enabled before clicking on it, the team decided: we would move back to Capybara.

I shed tears and wailed, for Capybara was slow. It took so long to re-run each time and progress through the test was a mystery. I preferred writing the tests in JavaScript — it felt more natural when driving a browser. However, the team had made a decision for good reason, even if this was not my preference.

Making a decision and taking action are quite different steps though. We had tests written in both Cypress and Capybara. Converting the Cypress ones into Capybara wasn't that easy:

• Capybara had a worse developer experience than Cypress. This put a drag on both writing new tests and converting the existing ones.
• We had used a Cypress plugin to do image comparisons for charts.

So we did what any good team does and wrote a list of tests to convert and then did nothing for several months. This also added some drag to our development work — do you update the existing Cypress test or do the larger job of converting it to Capybara?

Image comparison functionality

Things came to a head while I was working on the burndown charts for Aha! Develop. The image comparison we had in Cypress was a bit finicky to work with and the Capybara libraries that added image comparison had limitations. I wanted something better:

• Multiple image storage so we could have a comparison that passed for developers and on CI where the rendering might be slightly different.
• Image diffs so we could see where the image had changed.
• Customizable fuzziness. Sometimes anti-aliasing around chart lines and text has imperceptible differences that we'd like to ignore.
• Masking - If I take the image and make part of it transparent that area should be ignored.

Using the compare tool in ImageMagick, I was able to write a Capybara add-on that achieved all of the above and more

Contributions from others on our team added deferred checking, so that the image comparison errors wouldn't interrupt the whole test but would instead be output at the end, and tasks for downloading all the images from our CI server to make development faster.

It was an arduous task to convert all the image comparison tests from Cypress to Capybara. However, this process gave me hope that not only could we do this, but we could also make the whole experience even better than Cypress.

Capybara developer experience

Cypress has a nice developer experience. The UI shows the test steps. When the test file is updated, the test is restarted automatically and you can see the progress as it runs the test. This gives you a nice simple feedback loop.

When Capybara starts up, it needs to initialize the whole Rails application, start the web application, and start the browser. This means that, even with tools like Guard, the develop test loop is slow. The browser context is lost after each test run.

Our Capybara tests are written in RSpec. One of our team leads, Percy Hanna, created a test runner that wraps the RSpec runner so that, at the end of the test, it halts waiting for input and reloads the test file. We then added file watching, which allowed it to reload automatically when the test file or JS files changed.

What about reloading when the Ruby files changed? That turned out to be a harder problem. A running RSpec test did not seem to enjoy the Rails autoloader trying to change things.

The solution I came up with is to store information about the current browser session in the environment and then use Kernel.exec to relaunch the test.

On test start, it captures the existing browser session. From a developer perspective, this is a little slower than test loop when the test files changes. But it keeps your browser session open with the DevTools open if you were using them.

Making Capybara better than Cypress

When using Cypress, I kept the test running while I developed. I really valued being able to see the progress through the test file, with each step listed out. Hovering over the steps even shows a snapshot of the browser at that step. This was good but didn't always work because it was loading the html at the time back into the browser.

Ruby has an API called TracePoint. This lets you hook into the running Ruby code and call blocks when certain things happen, like a line being run or a method being called.

Using TracePoint, I was able to create a custom output that showed the line number and the line being run:

This looks great in the terminal and it's very helpful for understanding where the test is currently up to. If Cypress showed this information in the browser, could we do that with Capybara?

Of course we can. The information can be poked into the browser by executing a bit of JavaScript before and after each step. This also gives us a stepping stone to further improvements, such as:

• Showing the line currently running and progressing through the whole test
• Displaying the failure in the browser
• Taking a screenshot at each step
• Capturing the Rails controller traces whenever the browser makes a request

And miss Cypress I do not. With some time and some love, our Capybara test suite has far exceeded Cypress.

The syntax and methods are better for a team of Rails engineers. The querying and wait time execution in Capybara leads to flake-free tests. The lower-level nature of Capybara lends itself to understanding and extending to match the workflow.

Now the development process I often use is:

1. Create a Capybara scenario.
2. Start the runner to watch the code and launch a browser on my second monitor.
3. Edit the code and test scenario as I work, pushing the test further along towards the end goal.
4. Use Byebug in the test code to drive the browser.

Do you use Cypress, Capybara or some other browser testing tool? What are your experiences testing it and customizing it to match your team's workflow?

Aha! is happy, healthy, and hiring. Join us!

This feature took a lot of tinkering and guidance from other engineers. Without such a knowledgeable and helpful team, I would not have arrived at this clean solution. If you want to build lovable software with this talented and growing team, apply to an open role.