📱 23 / 100 Days Of Flutter, Kate's edition — golden, integration and end-to-end tests

Hi, Flutter community! 👋 I'm back with an update on my Flutter challenge.

I finally finished testing part of the Flutter Foundations Course 🎉

In summary, I've learned about the following types of tests:

Unit — verifies the behaviour of a method or class
Widget — verifies the behaviour of Flutter widgets without running the app itself
Golden — verifies visual output of Flutter widgets without running the app itself
UI integration — verifies the full app UI works by running the app on a device (some network services can be still mocked)
End-to-end — same as above, but without any mocking

In the previous part, I shared examples of Unit and Widget tests. This time, let's look at the other three.

Golden tests

When I just started learning about Flutter testing, I couldn't wait to touch on its visual regression testing capabilities known as the “golden” tests.

First of all, I like the name! From Flutter docs (source):

The term golden file refers to a master image that is considered the true rendering of a given widget, state, application, or other visual representation you have chosen to capture.

The master golden image files that are tested against can be created or updated by running flutter test --update-goldens on the test.

Sounds simple:

Write a common widget tess
Inject matchesGoldenFile checks when you need to check the visual output
Run the tests with the --update-goldens flag to generate base images
Enjoy your visual regression testing 🎉

In reality, though, there are some obstacles. First of all, Flutter widget tests don't load images, icons and fonts. Those assets are not needed for functional testing of the widget, but they are critical for visual testing.

By default, "Golden" screenshots are taken without fonts, icons, and images

To overcome this issue, golden tests require some setup code to properly load the app assets. Alternatively, a helper package "golden_toolkit" can be used (I haven't tried it myself yet).

Another problem is the generation of golden images in the first place. Images of the same app, generated on different versions of operating systems, can look slightly different and generate false positives. To use golden tests in CI, you need to come up with a strict process for image generation.

Integration tests

Integration tests in Flutter run the app on a given device (simulator or physical). This way developers can test entire user journeys such as registration flow. They also can test the app on a range of devices and configurations to ensure the app runs on the latest iPhone as well as on low-end Android.

While the official Flutter documentation doesn't draw a clear line between integration and end-to-end tests, Andrea Bizzotto sticks to the definition in which integration tests are primarily UI tests with all networking mocked. Mocking helps to speed up the tests and reduce costs of backend services such as Firebase.

End-to-end tests

These tests run the full app on a device without any mocking at all. They guarantee the entire app works as expected. I haven't published any app myself yet, but I suspect the end-to-end test would run the app in the environment very close to production.

Summary

Flutter has got robust and well-documented testing tooling to run unit, widget, golden, integration, and end-to-end tests.

I was lucky to learn about testing fundamentals from Andrea's Flutter course. For me, it was a good reminder of how tests can help design better software.

Now it's time to go back to the Flutter app architecture topic and build more sophisticated user flows.

Stay tuned 👋