From the Scandinavian Kraken that drags entire ships to their early graves to the Philippine Bakunawa said to rise from the waters to swallow the moon, our existential fears about what lurks at the bottom of our own oceans has inspired us to create hundreds of folkloric monsters across cultures. We’ve charted the surface of our planet, but the vast majority of the ocean floor remains a mystery.

The mysterious depths of modern software architectures makes it difficult for us to test a system for performance and reliability. The easy solution? Unmanned underwater probes: send inputs to the system, and judge its state based on the outputs that return. But this method is often too limited to tell us what’s really going on when something goes wrong. What if there were a way to make our tests themselves more observable?

In this talk, you will learn how to use open-source tools to instrument performance tests, identify bottlenecks at the code level using distributed tracing, and map out a system using a type of software bathymetry. You’ll also learn about the Mariana Trench expeditions, the difficulties of operating in a high-pressure underwater environment, and how to come face to face with monsters of our mythologies.

Key takeaways:

• Black-box testing can be useful, but we can provide more value with our testing by making our tests observable.
• OpenTelemetry is a set of tools that standardize the observability of our systems.
• Distributed tracing isn’t just for SREs; load testers can use it to be able to follow the path of a request throughout the system.
• We can use projects like k6 + Tempo + Grafana to improve the continuous reliability and performance of our systems.