66: Tracing Production with Kai Wern Choong

We talk with Kai Wern Choong about his experience with tracing performance problems in production. We talk about what “tracing” means in a BEAM system, available tools like recon, great resources for learning how the tools work and general approaches for troubleshooting live production Elixir systems. Kai also shares his Livebook notebook setup for demonstrating tracing techniques in an interactive way.

Show Notes online - http://podcast.thinkingelixir.com/66

Elixir Community News

• https://twitter.com/hexpm/status/1439268469296570368 – New version of Hex released that fixes issues with version downgrades and slow version resolutions. Run

  mix local.hex

  to upgrade.


• https://hexdocs.pm/credo/1.6.0-rc.0/changelog.html#1-6-0 – Credo 1.6.0 was released with new "First Run Mode"


• https://twitter.com/elixirphoenix/status/1438222537301843970 – Removing Phoenix compiler config change can speed up compilation times when on up-to-date versions


• https://github.com/Qqwy/elixir-type_check – New library called TypeCheck


• https://blog.oestrich.org/2021/09/introducing-aino/ – Eric Osterich is experimenting with an alternative HTTP framework for Elixir


• https://github.com/oestrich/aino – The Aino project


• https://techcrunch.com/2021/09/14/logistics-startup-stord-raises-90m-in-kleiner-perkins-led-round-becomes-a-unicorn-and-acquires-another-company/ – Stord, an Elixir company, raised $90M and became a "unicorn"


• https://spawnfest.org/ – SpawnFest 2021 happened


• https://github.com/spawnfest/eflambe – SpawnFest submission - rapid generation of flamegraphs


• https://github.com/spawnfest/Sketch – SpawnFest submission - library for creating generative art with Elixir


• https://github.com/spawnfest/lotus – SpawnFest submission - Lotus is a Surface UI wrapper for UIKit


• https://github.com/spawnfest/Discovery – SpawnFest submission - Platform for hosting realtime, stateful servers with zero downtime deployment and horizontal scaling on Kubernetes


• https://github.com/spawnfest/crash – SpawnFest submission - Crash is a simple (distributed) docker-on-docker Continuous Integration system written in Elixir

Do you have some Elixir news to share? Tell us at @ThinkingElixir or email at [email protected]

Discussion Resources

• https://kaiwern.com/posts/2021/06/27/debugging-with-tracing-in-elixir-with-recon_trace/


• https://www.naluri.life/


• https://ferd.github.io/recon/recon.html


• https://hex.pm/packages/recon


• https://twitter.com/kw7oe/status/1409126300636254219


• https://github.com/kw7oe/livebook-notebooks/blob/main/debugging-with-tracing-in-elixir-with-recon_trace.livemd – Livebook notebook showing tracing examples


• https://www.youtube.com/watch?v=sR9h3DZAA74 – Debugging Live Systems on the BEAM talk by Jeffery Utter


• https://www.youtube.com/watch?v=OR2Gc6_Le2U – Operable Erlang and Elixir talk by Fred Hebert (skip to 25:51, where he started talking about tracing)


• https://kaiwern.com/posts/2020/11/02/debugging-with-tracing-in-elixir/ – Tracing with dbg modules


• https://en.wikipedia.org/wiki/Tracing_(software)


• https://erlang.org/doc/man/dbg.html – dbg module in Erlang


• :dbg.fun2ms


• https://kubernetes.io/blog/2017/12/using-ebpf-in-kubernetes/


• https://github.com/cilium/cilium


• https://opentelemetry.io/docs/erlang/


• https://www.brendangregg.com/blog/2019-01-01/learn-ebpf-tracing.html


• https://www.erlang-solutions.com/capabilities/wombatoam/


• https://hex.pm/packages/recon_ex


• https://erlang-in-anger.com/ – Erlang in Anger (Chapter 9)


• https://www.thegreatcodeadventure.com/testing-genservers-with-erlang-trace/ – Testing GenServers with Erlang Trace


• http://blog.plataformatec.com.br/2016/04/how-to-trace-elixir-nodes-with-erlyberly/ – How to trace Elixir nodes with Erlyberly - Plataformatec Blog


• http://erlang.org/doc/man/seq_trace.html#sequential-tracing – Seq_trace


• http://erlang.org/doc/man/erlang.html#trace-3 – Erlang:trace


• https://github.com/andytill/erlyberly – Erlang tracing for the masses


• https://github.com/tatsuya6502/recon_ex – Elixir wrapper for Recon, tools to diagnose Erlang VM safely in production

Guest Information

• https://twitter.com/kw7oe – on Twitter


• https://github.com/kw7oe/ – on Github


• https://kaiwern.com/ – Blog

Find us online

• Message the show - @ThinkingElixir


• Email the show - [email protected]


• Mark Ericksen - @brainlid


• David Bernheisel - @bernheisel


• Cade Ward - @cadebward