Nikita Baksalyar

The Soul of a New Debugger

Sun, 12 Jul 2020 00:00:00 +0000

This blog post is a follow-up to A Future for Rust Debugging

Why developers are reluctant to use interactive debuggers and prefer prints instead?

This is a question that has been perplexing me, and while it’s not possible to have a definite answer, we can make some educated guesses. prints are simple and very effective: you don’t have to painstakingly step through your code to understand what’s happening and you don’t need any external tools – everything is already there, you just need to add a few statements and run your program again.

Here’s the trick, though: if you code in a language like Python, JavaScript or Ruby, all you need to do to run your program is to execute it. But there is a bit more friction with compiled native languages like C++ or Rust: every recompilation step costs time, and with larger code bases with many dependencies it can quickly become non-negligible. It’s even more complicated if you want to debug a problem that occurs on a remote machine (e.g., a production server) or on an embedded platform, since you will need to redeploy the newly compiled binary every time you want to run a new debug experiment.

Interactive debuggers solve these problems: you can take an existing program that was compiled in the debug mode and probe it to your liking, setting up conditional breakpoints, looking up current values of variables, and doing a lot of other useful things. However, in my view, the widely-used interactive debuggers have a set of problems of their own:

A lot of them were designed in a different era. At that time, we didn’t have always-on Internet connections, high-resolution multi-colour displays, and devices with many gigabytes of memory. While the fundamental principles remain the same, the modern needs have changed and so do modern means.
Many existing debuggers are general-purpose. While there is some support for scripting and language-specific extensions, it’s harder to use them for specific domains and it’s harder to integrate them with your dev environment. In practical terms, this means you have to use other tools in addition to debuggers to observe the behaviour of your software, and oftentimes it’s just easier & faster to move along with prints instead of trying to find a suitable tool.

I also see more general problems with regards to debuggers. They are perceived mostly as a tool to help you find and fix bugs in your code, but not as much as a tool of discovery and exploration. It’s been 8 years since Bret Victor published his “Learnable Programming”, and these ideas are as relevant today as ever. Debuggers should strive to be a good learning tool too.

Lastly, interactive debuggers rely on underlying principles that are quite similar to profilers, dynamic tracers like DTrace and eBPF, memory leak detectors, and other developer tools. Ideas and code can – and should – be shared across this ecosystem, but it seems like whilst one type of tools gets a lot of attention, the others may still lack support for important features.

So what can we do to try solving these problems?

Elements of a modern debugger

In my previous article, I argued for a case of extending the existing debuggers to provide better support for Rust. However, after some more research and thinking, I feel that we should consider the idea of creating a new debugger framework from scratch, taking inspiration from other great projects. Not-invented-here syndrome aside, I think there are some valid reasons for going in this direction.

Modern languages like Rust have lots of new important features that weren’t available in languages that the classic debuggers were written in. Things like fearless concurrency, first-class modules & packages, async I/O and zero-cost abstractions can affect the design of a new debugger in a significant way. With the Rust’s package manager, Cargo, extensibility becomes even more relevant and viable. We’ve already seen what modular debuggers are capable of – for example, the Illumos Modular Debugger, mdb, allows to debug both native and JavaScript code in Node.js with an mdb_v8 extension, and the architecture of the debugger itself allows to extend it further using a simple, modular interface. With language features like traits, it can become even easier to create your own domain-specific debuggers using a new framework.

Another project to take inspiration from is Delve, a Go debugger. It’s built with modern tools and has several features important for Go developers, but I want to highlight the JSON-RPC API it provides. This API addresses the important problem of integration with the development environment, and by using a well-defined protocol we can create an ecosystem for debuggers that’s similar to the one that’s flourished around the language server protocol. With an HTTP-based API, we can build custom debugger front-ends using HTML and WebAssembly and run them in web browsers. With the rich front-end tools & frameworks, it opens up lots of interesting options for data representation and visualisation. Integration doesn’t have to be one-sided, too: language servers can be reused for debugging purposes, and integrating a Rust-specific debugger with the Rust compiler would allow us to utilise the full power of the existing language syntax parser and other compiler components.

What’s next?

Overall, I believe this is a project worth building. While we’re seeing a lot of innovation in compilers and language design, debuggers are somewhat neglected, even though debugging is no less important; as Kernighan’s law postulates, it’s twice as hard as writing the code in the first place.

Creating a new debugger is an insurmountable task and a very long journey. But it can have a modest start: if we cover only a few popular operating systems and a few simple cases first, we can quickly achieve small wins that can save us a lot of time and frustration.

This post outlines the initial project plan for Headcrab, a Rust debugger library. I will be publishing more code and documentation in the coming weeks. If you are interested in updates, please follow me on Twitter. Progress updates will be also published on this blog.

You can find a more detailed roadmap in the project repository.

Resources and further reading

Debugging support in the Rust compiler
mdb, Illumos Debugger
Rosenberg, J.B. (1996). How debuggers work : Algorithms, data structures, and architecture. ISBN 0471149667.
Uresh Vahalia (1996). UNIX internals : the new frontiers. ISBN 9780131019089.

A Future for Rust Debugging

Tue, 19 May 2020 00:00:00 +0000

Recently, there has been a lot of progress in the Rust logging & tracing ecosystem: projects like tracing make our lives much simpler, allowing us to track down bugs even in complex asynchronous environments in production. However, they still can’t replace interactive debuggers like gdb or lldb, which provide so much more than just stack traces and variables printing.

In an ideal world, you can use your debugger as a read-eval-print loop, writing, rewriting, and testing entire functions in an interactive session. Techniques like post-mortem debugging make it possible to prod dead processes, providing immeasurable help in understanding the state of a program just before it crashed. But the harsh reality is that debuggers can be counter-intuituve, hard to use, or just suck, and it’s doubly challenging for Rust.

Considering the complexity of modern Rust projects, it’s sometimes outright impossible to debug certain cases because the commonly used debug tools like gdb or lldb are tailored mostly for C or C++, and they make certain assumptions about the way your code is written. Needless to say, these assumptions sometimes can be drastically wrong because C++ and Rust are very different languages. For example, here’s a relatively simple program that uses thread-local variables:

use std::cell::RefCell;
thread_local! {
  pub static XYZ: RefCell<u64> = RefCell::new(0);
}
fn main() {
  XYZ.with(|val| {
    *val.borrow_mut() = 42;
  });
}

And here’s what happens if I want to print the current value of XYZ in lldb:

Attempting to read thread-local variables.

The first intuitive attempt to print out the variable using the standard Rust syntax fails miserably because lldb doesn’t understand Rust at all; in fact, it uses a C++ expression parser instead. But even if we peek into the Rust thread-local vars implementation and use a C++ expression to print them, it wouldn’t work because lldb is made to think it works with C++, and it doesn’t know that Rust has a new name mangling scheme which has diverted from C++.

And thread-local variables isn’t a very contrived case: for example, they are used in the Tokio runtime implementation to store information about tasks, and it would be very helpful to inspect them for understanding how your async code is executed (or, at the very least, this kind of debug info can be helpful for developers of Tokio). Speaking of Tokio and the async ecosystem, it is impossible to print out async backtraces without also having to plod through async runtime internals. Sometimes a runtime can reschedule tasks, losing the original stack context in the process, and if you want to track the origin of an async failure in this case, you are left with only one option – logs and traces (if there’s an alternative way, I’d be happy to learn about it!).

All of these problems have the same underlying cause: Rust has to live in the C/C++-centric world. So what can we do about it (other than Rewriting-It-In-Rust, of course)? I think that the ideal systems programming language needs to have ideal, modern debugging tools.

Luckily for us, a debugger like lldb does not insist on using C/C++ for everything. It has a modular core, and languages support can be added as plugins – and that’s exactly what lldb does for C++, which is not really a special case, but a plugin that’s built on top of Clang, an LLVM-based C++ compiler. There has been an effort to implement a similar language plugin for Rust – which, however, is a tricky enterprise, because the lldb plugin API is private^[1] and it’s written – you guessed it – in C++. Implementing a parser for a subset of Rust in C++ is hard enough, but keeping up with all the changes in the language is barely possible.

But now, with an effort to “library-ify” Rust, I hope that this project can be reinvigorated. And while we are at it, we can try to apply the idea of compiler as a library in another interesting way: debugger as a library.

In this example, the private lldb plugin API, which implements the language support, calls into the shared Rust debugger library to parse debug expressions. The debugger library can then delegate parsing to Rust compiler libraries, and in turn use debugger APIs provided by lldb when needed^[2].

It should be noted that this hypothetical debugger library will work entirely in the Rust domain, so it will have the access to knowledge and power of mid-level internal representation. Of course, this doesn’t have to work for lldb only: the abstract interface can be ported to other debuggers. And if we make this library itself modular, it opens up a lot of interesting possibilities, like a more complete Rust REPL, custom visualizations, or domain-specific debuggers.

This is not a new idea and there are numerous good examples we can learn from. mdb, or a Modular Debugger, was extended to inspect NodeJS programs. The .NET Core runtime diagnostics tool has a debugger abstraction library with adapters for lldb and Windows Debugger. I’m sure there are many others, but most of these examples boil down to the same idea of extending the debugger core with language-specific capabilities.

Conclusion

Obviously, this is a huge effort, which might (and most certainly will) require lots of changes across different projects like LLVM, the Rust compiler, or even the DWARF debug format specification^[3]. Instability of private APIs is definitely one of the challenges here, but it can be partially solved by having stable forks – which is how it’s done in a large number of cases anyway^[4]. Supporting abstractions like traits can be another difficult task because debuggers are not particularly suited to work with generic code (which is compiled down to concrete types). But I believe that even a basic Rust-aware debugger can make a huge difference in users experience, and I deem it a worthy goal to pursue.

Thank you for reading this!

References

[1] [lldb-dev] Rust language support question.

[2] There is even a bindings crate for the public part of the lldb API.

[3] Tom Tromey discusses debugging support in rustc (video).

[4] E.g., see an Apple’s fork of lldb to add support for Swift.

Rust in Detail, Part 2

Mon, 09 Nov 2015 00:00:00 +0000

Part 2: Sending and Receiving Messages

In this series of articles we’ll follow the process of creating a scalable, real-time chat service — everything in excruciating detail, of course! The objective is to learn more about system APIs and the emerging language Rust on a practical (and not too contrived) example, going step by step.

The second part takes up where the part one left off and continues with the WebSocket protocol implementation. If you haven’t read the first part yet, it’s better to start there.

Rust in Detail: Writing Scalable Chat Service from Scratch

Fri, 10 Jul 2015 00:00:00 +0000

Part 1: Implementing WebSocket. Introduction.

In this series of articles we’ll follow the process of creating a scalable, real-time chat service.
The objective is to learn more about practical usage of the emerging language Rust and to cover the basic problems of using system programming interfaces, going step by step.

Part 1 covers the project setup and the implementation of a bare bones WebSocket server. You don’t need to know Rust to follow, but some prior knowledge of POSIX API or C/C++ would certainly help. And what would help you even more is a good cup of coffee — beware that this article is fairly long and very low-level.

Let’s begin the journey!