Documentation Driven Development

… and a wishlist for better Rust Tooling

I have been working on documenting a growing rust project, and ran into some papercuts along the way. Also, writing good documentation is actually very hard and sometimes tedious work, but I think it is very much worth it.

# Why I love Doctests

I can’t really stress how important both testing, and documentation is! Testing makes sure that your project meets its goals and certain quality criteria. And documentation makes sure that your potential users know how to effectively use your project. Without documentation (and marketing), a project will struggle hard to grow its user base.

However, maintaining good documentation is really hard. And depending on the language ecosystem, it is too easy for your code and documentation to drift apart and become broken / outdated. That is one reason why I really love the concept of doctests. On a high level, they are just example code snippets that you want to present to your users. But they also run as part of your testsuite, just like any other tests. So they can’t drift apart. It is a really good way to give usage examples of your API, and document corner cases all at the same time.

# Rusts Unique Position

Rust actually is in a very good position here. I originally come from the TypeScript ecosystem, and it is actually a huge pain that TypeScript suffers a lot from the “new day, new framework” syndrome. There is way too much choice among linters, build systems / bundlers, testing frameworks, benchmark frameworks, and probably documentation tools, although I haven’t really used any of those. It is quite tiring to wade through all the options. At least that ecosystem now has a code formatter that everybody agrees on.

It is really refreshing that Rust is super opinionated in this regard. It ships with a default build system (which also has workspace support), testing tool, a linter, a benchmarking tool (with support for custom frameworks) and a first class documentation tool. You get all of this out of the box (or rather, via rustup) and don’t have to worry about any of this. What a way to boost productivity!

Also, rusts tools are really good, even though they have some shortcomings as we will see.

# My Wishlist for better Tools

But unfortunately, testing and writing doctests is a bit painful right now. But as I see it, those obstacles should be fairly straight forward to overcome. Here is just a short wishlist of things I would love to see in the Rust ecosystem that would make my life a bit easier ;-)

• cargo#2832 and cargo#4324: In general, the output of cargo test is horrible. Especially if you have workspaces, or a number of examples and integration tests. It is super hard to actually find failures, and the signal to noise ratio is horrible. I would love if cargo had a better way to put tests into suites, and to better visualize those.

  Related, it would be nice to better filter tests, and to re-run only previously failed tests.

• cargo#6424: Apparently, cargo check does not check doctests? Oh well. I haven’t run into this so far, but its a minor oversight.

• rust-analyzer#4170 and rust-analyzer#571: My number one productivity win would probably come from being able to write doctests just like any other code, with syntax highlighting and auto-completion and realtime linting.

• rustfmt#3348: Similarly, code in doctests should be nicely formatted just like any other code.

• rust#44732: Especially for crate-level or module-level documentation, I would very much prefer to write that in an external markdown file. This would also solve all the problems with duplicated, nearly empty, or outdated README files.

• rustfmt#2036: A bit related to the point above, I would love to also have auto-formatting and re-wrapping for markdown. I spend way too much time manually formatting my doc comments.

• rust#43466: Another small quality of life issue which makes linking to other items easier and avoids the problems of links going stale when moving or renaming items.

• rust#45599 and rust#67295: Its a bit broken right now how to use feature-flagged things in doctests. But I found a workaround I explain in the next section.

• One pipedream would be to also have a kind of linter for doc comments. Things like length limits for captions, enforcing formatting rules such as “end captions with a period”. A spell checker would also be nice ;-)

Some of the mentioned issues are already implemented on nightly versions, and are just pending stabilization. Others are quite big and complex, or seem to stuck in bikeshedding discussions. But one thing that actually sounds quite doable is to have rustfmt format and re-wrap markdown. I might even look into this if I have too much free time on my hands, or as something of a hackweek project.

# Forcing Features in Tests

So the crate I am working on right now has quite some feature flags, and I would like to refer to those feature-flagged items in my doctests. But that breaks the tests when run with --no-default-features, or in general if the features I want to use are not default.

Also, you can’t #[cfg(test)] those items, since doctests won’t pick those up due to rust#45599 and rust#67295. But I have found a rather nice workaround for that. Whereas Rust disallows normal dependency cycles. Defining circular dev-dependencies actually works fine, which is quite magical. So you can just dev-depend on yourself, with certain features. The same also works across circular workspace crates.

[package]
name = "doctests"
version = "0.1.0"
edition = "2018"

[features]
featured = []

[dev-dependencies]
doctests = { path = ".", features = ["featured"] }

/// ```
/// doctests::featured();
/// ```
pub fn doctest() {}

#[cfg(feature = "featured")]
pub fn featured() {}

#[test]
fn unittest() {
    crate::featured();
}