bazel-rust-guided-experiment

This article is about incorporating the Bazel build tool into a Rust project so that both cargo and bazel-based builds, tests, and vendoring work. Specifically, this is a Rust project that makes use of vendored dependencies. This integration works out really nicely, but there were some painful details that I ran into. I'm writing this article to

  1. Highlight those pain points and what I should have done instead,
  2. Give a quick demo on using bazel and how cargo and bazel work together to vendor dependencies, and
  3. Point out opportunities to improve the rules_rust documentation.

I'm trying to keep this article focused on relevant issues that might come up with integrating bazel. There were separate issues that came up due to "incorrect" directory structure, using the wrong bazel rules, etc. Where I tried something that didn't work, I will point it out as a kind of warning sign.

👀 And I will point out places that might be problems in the future, or, in general, interesting things that you should think about to keep reading interactive.

🤦‍♂️ And I will respond like this when I made the wrong decision or ran into a sharp edge.

Motivation---Why Bazel?

Rust compilation times are a common complaint. There have been significant improvements here in raw Rust compiler performance, but we can achieve even bigger savings by caching intermediate results between compilation runs. We can see the value of this strategy in the push for turning on incremental mode for the Rust compiler. Unfortunately, this isn't available everywhere that we might want to build Rust code: local Docker, as part of a CI pipeline, etc.

My path to getting to Bazel has been somewhat circuitous. Here are some tools, blogs, and comments that got me here:

  1. cargo-chef is a tool to improve caching between Docker builds for Rust projects. This formalizes a process of turning a Rust project into a hollow skeleton just including its remote dependencies. This stems from the observation that the remote dependencies don't change frequently, so compiling them once as a Docker layer could be a good source of caching.

    cargo-chef lists some limitations, but they all really boil down to a simple caching strategy of one mega cache that only includes external dependencies. Local dependencies inside of the project are rebuilt from scratch each time, which is a major pain point when they dominate compilation times. This is something that I ran into at my previous company: there were some other issues that using cargo-chef presented (mostly around patched dependencies) but the main one was that we really wanted to cache local dependencies.

  2. fasterthanlime's blog on his ideal Rust setup. I really enjoy Amos's articles. I really enjoy the depth, the narrative, the content, and the characters...

    👀 What do you mean?

    🤦‍♂️ You don't... see it?

    👀 Ha ha... No.

    I really enjoyed that article in particular because it had a lot of stuff I could steal look into for my own Rust development. He has a specific section on CI which includes:

    ... but I also want sccache, which is a much better solution than any built-in CI caching.

    The basic idea behind sccache, at least in the way I have it set up, it's that it's invoked instead of rustc, and takes all the inputs (including compilation flags, certain environment variables, source files, etc.) and generates a hash. Then it just uses that hash as a cache key, using in this case an S3 bucket in us-east-1 as storage.

    I was poised to look into sccache, despite rumors of it being somewhat unreliable and unmaintained. And then...

  3. I saw a comment on Reddit for an alternative to sccache. The parent post was about a Rust build tool, Fleet, that claims to lead to 5x faster build times. bitemyapp comments on various ways the tool achieves its speedups and offers some of their own thoughts and advice on optimizing Rust build times. I've copied what they say regarding sccache:

    sccache with on-disk caching: sccache is really flaky and poorly maintained. I was only able to get the S3 backend to work by using a fork of sccache and even then it breaks all the time for mysterious reasons. Using the on-disk configuration is confusing to me, you can get the same benefit by just setting a shared Cargo target directory. I set CARGO_TARGET_DIR in my .zshrc to $HOME/.cargo/cache and that gets shared across all projects. My guess is [Fleet's authors] saw a benefit from using sccache because they hadn't tried that and were benefiting from the cross-project sharing.

    ...

    What I've found to be more effective than what Fleet does:

    • Bazel's caching is astoundingly good but cargo-raze is awkward and the ecosystem really wants to nudge you toward a pure rustc + vendored dependencies. I'm using this for CI/CD at my job right now because the caching is both more effective and far more reliable than sccache. It even knows what tests to skip if the inputs didn't change. You can override that if you want but I was very pleased. My team uses Cargo on their local development machines because the default on-disk cache is fine.
    • ...

    If you're intersted in checking out improving Rust compilation times, I recommend reading that comment in its entirety.

  4. And finally, the DevOps team at my old company was looking at bazel.

All of this nudged me to checking out bazel for my own learning. My default Rust project template uses vendored dependencies, and bazel has a reputation of struggling with remote dependencies, so I wanted to see how easy it was to get bazel working in that type of project setup.

I also wanted to see how bazel and cargo can work together, since I enjoy using cargo locally.

Structure of this repo

Each child directory is the entire completed state of the project for that particular goal. The readme for each child will go into detail on what we want to accomplish, the steps from the previous completed goal to the current target goal, and any errors we encounter along the way. The general flow is

  1. The initial rust project,
  2. Building with bazel,
  3. Vendoring with bazel, and finally
  4. Wrapping up.

TL;DR

If you want to vendor your dependencies locally, start from the vendor local manifests (Cargo.toml files) example.

If you are migrating an existing project, make sure that your Rust code is not at the root level of the repository. (This is likely if you are in a multi-lingual project where using bazel would lead to cross-language builds, and potentially less likely if you are working in a Rust-only project.)

Also make sure that you name your local dependency rust_library targets the same as their crate name.

The bazel slack rust channel seems to be active and I found people there to be very helpful! Thank you so much to them!

Is it worth the complexity?

If you're working just in one language, I don't know. If you spend a lot of time building, maybe! If your project is kind of small and build times are't significant, maybe stick with language tooling until that becomes painful.

🤷‍♂️ But I've also heard the advice that you should switch to bazel before this point.

If you work in a project that has different languages all working together, bazel is a nice unified build tool that tries (depending on support level) to work for all of them. In this case, I think the smart builds and caching is worth the extra work for large enough projects.

For using Rust and Bazel together specifically, the experience falls somewhere in between using cargo and using bazel with other languages. Rust and Bazel have benefits that some other bazel ecosystems really struggle with. On the other hand, bazel (or rules_rust) is more opinionated than cargo when it comes to directory structures. Most of my difficulty in this experiment was making my directory structure match what bazel (or rules_rust) expected. But after that struggle, I got a bazel environment that still cooperates with native rust tooling and IDE integration and handles transitive dependencies nicely. Both of these are problems that people have with bazel (watch some Bazel talks on Youtube and see how many of them talk about getting bazel to work with IntelliJ). Now that I've gone through this struggle, hopefully other people can get the cool benefits without all of the flailing I did.

This guide is really only possible because we are standing on the shoulders of giants---rules_rust developers really put in a lot of work to make these typically painful bazel issues into non-issues. But I think there still could be some work on building projects that work with cargo in bazel.

🤦‍♂️ Maybe cargo-raze, another tool for integrating cargo and bazel works better for this. But I also ran into problems using cargo-raze... Maybe I'll look into it again another time.

I think, really, the tradeoff comes down to the work needed to adapt your directory structure to what bazel is expecting vs the caching that bazel gives you.

There is something else to keep in mind regarding using bazel: we have had to add these build files ourselves to get this level of caching. We are taking on a maintenance cost. Bazel is also not completely stable---it is under active development. Some of their plans (besides general improvements) include getting rid of WORKSPACE files and replacing them with mod files or something (see https://bazel.build/docs/bzlmod), so there might be some pretty signficant organizational changes. That being said, this is a nice tool for coordinating with other languages, and there's a certain sense of coordination between companies too. Having a bunch of companies use this tool means that everyone can benefit (but that could also mean that there are a lot of changes to keep up with).

Separately, bazel can also be used to minimise the number of potential dependencies, providing a road bump from your code base from turning into a ball of mud. In this scenario, bazel is actually a tool against unbounded complexity of dependencies.

Stage 0

Source: https://github.com/prestontw/bazel-rust-guided-experiment/tree/main/src/stage-0

This is our starting version. This tries to mimic a monorepo where we have a backend (where our Rust code goes) as well as a frontend. We also have a utility deployment folder for general tools, which may include additional Rust code. Since this might contain more Rust code, I'm setting this up with a workspace Cargo.toml file.

For building the backend, this stage uses cargo.

How did we get here?

This is copied from Axum's readme example. For actually vendoring the dependencies, this uses cargo-vendor.

There are two things to note on this vendoring:

  • we used versioned directories via the --versioned-dirs flag to cargo vendor, and
  • it's actually nested inside of our vendored directory, specifically for Rust. This anticipates adding other vendored dependencies for our front-end code (and better supports bazel's structure later).

With the above two notes, the command that we run is

cargo vendor --versioned-dirs 3rd-party/crates

🤦‍♂️ I actually messed both of these things up when I originally went through this process. That resulted in large git diffs when I vendored using bazel (since it uses versions in its directory names) and lots of floundering when trying to get bazel to vendor dependencies to an appropriate location. This is an example of something that I can smooth over in this blog. I can make this mistake in my original process, then time travel as if I did it correctly originally in this post! Note that this is a pain point, though. If you run into issues like this in the future, the examples are your best friends.

We also change our cargo configuration file to read from our local directory, following the steps that the above command lists:

[source.crates-io]
replace-with = "vendored-sources"

[source.vendored-sources]
directory = "3rd-party/crates"

We can verify this is successful with cargo build --offline.

What's next?

Well, this currently isn't building with bazel. Let's try to do that, while maintaining cargo build-ability so local development is nice and what developers are used to. Onwards to the next stage!

Stage 1: Let's build with bazel!

Source: https://github.com/prestontw/bazel-rust-guided-experiment/tree/main/src/stage-1-bazel-from-rules-rust-example

Alright, we have a Rust project that somewhat represents a case that you might want to bazel-ify: we have a bunch of different languages all living in the same repository, and we are interested in caching our build steps. Let's go through incorporating bazel into the Rust parts of this project. Our goal for this stage will just be to get the project building with bazel---let's hold off on vendoring with bazel for a little later. We are all about those quick iteration times and getting feedback!

How did we get here?

Well, we have a project that currently builds with cargo. We have four main pieces of documentation for Rust-specific bazel rules that will be helpful:

Let's get started! First, we will setup our project. The root-level documentation helpfully has such a subsection: https://bazelbuild.github.io/rules_rust/#setup. Let's copy those contents to a WORKSPACE.bazel file we put at the root level of our project:

load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")

# To find additional information on this release or newer ones visit:
# https://github.com/bazelbuild/rules_rust/releases
http_archive(
    name = "rules_rust",
    sha256 = "39655ab175e3c6b979f362f55f58085528f1647957b0e9b3a07f81d8a9c3ea0a",
    urls = [
        "https://mirror.bazel.build/github.com/bazelbuild/rules_rust/releases/download/0.2.0/rules_rust-v0.2.0.tar.gz",
        "https://github.com/bazelbuild/rules_rust/releases/download/0.2.0/rules_rust-v0.2.0.tar.gz",
    ],
)

load("@rules_rust//rust:repositories.bzl", "rules_rust_dependencies", "rust_register_toolchains")

rules_rust_dependencies()

rust_register_toolchains()

Later in that page they mention how we can specify the Rust version we want to use:

rust_register_toolchains(version = "1.59.0", edition="2018")

Let's do that as well!

👀 I really like specifying the Rust version for a project in a rust-toolchain file. It would be nice if I could specify a path to one instead of duplicating version information in bazel. This might be a nice feature to request!

Alright, this is enough for our first attempt at our WORKSPACE file. In that same page, it mentions crate_universe, an experimental set of rules for generating bazel targets for external dependencies. We want this eventually, let's see if it's a good starting point for our first BUILD file!

👀 There are other options for working with dependencies, such as cargo-raze. I went with crate_universe because I heard rumors that cargo-raze (or its functionality) was being merged into the main rules_rust rule set. I also tried making it work when using crate_universe ran into some snags, but ended up giving up and trying crate_universe again.

Slow down, eager beaver! Looking at the setup (https://bazelbuild.github.io/rules_rust/crate_universe.html#setup), there are actually some changes that we need to make to our WORKSPACE file:

load("@rules_rust//crate_universe:repositories.bzl", "crate_universe_dependencies")

crate_universe_dependencies()

One of the ways that we can handle our dependencies is through a cargo workspace: https://bazelbuild.github.io/rules_rust/crate_universe.html#cargo-workspaces. This matches our project structure already, yay! Let's copy that code:

load("@rules_rust//crate_universe:defs.bzl", "crates_repository")

crates_repository(
    name = "crate_index",
    lockfile = "//:Cargo.Bazel.lock",
    manifests = [
      "//:Cargo.toml",
    ],
)

load("@crate_index//:defs.bzl", "crate_repositories")

crate_repositories()

There's another code example for a BUILD file for a library, but we are building a binary. Let's copy a barebones version of the rust_binary example to build our backend service:

load("@rules_rust//rust:defs.bzl", "rust_binary")
load("@crate_index//:defs.bzl", "all_crate_deps")

rust_binary(
    name = "hello_world",
    srcs = ["src/main.rs"],
)

Sweet! Fast bazel-cached builds here we come! Let's try building all targets with

bazel build //...

and watch bazel fill in that initial cache!

...

Wait.

I'm getting an error about Cargo.Bazel.lock not existing. That makes sense---it's mentioned in https://bazelbuild.github.io/rules_rust/crate_universe.html#crates_repository. Let's add it with

touch Cargo.Bazel.lock

and try again!

Alright, I see another error:

ERROR: An error occurred during the fetch of repository 'crate_index':
   Traceback (most recent call last):
        File "/private/var/tmp/_bazel_preston/8649a483de3e22846dc6c9def4d70061/external/rules_rust/crate_universe/private/crates_repository.bzl", line 33, column 28, in _crates_repository_impl
                lockfile = get_lockfile(repository_ctx)
        File "/private/var/tmp/_bazel_preston/8649a483de3e22846dc6c9def4d70061/external/rules_rust/crate_universe/private/generate_utils.bzl", line 259, column 35, in get_lockfile
                path = repository_ctx.path(repository_ctx.attr.lockfile),
Error in path: Unable to load package for //:Cargo.Bazel.lock: BUILD file not found in any of the following directories. Add a BUILD file to a directory to mark it as a package.
 - /Users/preston/git/bazel-rust-guided-experiment/stage-1-bazel-from-rules-rust-example

Looking at that documentation, it does have a BUILD file at the same level as the Cargo.Bazel.lock file---let's add an empty one and see if it works!

Alright, we are getting closer. New error:

An error occurred during the fetch of repository 'crate_index':
   Traceback (most recent call last):
        File "/private/var/tmp/_bazel_preston/8649a483de3e22846dc6c9def4d70061/external/rules_rust/crate_universe/private/crates_repository.bzl", line 44, column 28, in _crates_repository_impl
                repin = determine_repin(
        File "/private/var/tmp/_bazel_preston/8649a483de3e22846dc6c9def4d70061/external/rules_rust/crate_universe/private/generate_utils.bzl", line 326, column 13, in determine_repin
                fail((
Error in fail: The current `lockfile` is out of date for 'crate_index'. Please re-run bazel using `CARGO_BAZEL_REPIN=true` if this is expected and the lockfile should be updated.

Again, this is somewhat expected---the subsection https://bazelbuild.github.io/rules_rust/crate_universe.html#repinning--updating-dependencies points it out.

We can either run bazel sync as that section specifies, or follow that error message and pass that specific flag to the command we ran above. Whichever we do, we end up with

no such package '@crate_index//': Command [/private/var/tmp/_bazel_preston/8649a483de3e22846dc6c9def4d70061/external/crate_index/cargo-bazel, "splice", "--output-dir", /private/var/tmp/_bazel_preston/8649a483de3e22846dc6c9def4d70061/external/crate_index/splicing-output, "--splicing-manifest", /private/var/tmp/_bazel_preston/8649a483de3e22846dc6c9def4d70061/external/crate_index/splicing_manifest.json, "--extra-manifests-manifest", /private/var/tmp/_bazel_preston/8649a483de3e22846dc6c9def4d70061/external/crate_index/extra_manifests_manifest.json, "--cargo", /private/var/tmp/_bazel_preston/8649a483de3e22846dc6c9def4d70061/external/rust_darwin_aarch64/bin/cargo, "--rustc", /private/var/tmp/_bazel_preston/8649a483de3e22846dc6c9def4d70061/external/rust_darwin_aarch64/bin/rustc] failed with exit code 1.
STDOUT ------------------------------------------------------------------------

STDERR ------------------------------------------------------------------------
Error: Some manifests are not being tracked. Please add the following labels to the `manifests` key: {
    "//backend:Cargo.toml",
}

This makes sense, and was something that I was wondering about when we filled in the manifests originally.

👀 ... Were you?

🤦‍♂️ Yes, actually, part of the point of this exercise is to

  1. go through and record errors that I run into,
  2. provide an example of how someone went through the rules_rust documentation, and
  3. do the minimum amount of work needed to get things in a functioning state.

If I do something extra at some early point, it might be unnecessary after all!

👀 Is this, perhaps, foreshadowing..?

Let's quickly add this manifest to our WORKSPACE.bazel file so that crates_repository looks like

crates_repository(
    name = "crate_index",
    lockfile = "//:Cargo.Bazel.lock",
    manifests = [
      "//:Cargo.toml",
      "//backend:Cargo.toml",
    ],
)

Let's try building now! If we run bazel build //..., we get... Rust compilation errors?

Ahh, it's all for our dependencies. Let's specify that the deps for our hello_world target are all_crate_deps() and let's see if that works:

rust_binary(
    name = "hello_world",
    srcs = ["src/main.rs"],
    deps = all_crate_deps(),
)

Oh. My. Goodness. It works! Running

bazel run //backend:hello_world

seems to work fine. And we can verify that it's working by going to localhost:3000 and seeing that classic greeting.

What did we do?

Whew! We went through several different pages of documentation and got bazel to build our project. Hooray! It's not using our cached dependencies, though...

What's next?

Let's get bazel to use our dependencies on-disk. Strap in, because the next stage is a little bumpy.

👀 Did anyone else see https://bazelbuild.github.io/rules_rust/crate_universe.html#crates_vendor? Just me?

🤦‍♂️ I saw it, but it wasn't in the top-level example for one of the two ways to support a cargo workflow.

👀 It's used in https://github.com/bazelbuild/rules_rust/blob/0265c293f195a59da45f83aafcfca78eaf43a4c5/examples/crate_universe/vendor_local_manifests/BUILD.bazel#L5...

🤦‍♂️ Again, an example of someone going through the documentation as written. Maybe chalk it up to... foreshadowing? ⛈

Stage 2: Let's vendor our dependencies through bazel!

Source: https://github.com/prestontw/bazel-rust-guided-experiment/tree/main/src/stage-2-crates-vendor

To recap, we can build our project through bazel! Whoopee! But it isn't making use of our vendored dependencies that we got through cargo vendor! Let's try to remedy this. This is our goal for this stage.

Watch out, there is a little bit of flailing here. Maybe not narratively

🤦‍♂️ Hopefully!

but this is something that I struggled with. While trying to get things working, I might move from attempt to attempt quickly.

How did we get here?

As alluded to before, there is a rule for vendoring dependencies: https://bazelbuild.github.io/rules_rust/crate_universe.html#crates_vendor. However, I don't really get its example. It specifies an annotation---is this related to a dependency? Does this supplant Cargo.toml information?

👀 I think there is a difference between minimal examples to help beginners and more full-featured examples to show what is possible. One thing that could help a beginner with filtering out some of the more advanced options is: documentation! In the context of code in examples, this most likely lives as comments. Having comments in this example would have been really helpful---it seemed so different from what I wanted that I ended up getting scared away.

This is too much. But, there are other worked examples! Let's try copying an example from the rules_rust repo. This is for our BUILD file in our backend directory. We will tweak it slightly becase we want to reuse the vendored dependencies we set up in cargo vendor:

load("@rules_rust//crate_universe:defs.bzl", "crates_vendor")

crates_vendor(
    name = "crates_vendor",
    manifests = [":Cargo.toml"],
    vendor_path = "3rd-party/crates",
    mode = "local",
)

load("@rules_rust//rust:defs.bzl", "rust_binary")
# load("@crate_index//:defs.bzl", "all_crate_deps")
load("//3rd-party/crates:defs.bzl", "all_crate_deps")

This leads to error messages saying that 3rd-party/crates is not a package. Hmm. Maybe it's due to a kind of cyclic issue? crates_vendor would create the //3rd-party/crates package, but then we try to use that same package potentially before it's created since it's in the same file. That's weird. Let's try upgrading our version of rules_rust and see if that solves it for us.

👀 rules_rust is now on version 0.5.0 while the documentation mentions 0.2.0. To further complicate things, while I was writing this example, the most recent release was version 0.4.0.

Keeping documentation up to date with code is hard, as evinced by the fact that this guide is immediately behind! I don't feel like this is a complaint regarding rust_rules's documentation because I can go and improve the situation through pull requests!

After updating our version (and syncing our lockfile), we see the same error. Maybe we can break the cycle by removing the code that relies on //3rd-party/crates. Now our backend BUILD file looks like

load("@rules_rust//crate_universe:defs.bzl", "crates_vendor")

crates_vendor(
    name = "crates_vendor",
    manifests = [":Cargo.toml"],
    vendor_path = "3rd-party/crates",
    mode = "local",
)

Note that we need to use bazel run instead of bazel build.

This actually works after one small correction. There's an issue with some versions of tokio that leads to errors on usage with bazel. The vendor_local_manifests example has a fix---let's borrow other people's workarounds.

There's a bigger philosophical issue here, though. This places the dependencies inside of backend. Whoops! We originally placed dependencies in the root level 3rd-party to enable sharing between different rust projects, potentially outside of the backend directory. If we want some of our utilities to use the same version as we do in our backend, this nested directory structure isn't condusive to that.

👀 The utilities dependency issue is just an example. Having other smaller services in a services directory would also motivate this sharing.

Trying /3rd-party/crates as the vendor_path gives me permissions issues, which makes me think that it's not actually true that

Absolute paths will be treated as relative to the workspace root

I've reported some of this difficulty in this GitHub issue.

Just as an experiment, let's try specifying the full path to 3rd-party/crates. This "works" as in it's able to vendor the dependencies.

The full path won't work in general as soon as we go to another machine, so let's try a relative path! Let's try ../3rd-party/crates and see if it works.

🤦‍♂️ I think at this point, https://bazelbuild.github.io/rules_rust/crate_universe.html#crates_vendor is starting to make some more sense. We have a specific target for vendoring the dependencies, and, for some reason, the example that we've been following combined that target with the general rust library target.

Anyway, let's try loading from it. Filling back in code that we removed, our BUILD file now ends with

load("//3rd-party/crates:defs.bzl", "all_crate_deps")

rust_binary(
    name = "hello_world",
    srcs = ["src/main.rs"],
    deps = all_crate_deps(normal = True,),
)

Finally, let's build just to make sure that everything works.

ERROR: /Users/preston/git/bazel-rust-guided-experiment/stage-2-crates-vendor/backend/BUILD.bazel:13:12: //backend:hello_world: invalid label '//backend/../3rd-party/crates/axum-0.5.6:axum' in element 0 of attribute 'deps' in 'rust_binary' rule: invalid package name 'backend/../3rd-party/crates/axum-0.5.6': package name component contains only '.' characters

So it looks like relative paths didn't work.

👀 So time for the file actually mentioned in the documentation, right?

🤦‍♂️ Yes. It still feels weird to me that the path being relative to the workspace root didn't work, but oh well!

Copying the vendoring part of our backend/BUILDFILE to another directory, we see

ERROR: /Users/preston/git/bazel-rust-guided-experiment/stage-2-crates-vendor/3rd-party/BUILD.bazel:6:14: no such target '//:Cargo.toml': target 'Cargo.toml' not declared in package ''; however, a source file of this name exists.  (Perhaps add 'exports_files(["Cargo.toml"])' to /BUILD?)

when running bazel run //3rd-party:crates_vendor. Following this suggestion works!

Now that we aren't using crate_index, let's remove it from our WORKSPACE file. It now ends with

load("@rules_rust//crate_universe:crates_deps.bzl", "crate_repositories")

crate_repositories()

The final version of our backend/BUILD.bazel and 3rd-party/BUILD.bazel are:

exports_files(["Cargo.toml"])

load("@rules_rust//rust:defs.bzl", "rust_binary")
load("//3rd-party/crates:defs.bzl", "all_crate_deps")

rust_binary(
    name = "hello_world",
    srcs = ["src/main.rs"],
    deps = all_crate_deps(normal = True,),
)

load("@rules_rust//crate_universe:defs.bzl", "crates_vendor")

crates_vendor(
    name = "crates_vendor",
    manifests = ["//backend:Cargo.toml"],
    mode = "local",
    vendor_path = "crates",
    tags = ["manual"],
)

Building our backend target again works! And uses our vendored dependencies.

Cleaning up previous steps

👀 Hmm, look at that, we don't need some of these files anymore!

🤦‍♂️ Yes, this actually took me reaching out on slack to realize. Because we aren't using crate_index anymore, we can also remove our old BUILD.bazel at the root level and the Cargo.Bazel.lock.

What did we do?

After a little bit of struggling with mixed BUILD files, we added a new BUILD file just for vendoring. If we vendor front-end dependencies, they can go here too! With this new separation, we got bazel building the project with our vendored dependencies.

https://github.com/prestontw/bazel-rust-guided-experiment/pull/2 is all of the steps we did but condensed down to avoid the experimentation and flailing:

  • we picked going with the 3rd-party BUILD file from the beginning,
  • we exposed backend's Cargo.toml, and
  • we updated out .cargo/config.toml file to point to the new directory.

What's next?

Let's go through the process of upgrading rust_rules since a new version came out since I started writing this!

Stage 3: Wrapping up

Source: https://github.com/prestontw/bazel-rust-guided-experiment/tree/main/src/stage-3-upgrade-version

Wow, we've done it! We've built our project with bazel, we are getting bazel to use our vendored dependencies, things are looking up for us!

There's actually one part that I forgot, something that is pretty important in a realistic Rust project. And that's local dependencies.

👀 There's also some small things that we can do, like using a more recent version of rules_rust.

How did we get here?

Let's go through the upgrade first, since it's pretty quick! rules_rust has its releases available on https://github.com/bazelbuild/rules_rust/releases. Let's update it:

http_archive(
    name = "rules_rust",
    sha256 = "73580f341f251f2fc633b73cdf74910f4da64d06a44c063cbf5c01b1de753ec1",
    urls = [
        "https://mirror.bazel.build/github.com/bazelbuild/rules_rust/releases/download/0.5.0/rules_rust-v0.5.0.tar.gz",
        "https://github.com/bazelbuild/rules_rust/releases/download/0.5.0/rules_rust-v0.5.0.tar.gz",
    ],
)

👀 Boom, done, next!

Local Dependency

This requires a little project restructuring. Let's move what we currently have in backend to a subdirectory so we can add a sibling project to contain the dependency. Let's call the new directory for the server, well, server. For simplicity's sake, let's call the new library math to contain the classic double function!

Let's work on making this project build on the cargo side, then make the necessary changes on the bazel side.

First, let's update our workspace Cargo.toml file:

[workspace]

members = [
  "backend/server",
  "backend/math",
  # deployments/best-ci-util-ever
]

And let's include a dependency on math and use the double function in server.

[dependencies]
math = { path = "../math" }

Running

cargo run

is available at localhost:3000!

🤦‍♂️ Sweet, seems good to me!

👀 What do you think, dear reader? Take a moment---does this directory structure match previous Rust projects you've worked in before?

Using a local dependency from bazel's side

We will follow https://bazelbuild.github.io/rules_rust/defs.html#rust_binary for this.

Specifically,

the library BUILD file

and

the binary BUILD file

Similar to how we exported our backend's Cargo.toml, we specify that math is a package. If we wanted to restrict access to this library, in case we wanted to reinforce architectural directory structure (like in https://youtu.be/5OjqD-ow8GE?t=2089), we could specify visibility other than public, but let's do that now for the sake of getting up and running quickly.

package(default_visibility = ["//visibility:public"])
exports_files(["Cargo.toml"])

load("@rules_rust//rust:defs.bzl", "rust_library")

rust_library(
    name = "math_lib",
    srcs = ["src/lib.rs"],
)

Before we update our backend/server/BUILD.bazel to list this library as a dependency, let's see if deps = all_crate_deps(), does this for us automatically. Let's run

bazel build //backend/server:hello_world

🤦‍♂️ Hmm, I'm running into a bunch of errors like

error[E0405]: cannot find trait `IntoResponse` in this scope
  --> backend/server/src/main.rs:41:11
   |
41 | ) -> impl IntoResponse {
   |           ^^^^^^^^^^^^ not found in this scope

I'll try running our vendoring command again and see if that works:

bazel run //3rd-party:crates_vendor

/Users/preston/git/bazel-rust-guided-experiment/src/stage-3-upgrade-version/3rd-party/BUILD.bazel:6:14: no such package 'backend': BUILD file not found in any of the following directories. Add a BUILD file to a directory to mark it as a package.

- /Users/preston/git/bazel-rust-guided-experiment/src/stage-3-upgrade-version/backend and referenced by '//3rd-party:crates_vendor'

Ahh, yep, let's update our paths for our vendoring BUILD.bazel.

👀 If math needed dependencies to function, we would probably add its Cargo.toml at this point in the same location.

Trying to re-vendor, I get

INFO: Build completed successfully, 539 total actions
thread 'main' panicked at 'called `Option::unwrap()` on a `None` value', external/rules_rust/crate_universe/src/splicing/splicer.rs:78:86
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

🤦‍♂️ Let's try adding math's Cargo.toml to this list.

👀 Should this work? If we just add the Cargo.toml here, do we expect an error or a successful result? If it's an error, what error do we expect?

Before bazel will let us do this, we need to export math's Cargo.toml. But, I'm still running into the same error. This might be evidence that we need to add our dependency on math to our server/BUILD.bazel, or it could be that we are off the beaten path.

🤦‍♂️ I actually took a beat here to take a break. I checked to see if there were any examples that were doing what we are trying to do. I couldn't find any. After some more flailing, running lots of bazel clean's, trying to revendor and running into errors, I decided to try another approach.

I started from stage-2, but instead of going straight to adding a local dependency, I moved the folders to a structure that would support this. I noticed two things:

  1. I was running into the same error, and
  2. The directory structure didn't make much sense.

Why is there this random nested folder? This pointed me to think that I was too quick to copy the server functionality into a sub-folder. Let's try again, but with server's files inside of backend.

That gives us

[workspace]

members = [
  "backend",
  "backend/math",
  # deployments/best-ci-util-ever
]

and

[package]
name = "example-readme"
version = "0.1.0"
edition = "2018"
publish = false

[dependencies]
math = { path = "./math" }

for our root-level and "server" Cargo.toml's. Let's try from this point and see if we run into the same errors.

👀 Let's check after all of this flailing that we are actually usingnotrust math and that everything is setup correctly, including Cargo.toml's as well as BUILD.bazel's. This includes incorporating our dependency on math in our BUILD.bazel:

    deps = ["//backend/math:math_lib"] + all_crate_deps(),

Unexpectedly, this doesn't work:

ERROR: /Users/preston/git/bazel-rust-guided-experiment/src/stage-3-upgrade-version/backend/BUILD.bazel:6:12: Compiling Rust bin hello_world (1 files) failed: (Exit 1): process_wrapper failed: error executing command bazel-out/darwin_arm64-opt-exec-2B5CBBC6/bin/external/rules_rust/util/process_wrapper/process_wrapper --arg-file ... (remaining 130 arguments skipped)

Use --sandbox_debug to see verbose messages from the sandbox
error[E0433]: failed to resolve: use of undeclared crate or module `math`
  --> backend/src/main.rs:44:13
   |
44 |         id: math::double(1337),
   |             ^^^^ use of undeclared crate or module `math`

error: aborting due to previous error

This works with cargo, let's play around with our bazel rules a little. Looking at an example, they specify the local dependency without the :<name> format. Let's make the necessary changes to our math library and reference it as such:

    name = "math",

    deps = ["//backend/math"] + all_crate_deps(),

Running bazel build //...

INFO: Analyzed 3 targets (2 packages loaded, 4 targets configured).
INFO: Found 3 targets...
INFO: Elapsed time: 0.898s, Critical Path: 0.78s
INFO: 2 processes: 1 internal, 1 darwin-sandbox.
INFO: Build completed successfully, 2 total actions

🤦‍♂️ I can't belive that works!

👀 Most of the pain points we've run into seem to be assumptions around directory structure that aren't immediately clear. We faced this for dependencies, original repo layout, and now for integrating local dependencies. But there seems to be some kind of weird coupling between local dependency crate names and how we label them as rust_library's!?

A lot of the directory issues made a lot of sense and would probably be how a more robust, more mature codebase would be structured, but all of those project structures worked fine just with Cargo. Bazel is more opinionated in its folder structure. One of the reasons why I'm writing this is to show a directory structure that works.

What did we do?

We made our project a little more realistic! We used a more up to date version of rules_rust and added a local dependency to our project! We also found another assumption bazel makes for source folders and an unexpected assumption around local dependency names.

What's next?

It's up to you!

🎉 Doo-do-de-do do-do-do-doo!

Some ideas I have are:

  • Running tests through bazel. I've done this before rather quickly, it shouldn't be too bad. They have nice utility rules we can use to enumerate all of the tests automatically.
  • Running clippy and formatting through bazel.
  • Building a Docker container containing our Rust application.
  • Actually using this in a CI pipeline, such as CircleCI!

As far as what I want to do now that this is done:

  • Give documentation feedback to rules_rust.

    • Some of the naming is confusing for a beginner, such as crates_repository vs crate_repositories, but it seems like it is with the rest of the bazel ecosystem. Maybe improved documentation on what these nouns mean would be helpful?
    • Add an example of vendoring with manifests in the rules_rust documentation. Both the examples for Cargo Workspaces and Direct Packages use crates_repository. I think that would make it clearer that there is a matrix of decisions between expressing dependencies (Cargo.toml or directly) and how those are made available for use (downloaded or vendored).
    • Add more comments in the sample files! This can help beginners decipher what is necessary, why we are specifying something, etc. There's a tension between making things beginner friendly and being complete in documentation, and I think code comments can help with this.

  • Think about how much of this can be automatic. One easy one would be using the rust-toolchain file instead of duplicating that information in the WORKSPACE.bazel file.

    There have been several times in this experience where cargo build works fine, but bazel is confused and errors. A more difficult, but more amazing ask, is if a lot of this bazel infrastructure could be automatically generated. crates_universe does that for dependencies, but I'm imagining something like having something that we can run as cargo run --bin <x> being available as bazel run //<x> without a specific rust_binary BUILD.bazel file being written by the user. There are some other rules that accomplish this to varying degrees, like npm packages with bin entries. cargo-raze might actually do this---I might take another look at it and see if I can get it working.

🤦‍♂️ This has been fun, y'all. Thank you for sticking through this with me. I hope that I've shown you that it is possible to integrate bazel and cargo and that there are many opportunities for improvements here.