Rust Procedural Macro Spans Quirks Mode and Hygiene

Rustc Version 1.45 and Span::mixed_site, Span::resolved_at, Span::located_at

Rust version 1.45 stabilizes a lot of facilities related to procedural macros and blame/error reporting. While getting up to speed from rustc ~1.16 I found that these really are mostly facilities and not so much burdens. In most cases the worst result of a bad span implementation in a macro will be a confusing error message. However for Rdxl macros this is not the case. In Rdxl, breaking spaces are inserted depending on the position of lexed symbols, so precise span locations are required for this major feature to work properly.


is not the same as

xhtml!(a + c)

Span lines/columns and cursor positions

The first concept that is important to understand the precise nature of Spans is the difference between Spans and LineColumn locations. A LineColumn is a text cursor located in between string codepoints somewhere in a file. These could be the start of a line, the end of a line, or somewhere positioned in between symbols. A Span, by comparison, is the representation of a range of lexed symbols from one LineColumn to the next. Each Span has a SourceFile, a .start() LineColumn and an .end() LineColumn.

A note of caution

Spans are not "just" cursors and lines and columns. As of the current rust stable release 1.45 and nightly 1.47 spans can not be instantiated like a Token can be. Spans cannot be passed through from a proc_macro::TokenStream into a proc_macro2::TokenStream. This means, for now, that macros that parse tokens into a proc_macro2::TokenStream, like the quote crate, must shed all spans from the quoted section. No information regarding how deep in or where exactly a token is located at can be reexported. This information is lost. This makes testing procedural macros very tricky and for now the best utility for doing so is maybe "cargo expand". That means that a human has to look through each macro dump to see if it looks good or not. The result is that probably very few people debug their macros this way infrequently or at all.