Rust Procedural Macro Spans Quirks Mode and Hygiene

Rustc Version 1.45 and Span::mixed_site, Span::resolved_at, Span::located_at

Rust version 1.45 stabilizes a lot of facilities related to procedural macros and blame/error reporting. While getting up to speed from rustc ~1.16 I found that these really are mostly facilities and not so much burdens. In most cases the worst result of a bad span implementation in a macro will be a confusing error message. However for Rdxl macros this is not the case. In Rdxl, breaking spaces are inserted depending on the position of lexed symbols, so precise span locations are required for this major feature to work properly.

xhtml!(a+c)

is not the same as

xhtml!(a + c)

Span lines/columns and cursor positions

The first concept that is important to understand the precise nature of Spans is the difference between Spans and LineColumn locations. A LineColumn is a text cursor located in between string codepoints somewhere in a file. These could be the start of a line, the end of a line, or somewhere positioned in between symbols. A Span, by comparison, is the representation of a range of lexed symbols from one LineColumn to the next. Each Span has a SourceFile, a .start() LineColumn and an .end() LineColumn.

A note of caution

Spans are not "just" cursors and lines and columns. As of the current rust stable release 1.45 and nightly 1.47 spans can not be instantiated like a Token can be. Spans cannot be passed through from a proc_macro::TokenStream into a proc_macro2::TokenStream. This means, for now, that macros that parse tokens into a proc_macro2::TokenStream, like the quote crate, must shed all spans from the quoted section. No information regarding how deep in or where exactly a token is located at can be reexported. This information is lost. This makes testing procedural macros very tricky and for now the best utility for doing so is maybe "cargo expand". That means that a human has to look through each macro dump to see if it looks good or not. The result is that probably very few people debug their macros this way infrequently or at all.

Andrew Johnson

Open Source:

• Github

Other Platforms

• Medium

Published Books:

• Hands on Functional Programming in Rust
• LSTS Tutorial

Books that I am reading:

• E's
• Combinatorial Optimization
• Categories for the Working Mathematician

Top Referrers:

1. Google Search
2. DuckDuckGo Search
3. Sogou Search
4. Bing Search
5. Baidu Search

Tools:

• Perplexity 😵 Modelling Language
• A Categorical view of Typed Lambda Calculus with flexible Soundness Guarantees
• JLPT Flash Cards

Weblog:

• Unifying Input Experience from Desktop and Mobile
• Directed Graphs, Tensors, and Neural Networks in Rust
• Speech to Text Facilities in Rust
• Compiling Word Lists from Wiktionary Content
• RDXL Static Site Generation Template
• Anonymous Enum Types and Polymorphism in Rust
• Rust Smallvec Optimization
• FRU Syntax vs Builder Pattern in Rust
• Using RDXL with Rocket
• Parsing Natural Language with Machine Learning and Ontologies
• General Concepts for Multilingual Programming
• Closures in Rust
• Rust, shared: the mutability checker
• Rust, typed: type states and zero-sized types
• Rust, abstracted: lambda lifting
• Rust, asynchronized: what is a Rust Future?
• Is Rust a purely functional programming language?