pub trait Lexeme<StorageT>: Debug + Display + Eq + Hash + Copy {
    fn new(tok_id: StorageT, start: usize, len: usize) -> Self
        Self: Sized
; fn new_faulty(tok_id: StorageT, start: usize, len: usize) -> Self
        Self: Sized
; fn tok_id(&self) -> StorageT; fn span(&self) -> Span; fn faulty(&self) -> bool; }
Expand description

A lexeme represents a segment of the user’s input that conforms to a known type: this trait captures the common behaviour of all lexeme structs.

Lexemes are assumed to have a definition which describes all possible correct lexemes (e.g. the regular expression [0-9]+ defines all integer lexemes). This trait also allows “faulty” lexemes to be represented – that is, lexemes that have resulted from error recovery of some sort. Faulty lexemes can violate the lexeme’s type definition in any possible way (e.g. they might span more or less input than the definition would suggest is possible).

Required Methods

Create a new lexeme with ID tok_id, a starting position in the input start, and length len.

Lexemes created using this function are expected to be “correct” in the sense that they fully respect the lexeme’s definition semantics. To create faulty lexemes, use new_faulty.

Create a new faulty lexeme with ID tok_id and a starting position in the input start.

The token ID.

Obtain this Lexeme’s Span.

Returns true if this lexeme is “faulty” i.e. is the result of error recovery in some way. If true, note that the lexeme’s span may be greater or less than you may expect from the lexeme’s definition.