Module `CLexer.LexerDiff`

Create a lexer. true enables alternate handling for computing diffs. It ensures that, ignoring white space, the concatenated tokens equal the input string. Specifically:

for strings, return the enclosing quotes as tokens and treat the quoted value as if it was unquoted, possibly becoming multiple tokens
for comments, return the "(*" as a token and treat the contents of the comment as if it was not in a comment, possibly becoming multiple tokens
return any unrecognized Ascii or UTF-8 character as a string

type te = Tok.t
type 'c pattern = 'c Tok.p

val tok_pattern_eq : 'a pattern -> 'b pattern -> ('a, 'b) Util.eq option
val tok_pattern_strings : 'c pattern -> string * string option
val tok_func : te Gramlib.Plexing.lexer_func
val tok_using : 'c pattern -> unit
val tok_removing : 'c pattern -> unit
val tok_match : 'c pattern -> te -> 'c
val tok_text : 'c pattern -> string