How context sensitive lexical analysis is done?

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

How context sensitive lexical analysis is done?

Bakhvalov, Denis via cfe-dev
Hi,

I see that clang uses "non-reference lexical grammar" to deal with
context-sensitive lexical analysis. Could anybody give some pointer on
how exactly this is done to deal with the context-sensitive grammar?

https://en.wikipedia.org/wiki/The_lexer_hack#Alternative_solutions

I am interested in parse shell code.

For example, in normal mode, no spaces are allowed in the assignment.

x=10

But in math mode, spaces are allowed.

((x = 10))

Bash currently just tokenizes both cases as single tokens (`x=10` and
`((x = 10))`, respectively).

I think that if one were to tokenize at a finer level and let the
grammar to handle the assignment, then one will need different lexers
in different contexts. But this potentially can allow the shell
language to be more expressive. So I'd like to understand how this can
be done.

--
Regards,
Peng
_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev