Traversing the (macro-expanded) tokens that make up a Decl?

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Traversing the (macro-expanded) tokens that make up a Decl?

Richard Pennington via cfe-dev
My problem is as follows:  I have a FunctionDecl d for which I know that
d->isConstexpr(), so somewhere in d->getSourceRange() the token
"constexpr" should appear.  Now, I want to know whether that "constexpr"
is from the expansion of a specific object-like macro

   #define M constexpr

(whose use could be nested in some other macros, so it's important here
to look into macro invocations in general).

I'm trying to traverse d->getSourceRange() with
Lexer::MeasureTokenLength/SourceManager::getCharacterData to get at a
token's content, then SourceLocation::getLocWithOffset to get at the
next token.  Which works more-or-less well.  However:

* When d->getSourceRange() happens to start off within a macro
expansion, I could fall off its end before reaching the declaration's
end.  This one is easy, I can use
SourceManager::isAtEndOfImmediateMacroExpansion to climb back out of the
macro expansion and continue traversal one level further out.

* But when the traversal reaches a macro invocation, I get the token of
that invocation (multiple tokens in case of a function-like macro with
its parentheses and arguments), without a clue that it is a macro and
without a way to step into its expansion.  Am I missing anything, or is
there just no way to do that?

(I find at least two unanswered questions on stackoverflow that look
like they ask the same question,
<http://stackoverflow.com/questions/30831936/how-to-get-tokens-for-a-declaration-using-clang-c-api>
and
<http://stackoverflow.com/questions/36235445/clang-getting-preprocessed-characters-with-sourcelocation-sourcerange>.)
_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Traversing the (macro-expanded) tokens that make up a Decl?

Richard Pennington via cfe-dev
On Thu, Mar 2, 2017 at 1:10 AM, Stephan Bergmann via cfe-dev <[hidden email]> wrote:
My problem is as follows:  I have a FunctionDecl d for which I know that d->isConstexpr(), so somewhere in d->getSourceRange() the token "constexpr" should appear.  Now, I want to know whether that "constexpr" is from the expansion of a specific object-like macro

  #define M constexpr

(whose use could be nested in some other macros, so it's important here to look into macro invocations in general).

I'm trying to traverse d->getSourceRange() with Lexer::MeasureTokenLength/SourceManager::getCharacterData to get at a token's content, then SourceLocation::getLocWithOffset to get at the next token.  Which works more-or-less well.  However:

* When d->getSourceRange() happens to start off within a macro expansion, I could fall off its end before reaching the declaration's end.  This one is easy, I can use SourceManager::isAtEndOfImmediateMacroExpansion to climb back out of the macro expansion and continue traversal one level further out.

* But when the traversal reaches a macro invocation, I get the token of that invocation (multiple tokens in case of a function-like macro with its parentheses and arguments), without a clue that it is a macro and without a way to step into its expansion.  Am I missing anything, or is there just no way to do that?

You might try to make a lexer to Lex(token) for you until you've reached the end of the range, rather than navigating by source locations.  Though actually I think we should make the AST record the location of the `constexpr` keyword, so that applications don't have to jump through these hoops.

-- James

_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Traversing the (macro-expanded) tokens that make up a Decl?

Richard Pennington via cfe-dev
On 03/03/2017 01:05 AM, James Dennett wrote:

> On Thu, Mar 2, 2017 at 1:10 AM, Stephan Bergmann via cfe-dev
> <[hidden email] <mailto:[hidden email]>> wrote:
>
>     My problem is as follows:  I have a FunctionDecl d for which I know
>     that d->isConstexpr(), so somewhere in d->getSourceRange() the token
>     "constexpr" should appear.  Now, I want to know whether that
>     "constexpr" is from the expansion of a specific object-like macro
>
>       #define M constexpr
>
>     (whose use could be nested in some other macros, so it's important
>     here to look into macro invocations in general).
>
>     I'm trying to traverse d->getSourceRange() with
>     Lexer::MeasureTokenLength/SourceManager::getCharacterData to get at
>     a token's content, then SourceLocation::getLocWithOffset to get at
>     the next token.  Which works more-or-less well.  However:
>
>     * When d->getSourceRange() happens to start off within a macro
>     expansion, I could fall off its end before reaching the
>     declaration's end.  This one is easy, I can use
>     SourceManager::isAtEndOfImmediateMacroExpansion to climb back out of
>     the macro expansion and continue traversal one level further out.
>
>     * But when the traversal reaches a macro invocation, I get the token
>     of that invocation (multiple tokens in case of a function-like macro
>     with its parentheses and arguments), without a clue that it is a
>     macro and without a way to step into its expansion.  Am I missing
>     anything, or is there just no way to do that?

(Another issue to watch out for would be preprocessing directives, e.g.,
skipping over conditional groups when encountering an #else line.)

> You might try to make a lexer to Lex(token) for you until you've reached
> the end of the range, rather than navigating by source locations.

But I assume I'd need to re-lex all from the beginning, so that the
lexer has an accurate idea of what macros are currently defined and
whether we're in an #if group (so it wouldn't fail with an error if it
encounters a---semingly unmatched---#endif line, if that actually is an
issue).

> Though actually I think we should make the AST record the location of
> the `constexpr` keyword, so that applications don't have to jump through
> these hoops.

Yeah, I guess I understand now why new "get SourceLocation of ..."
functions keep appearing in include/clang/AST/ as needed (e.g.,
FunctionDecl::getExceptionSpecSourceRange).  But would we want to track
the locations of e.g. all specifiers in Decl instances?  (Or would that
be considered making such instances "too fat"?)

Anyway, my immediate need for the original issue is meanwhile gone
again, so I at least won't pursue this further for now.
_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Loading...