Chaining ASTConsumers

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Chaining ASTConsumers

ASDen
Hello ,

I'm writing a source to source tool, according to list prefred way is Rewriter class , but also i can modify list by a series of traversals

the problem is Rewriter class method is really primitive,, isn't suitable for big changes (esp. in structure )

what i'm asking for is, can i chain ASTConsumer's ,,so i can progressively modify AST and pretty-print it finally
but it looks like (from ParseAST) i would have to Parser -> ASTConsumer ->  pretty-print ->Parser ->ASTConsumer -> .. and so on
which is very tedious

is the there an explicit AST data structure ? can i just parse once and modify resulting AST many times  (traverse,add/delete nodes it not re-parse source) ?

Thanks,
Mohamed Yousef

_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: Chaining ASTConsumers

Demon Insight
I've been successful with the following approach:

- use an ASTConsumer implementation that can delegate to yet another instance of an ASTConsumer
- the first consumer performs its ASTConsumer callbacks (and, if necessary, node transformations) *before* the delegate consumer:

ASTConsumer * m_delegate;

void HandleTopLevelDecl (DeclGroupRef group)
{
        for (DeclGroupRef::iterator node = group.begin(), n_limit = group.end (); n != n_lmit; ++ n)
        {
                // [do whatever you want with 'node', including AST transforms: this will typically use DeclVisitor and StmtVisitor to handle just the decls and expressions of interest]
        }

        m_delegate-> HandleTopLevelDecl (visit); // now let the "real" consumer see what we've done
}

// forward other ASTConsumer methods to 'm_delegate' as well so it knowns when a translation unit is done, etc

So, when my ASTConsumer is an AST transformer and m_delegate is, say, a CodeGenerator instance, I get to modify AST just before it gets compiled into native code. It works because the "real" AST consumer (CodeGenerator) only sees the nodes that I want it to see, possibly after a transformation.

Of course, not every AST transform can be done this way, but for additive/substitutive tree transformations this works well and does not require a pretty-printing pass. It is easy to imagine this working for an entire chain/pipeline of AST manipulators.

HTH,
Vlad


On May 7, 2010, at 7:22 AM, Mohamed Yousef wrote:

> Hello ,
>
> I'm writing a source to source tool, according to list prefred way is Rewriter class , but also i can modify list by a series of traversals
>
> the problem is Rewriter class method is really primitive,, isn't suitable for big changes (esp. in structure )
>
> what i'm asking for is, can i chain ASTConsumer's ,,so i can progressively modify AST and pretty-print it finally
> but it looks like (from ParseAST) i would have to Parser -> ASTConsumer ->  pretty-print ->Parser ->ASTConsumer -> .. and so on
> which is very tedious
>
> is the there an explicit AST data structure ? can i just parse once and modify resulting AST many times  (traverse,add/delete nodes it not re-parse source) ?
>
> Thanks,
> Mohamed Yousef
> _______________________________________________
> cfe-dev mailing list
> [hidden email]
> http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev


_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: Chaining ASTConsumers

Douglas Gregor
In reply to this post by ASDen

On May 7, 2010, at 6:22 AM, Mohamed Yousef wrote:

> Hello ,
>
> I'm writing a source to source tool, according to list prefred way is Rewriter class , but also i can modify list by a series of traversals
>
> the problem is Rewriter class method is really primitive,, isn't suitable for big changes (esp. in structure )

Right.

> what i'm asking for is, can i chain ASTConsumer's ,,so i can progressively modify AST and pretty-print it finally
> but it looks like (from ParseAST) i would have to Parser -> ASTConsumer ->  pretty-print ->Parser ->ASTConsumer -> .. and so on
> which is very tedious

The AST pretty-print isn't guaranteed to produce output that can be parsed. It would be wonderful if we could get there, but it's going to take some serious work to get it right.

> is the there an explicit AST data structure ? can i just parse once and modify resulting AST many times  (traverse,add/delete nodes it not re-parse source) ?

The Clang ASTs are explicit data structures representing the entire translation unit. Once constructed, ASTs are essentially immutable, so you won't be able to add/delete nodes easily. You might be able to write an AST -> AST transformation that performs the modifications on the source AST as it builds the resulting AST (perhaps even building up a chain of such transformations).

        - Doug
_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: Chaining ASTConsumers

ASDen

The AST pretty-print isn't guaranteed to produce output that can be parsed. It would be wonderful if we could get there, but it's going to take some serious work to get it right.


how and in what aspects specifically (examples of pretty output that can't be re-parsed ? ) , the only one i saw was Null nodes (still on testing small C things , haven't tried any real thing yet :) )

I've managed to pseudo-chain ASTConsumers , i used a variation on Vlad's idea , a custom ParseAST that accepts a vector of ASTConsumers , calls them one by one.
I'm also getting good results with in-place AST modifications , that i chain as separate ASTConsumers ,, that's why i need to know the exact limitations of pretty-print , upon which i depend now

Thanks,
Mohammd Yousef

_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev