[Clang Tools] Source-to-source transformations

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

[Clang Tools] Source-to-source transformations

Tom Stellard via cfe-dev
Hello all,

We are developing a tool for performing some source-to-source transformations. I have some technical questions regarding the approach we are following for this purpose:
As far as I understand, Clang AST is meant to be immutable, therefore once created (for instance with the ASTFrontendAction), I guess no Node (Stmt, Decl, whatever) must be modified.
Thus, our approach is basically to parse the source file and generate its AST, feed our algorithms for generating code using some information of the AST (patterns found with MatcherFinder, variable names, etc.), and then generating code (basically strings) using Replacements and Rewriter.
My main concerns are:
  • I have not delved deep in all the FrontendAction subclasses in Clang. For our purpose, is ASTFrontendAction suitable? Are there any other better choices?
  • Is it fine to not modify the original AST when performing source-to-source transformations? My first idea was to modify it until I read the "immutability philosophy". On the other hand, I only care about the AST for parsing, not for the output (I am already performing the modifications with Rewriter, right?).
I am asking this just to be sure that I understand how Clang Tools work or if I am misunderstanding something.
Thank you so much.

Kind regards,

--

Marcos Horro

PhD Student

Universidade da Coruña (UDC)

http://gac.udc.es/~marcos.horro/ 


_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: [Clang Tools] Source-to-source transformations

Tom Stellard via cfe-dev
Am So., 1. Dez. 2019 um 23:23 Uhr schrieb Marcos Horro Varela via
cfe-dev <[hidden email]>:
> I have not delved deep in all the FrontendAction subclasses in Clang. For our purpose, is ASTFrontendAction suitable? Are there any other better choices?

Not an expert, but I think deriving from ASTFontendAction is exactly
intended for such as use case.

> Is it fine to not modify the original AST when performing source-to-source transformations? My first idea was to modify it until I read the "immutability philosophy". On the other hand, I only care about the AST for parsing, not for the output (I am already performing the modifications with Rewriter, right?).

It is always fine to **not** do something. Indeed, after parsing, the
AST is not supposed to be modified and you would get unexpected
results if you try (some analysis is baked in at this stage, e.g. name
lookup, overload resolution, implicit casts, etc). You can, however,
create a new but modified AST from the parsed one. Template
instantiation uses this via the TreeTransform helper. I still suggest
to avoid this if you don't need to. Instead as you mentioned you can
use clang::Rewriter to directly modify the input source characters
that can be found using SourceLocation. Note there is also
clang::tooling::Transformer to help with this task.

Michael
_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: [Clang Tools] Source-to-source transformations

Tom Stellard via cfe-dev
+1 to taking a look at clang::tooling::Transformer, which is the adaptor that ties the libraries in clang::transformer to actual tools. I'm happy to answer questions.

You should also consider whether your tool can be written as a clang tidy check rather than using ASTFrontendAction, since that saves you from needing to write your own tool around the transformation.  However, it also means you're confined to clang-tidy's interface. So, it has pluses and minuses.  
If you decide to go with clang-tidy, you can use the adaptor https://github.com/llvm/llvm-project/blob/master/clang-tools-extra/clang-tidy/utils/TransformerClangTidyCheck.h to tie to clang::transformer. 

On Mon, Dec 2, 2019 at 11:56 AM Michael Kruse via cfe-dev <[hidden email]> wrote:
Am So., 1. Dez. 2019 um 23:23 Uhr schrieb Marcos Horro Varela via
cfe-dev <[hidden email]>:
> I have not delved deep in all the FrontendAction subclasses in Clang. For our purpose, is ASTFrontendAction suitable? Are there any other better choices?

Not an expert, but I think deriving from ASTFontendAction is exactly
intended for such as use case.

> Is it fine to not modify the original AST when performing source-to-source transformations? My first idea was to modify it until I read the "immutability philosophy". On the other hand, I only care about the AST for parsing, not for the output (I am already performing the modifications with Rewriter, right?).

It is always fine to **not** do something. Indeed, after parsing, the
AST is not supposed to be modified and you would get unexpected
results if you try (some analysis is baked in at this stage, e.g. name
lookup, overload resolution, implicit casts, etc). You can, however,
create a new but modified AST from the parsed one. Template
instantiation uses this via the TreeTransform helper. I still suggest
to avoid this if you don't need to. Instead as you mentioned you can
use clang::Rewriter to directly modify the input source characters
that can be found using SourceLocation. Note there is also
clang::tooling::Transformer to help with this task.

Michael
_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev

_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev

smime.p7s (6K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [Clang Tools] Source-to-source transformations

Tom Stellard via cfe-dev
In reply to this post by Tom Stellard via cfe-dev
Michael, Yitzhak,

Thank you so much for your comments. I think I will go with the ASTFrontendAction in detriment of clang-tidy in order to have more "freedom" when writing the tool. Nonetheless, I am quite concerned regarding my decision of not modifying the AST. Maybe using the TreeTransform helper would be a better decision. On the other hand, I do not see the utility of performing modifications in the AST; could you provide me an example where it is useful rather than "just parsing and replacing code"?
Again, thank you so much.

Best regards,

--

Marcos Horro

PhD Student

Universidade da Coruña (UDC)

http://gac.udc.es/~marcos.horro/ 



De: Yitzhak Mandelbaum
Enviado: Martes, 03 de decembro de 2019 20:59
Para: Michael Kruse
CC: Marcos Horro Varela; via cfe-dev
Asunto: Re: [cfe-dev] [Clang Tools] Source-to-source transformations

+1 to taking a look at clang::tooling::Transformer, which is the adaptor that ties the libraries in clang::transformer to actual tools. I'm happy to answer questions.

You should also consider whether your tool can be written as a clang tidy check rather than using ASTFrontendAction, since that saves you from needing to write your own tool around the transformation.  However, it also means you're confined to clang-tidy's interface. So, it has pluses and minuses.  
If you decide to go with clang-tidy, you can use the adaptor https://github.com/llvm/llvm-project/blob/master/clang-tools-extra/clang-tidy/utils/TransformerClangTidyCheck.h to tie to clang::transformer. 

On Mon, Dec 2, 2019 at 11:56 AM Michael Kruse via cfe-dev <[hidden email]> wrote:
Am So., 1. Dez. 2019 um 23:23 Uhr schrieb Marcos Horro Varela via
cfe-dev <[hidden email]>:
> I have not delved deep in all the FrontendAction subclasses in Clang. For our purpose, is ASTFrontendAction suitable? Are there any other better choices?

Not an expert, but I think deriving from ASTFontendAction is exactly
intended for such as use case.

> Is it fine to not modify the original AST when performing source-to-source transformations? My first idea was to modify it until I read the "immutability philosophy". On the other hand, I only care about the AST for parsing, not for the output (I am already performing the modifications with Rewriter, right?).

It is always fine to **not** do something. Indeed, after parsing, the
AST is not supposed to be modified and you would get unexpected
results if you try (some analysis is baked in at this stage, e.g. name
lookup, overload resolution, implicit casts, etc). You can, however,
create a new but modified AST from the parsed one. Template
instantiation uses this via the TreeTransform helper. I still suggest
to avoid this if you don't need to. Instead as you mentioned you can
use clang::Rewriter to directly modify the input source characters
that can be found using SourceLocation. Note there is also
clang::tooling::Transformer to help with this task.

Michael
_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev

_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: [Clang Tools] Source-to-source transformations

Tom Stellard via cfe-dev
Am Di., 3. Dez. 2019 um 14:54 Uhr schrieb Marcos Horro Varela
<[hidden email]>:
> Thank you so much for your comments. I think I will go with the ASTFrontendAction in detriment of clang-tidy in order to have more "freedom" when writing the tool. Nonetheless, I am quite concerned regarding my decision of not modifying the AST. Maybe using the TreeTransform helper would be a better decision. On the other hand, I do not see the utility of performing modifications in the AST; could you provide me an example where it is useful rather than "just parsing and replacing code"?

If you need semantic analysis such as name lookup, overload
resolution, etc. for your changed code. For instance, you want to
replace a type, which might transitively require to change types of
other variables, function result- and parameter types. However, the
AST was not designed with such a use case in mind and you will need to
write a lot of code to handle special situations (e.g. re-do overload
resolution with the new type; however the overload set depends on
clang::Scope which is only available during parsing). I strongly
suggest to pursue the parse-and-replace approach if possible.

Michael
_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: [Clang Tools] Source-to-source transformations

Tom Stellard via cfe-dev
If you need semantic analysis such as name lookup, overload
resolution, etc. for your changed code. For instance, you want to
replace a type, which might transitively require to change types of
other variables, function result- and parameter types. However, the
AST was not designed with such a use case in mind and you will need to
write a lot of code to handle special situations (e.g. re-do overload
resolution with the new type; however the overload set depends on
clang::Scope which is only available during parsing). I strongly
suggest to pursue the parse-and-replace approach if possible.

I see, in my case I think is crystal clear that I do not have to perform modifications in the AST so my life will be easier.
Thanks for the clarification and prompt response.

Bests,
On Dec 5 2019, at 2:03 pm, Michael Kruse <[hidden email]> wrote:
Am Di., 3. Dez. 2019 um 14:54 Uhr schrieb Marcos Horro Varela
Thank you so much for your comments. I think I will go with the ASTFrontendAction in detriment of clang-tidy in order to have more "freedom" when writing the tool. Nonetheless, I am quite concerned regarding my decision of not modifying the AST. Maybe using the TreeTransform helper would be a better decision. On the other hand, I do not see the utility of performing modifications in the AST; could you provide me an example where it is useful rather than "just parsing and replacing code"?

If you need semantic analysis such as name lookup, overload
resolution, etc. for your changed code. For instance, you want to
replace a type, which might transitively require to change types of
other variables, function result- and parameter types. However, the
AST was not designed with such a use case in mind and you will need to
write a lot of code to handle special situations (e.g. re-do overload
resolution with the new type; however the overload set depends on
clang::Scope which is only available during parsing). I strongly
suggest to pursue the parse-and-replace approach if possible.

Michael

_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev