[Proposal] New tool contributions to clang repository (gcc/g++ compilation database converter, plist to HTML converter)

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

[Proposal] New tool contributions to clang repository (gcc/g++ compilation database converter, plist to HTML converter)

David Blaikie via cfe-dev

Hi All,
 
There are multiple features in CodeChecker which could be split into smaller tools so the community could benefit from them.

1. gcc/g++ compilation database to Clang compilation database conversion tool
2. Plist to HTML report conversion tool

1. gcc/g++ compilation database to Clang compilation database conversion tool

One problem is what we want to address is that gcc/g++ compilation commands cannot be forwarded without modification
to the Clang based tools as they were logged or exported by the build system.
 
Post-processing of the compilation command is required:
- might contain compilation flags and build targets which are not compatible with Clang (needs to be filtered out)
- the compilation argument filtering needs to be done in every tool which uses the compilation database json (scan-build, scan-build-py, CodeChecker)
- when gcc is used as a cross compiler, hard-coded include paths, flags and compilation target of gcc/g++ are missing from the command line argument list. These needs to be auto-detected and passed to Clang before analysis.
- C/C++ standard versions might be missing from the compilation commands which could change the analysis results (c++11 checker triggers for c++98 code)
 
Our idea is to create a separate tool which could transform a gcc/g++ compilation database file into a compile command json file which can be used by Clang based tools without further modification.
The non-compatible flags are filtered out and the necessary cross compilation include paths and compilation flags are added.
Other Clang based tools could also benefit from the converted compilation database json, like clangd.
 
2. Plist to HTML report conversion tool (see the source code at https://github.com/Ericsson/codechecker/tree/master/vendor/plist_to_html )

Currently Clang Static Analyzer can produce HTML output but this should not be done necessarily by the analyzer.
Plist reports can contain much more information and are easier to manage and parse by other tools compared to the html reports.
With our HTML generator tool we can generate static HTML reports which look similar to the report visualization in Xcode or
to the report viewer web UI of CodeChecker. You can move on each event through a bug path and arrows are shown for easier understanding.
 
Let us know what do you think about these tools, how well would they fit near the available tools.
Your feedback is appreciated.
 
Thanks,
Gyorgy Orban
CodeChecker Team


_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: [Proposal] New tool contributions to clang repository (gcc/g++ compilation database converter, plist to HTML converter)

David Blaikie via cfe-dev
Hi Gyorgy,

On Sep 25, 2018, at 11:41 AM, György Orbán via cfe-dev <[hidden email]> wrote:
2. Plist to HTML report conversion tool

I’d really love to have that in Clang, and I think that is an amazing tool!
However, I’ve tried to launch it and failed — it seemed to look for files which are not there.
I would be happy to review patches for that — but I think the tool would require quite a lot of work.


1. gcc/g++ compilation database to Clang compilation database conversion tool

One problem is what we want to address is that gcc/g++ compilation commands cannot be forwarded without modification
to the Clang based tools as they were logged or exported by the build system.
 
Post-processing of the compilation command is required:
- might contain compilation flags and build targets which are not compatible with Clang (needs to be filtered out)
- the compilation argument filtering needs to be done in every tool which uses the compilation database json (scan-build, scan-build-py, CodeChecker)
- when gcc is used as a cross compiler, hard-coded include paths, flags and compilation target of gcc/g++ are missing from the command line argument list. These needs to be auto-detected and passed to Clang before analysis.
- C/C++ standard versions might be missing from the compilation commands which could change the analysis results (c++11 checker triggers for c++98 code)
 
Our idea is to create a separate tool which could transform a gcc/g++ compilation database file into a compile command json file which can be used by Clang based tools without further modification.
The non-compatible flags are filtered out and the necessary cross compilation include paths and compilation flags are added.
Other Clang based tools could also benefit from the converted compilation database json, like clangd.
 
2. Plist to HTML report conversion tool (see the source code at https://github.com/Ericsson/codechecker/tree/master/vendor/plist_to_html)

Currently Clang Static Analyzer can produce HTML output but this should not be done necessarily by the analyzer.
Plist reports can contain much more information and are easier to manage and parse by other tools compared to the html reports.
With our HTML generator tool we can generate static HTML reports which look similar to the report visualization in Xcode or
to the report viewer web UI of CodeChecker. You can move on each event through a bug path and arrows are shown for easier understanding.
 
Let us know what do you think about these tools, how well would they fit near the available tools.
Your feedback is appreciated.
 
Thanks,
Gyorgy Orban
CodeChecker Team

_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev


_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: [Proposal] New tool contributions to clang repository (gcc/g++ compilation database converter, plist to HTML converter)

David Blaikie via cfe-dev
I think you'll have to first run a "make" on the package (vendor/plist-to-html in the CodeChecker repository) so third-party dependencies such as CodeMirror are downloaded and put to the appropriate place.

On Wed, 26 Sep 2018, 19:59 George Karpenkov via cfe-dev, <[hidden email]> wrote:
Hi Gyorgy,

On Sep 25, 2018, at 11:41 AM, György Orbán via cfe-dev <[hidden email]> wrote:
2. Plist to HTML report conversion tool

I’d really love to have that in Clang, and I think that is an amazing tool!
However, I’ve tried to launch it and failed — it seemed to look for files which are not there.
I would be happy to review patches for that — but I think the tool would require quite a lot of work.


1. gcc/g++ compilation database to Clang compilation database conversion tool

One problem is what we want to address is that gcc/g++ compilation commands cannot be forwarded without modification
to the Clang based tools as they were logged or exported by the build system.
 
Post-processing of the compilation command is required:
- might contain compilation flags and build targets which are not compatible with Clang (needs to be filtered out)
- the compilation argument filtering needs to be done in every tool which uses the compilation database json (scan-build, scan-build-py, CodeChecker)
- when gcc is used as a cross compiler, hard-coded include paths, flags and compilation target of gcc/g++ are missing from the command line argument list. These needs to be auto-detected and passed to Clang before analysis.
- C/C++ standard versions might be missing from the compilation commands which could change the analysis results (c++11 checker triggers for c++98 code)
 
Our idea is to create a separate tool which could transform a gcc/g++ compilation database file into a compile command json file which can be used by Clang based tools without further modification.
The non-compatible flags are filtered out and the necessary cross compilation include paths and compilation flags are added.
Other Clang based tools could also benefit from the converted compilation database json, like clangd.
 
2. Plist to HTML report conversion tool (see the source code at https://github.com/Ericsson/codechecker/tree/master/vendor/plist_to_html)

Currently Clang Static Analyzer can produce HTML output but this should not be done necessarily by the analyzer.
Plist reports can contain much more information and are easier to manage and parse by other tools compared to the html reports.
With our HTML generator tool we can generate static HTML reports which look similar to the report visualization in Xcode or
to the report viewer web UI of CodeChecker. You can move on each event through a bug path and arrows are shown for easier understanding.
 
Let us know what do you think about these tools, how well would they fit near the available tools.
Your feedback is appreciated.
 
Thanks,
Gyorgy Orban
CodeChecker Team

_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev

_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev

_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: [Proposal] New tool contributions to clang repository (gcc/g++ compilation database converter, plist to HTML converter)

David Blaikie via cfe-dev

Hi,


There was a problem with the missing severity levels. Usually the converter is used through CodeChecker, where they are available.

It should be fixed now on the master branch.

As Whisperity already mentioned, with make you can build the package and download the javascript dependencies.


$ cd vendor/plist_to_html

$ make clean all

$ ./build/bin/plist-to-html --help


Let us know if you still experience some issues.


Br,

Gyorgy Orban



From: Whisperity <[hidden email]>
Sent: Thursday, September 27, 2018 8:54:49 AM
To: George Karpenkov
Cc: György Orbán; via cfe-dev
Subject: Re: [cfe-dev] [Proposal] New tool contributions to clang repository (gcc/g++ compilation database converter, plist to HTML converter)
 
I think you'll have to first run a "make" on the package (vendor/plist-to-html in the CodeChecker repository) so third-party dependencies such as CodeMirror are downloaded and put to the appropriate place.

On Wed, 26 Sep 2018, 19:59 George Karpenkov via cfe-dev, <[hidden email]> wrote:
Hi Gyorgy,

On Sep 25, 2018, at 11:41 AM, György Orbán via cfe-dev <[hidden email]> wrote:
2. Plist to HTML report conversion tool

I’d really love to have that in Clang, and I think that is an amazing tool!
However, I’ve tried to launch it and failed — it seemed to look for files which are not there.
I would be happy to review patches for that — but I think the tool would require quite a lot of work.


1. gcc/g++ compilation database to Clang compilation database conversion tool

One problem is what we want to address is that gcc/g++ compilation commands cannot be forwarded without modification
to the Clang based tools as they were logged or exported by the build system.
 
Post-processing of the compilation command is required:
- might contain compilation flags and build targets which are not compatible with Clang (needs to be filtered out)
- the compilation argument filtering needs to be done in every tool which uses the compilation database json (scan-build, scan-build-py, CodeChecker)
- when gcc is used as a cross compiler, hard-coded include paths, flags and compilation target of gcc/g++ are missing from the command line argument list. These needs to be auto-detected and passed to Clang before analysis.
- C/C++ standard versions might be missing from the compilation commands which could change the analysis results (c++11 checker triggers for c++98 code)
 
Our idea is to create a separate tool which could transform a gcc/g++ compilation database file into a compile command json file which can be used by Clang based tools without further modification.
The non-compatible flags are filtered out and the necessary cross compilation include paths and compilation flags are added.
Other Clang based tools could also benefit from the converted compilation database json, like clangd.
 
2. Plist to HTML report conversion tool (see the source code at https://github.com/Ericsson/codechecker/tree/master/vendor/plist_to_html)

Currently Clang Static Analyzer can produce HTML output but this should not be done necessarily by the analyzer.
Plist reports can contain much more information and are easier to manage and parse by other tools compared to the html reports.
With our HTML generator tool we can generate static HTML reports which look similar to the report visualization in Xcode or
to the report viewer web UI of CodeChecker. You can move on each event through a bug path and arrows are shown for easier understanding.
 
Let us know what do you think about these tools, how well would they fit near the available tools.
Your feedback is appreciated.
 
Thanks,
Gyorgy Orban
CodeChecker Team

_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev

_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev

_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: [Proposal] New tool contributions to clang repository (gcc/g++ compilation database converter, plist to HTML converter)

David Blaikie via cfe-dev
In reply to this post by David Blaikie via cfe-dev
Hi György!

I initially though the GCC conversion logic could be put directly in Clang, so that users to not have to run a separate tool. I thought perhaps in clangTooling would be a good fit then a lot of tools could reuse it. In my ideal world (as a user!) I would just have to run CMake then Clangd, Clang-tidy, etc would be able to load the compilation database without me having to worry about another tool to run after executing CMake. Otherwise, it would be more difficult to set things up. For example, I (as a user) would have to wrap by build workflow to run cmake plus that conversion tool and point Clangd to that newly generated CDB. Or, Clangd would have to detect that the CDB contains gcc commands, run the external tool (which might or might not be there) to convert the CDB then load it. I’m curious to know what you think about those scenarios.

Regards,
Marc-André


On Sep 25, 2018, at 2:41 PM, György Orbán via cfe-dev <[hidden email]> wrote:

Hi All,
 
There are multiple features in CodeChecker which could be split into smaller tools so the community could benefit from them.

1. gcc/g++ compilation database to Clang compilation database conversion tool
2. Plist to HTML report conversion tool

1. gcc/g++ compilation database to Clang compilation database conversion tool

One problem is what we want to address is that gcc/g++ compilation commands cannot be forwarded without modification
to the Clang based tools as they were logged or exported by the build system.
 
Post-processing of the compilation command is required:
- might contain compilation flags and build targets which are not compatible with Clang (needs to be filtered out)
- the compilation argument filtering needs to be done in every tool which uses the compilation database json (scan-build, scan-build-py, CodeChecker)
- when gcc is used as a cross compiler, hard-coded include paths, flags and compilation target of gcc/g++ are missing from the command line argument list. These needs to be auto-detected and passed to Clang before analysis.
- C/C++ standard versions might be missing from the compilation commands which could change the analysis results (c++11 checker triggers for c++98 code)
 
Our idea is to create a separate tool which could transform a gcc/g++ compilation database file into a compile command json file which can be used by Clang based tools without further modification.
The non-compatible flags are filtered out and the necessary cross compilation include paths and compilation flags are added.
Other Clang based tools could also benefit from the converted compilation database json, like clangd.
 
2. Plist to HTML report conversion tool (see the source code at https://github.com/Ericsson/codechecker/tree/master/vendor/plist_to_html )

Currently Clang Static Analyzer can produce HTML output but this should not be done necessarily by the analyzer.
Plist reports can contain much more information and are easier to manage and parse by other tools compared to the html reports.
With our HTML generator tool we can generate static HTML reports which look similar to the report visualization in Xcode or
to the report viewer web UI of CodeChecker. You can move on each event through a bug path and arrows are shown for easier understanding.
 
Let us know what do you think about these tools, how well would they fit near the available tools.
Your feedback is appreciated.
 
Thanks,
Gyorgy Orban
CodeChecker Team

_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev


_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev