[RFC] Tutorial for Clang Analyzer Plugins

classic Classic list List threaded Threaded
26 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Re: [RFC] Tutorial for Clang Analyzer Plugins

Anna Zaks

On Jan 7, 2013, at 4:12 PM, Anna Zaks <[hidden email]> wrote:


On Jan 7, 2013, at 4:01 PM, Jordan Rose <[hidden email]> wrote:


On Jan 7, 2013, at 14:00 , Anna Zaks <[hidden email]> wrote:


On Jan 7, 2013, at 12:50 PM, Daniel Dunbar <[hidden email]> wrote:

It makes total sense to me to have the internal documentation browsable on the websites somewhere, and this is what we do for LLVM and Clang and all the other projects, so I see no reason for the analyzer to be different. Note that its only "exposed" as much as there is a URL for it. You can decide if and when you want to link to it from the website.

Makes sense.

This isn't exactly true in Sphinx, because the default mode becomes "include" instead of "exclude".

It probably does make sense to make the "Development" part of the analyzer site become equivalent to LLVM and Clang's "docs".

I am not sure I agree that everything/anything under "Development" tab should be in different format than the rest of the site.

We've discussed this more internally. It makes sense for the analyzer manual to be in the same format as the rest of the site. So unless the rest of the analyzer page gets converted to Sphinx, the manual should stay in HTMl format. 

Unfortunately, we do not have time for full Sphinx conversion. But would be glad if someone else would contribute the effort to do it.

Sam, I am going to review the tutorial in more detail soon.

Thanks,
Anna.


This is where the current Checker Developer Manual lives (and thanks for working on that, Sam; I think I agree with Anna that we'd rather have one merged doc for now), and where some of the current internal stuff could go if it were cleaned up a bit more, but I think we'd want to help moderate that transition. (If you think we should move our text-file notes out of docs/ for now, we could do that too.)

Jordan


_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev


_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: [RFC] Tutorial for Clang Analyzer Plugins

Anna Zaks
Sam,

I think that the weakest spot in the analyzer documentation right now is the Checker Development Manual. So it would be great if we could use your content to improve it. We basically want to keep the format as is but extend/fill in the missing sections. The code segments that are be used to present different concepts could come from referenced checker/checkers. 

I've looked at your tutorial in more detail and here are some concerns/comments.

--------------- Bigger issues --

Section on Checker as a Plugin
We've discussed this internally and decided that we should not advertise "the checker as a plugin" capability until we are ready to support it. Currently, we have not tested it on any platforms other than OS X, so we are not sure, for example, if the plugins are going to work on Windows. We also do not know if the compilers shipped on various platforms have their symbols stripped or not. In addition, there is no effort to guarantee a rigid checker API. The compiler and the plugins are expected to use the same headers. Due to this, the plugins will only work with the compiler they are written for. With the second restriction, in most cases, just adding the checker to the source code of the compiler would be a good option. 

Said that, it would be too bad to loose all the content you've added to explain how the plugins could be used. Maybe we could make it an internal document in the docs directory (not exposed on the website) with a disclaimer that it is an experimental feature.

The second issue is that the LockUnlockChecker checker is rather weak. It does not solve a real problem (ex: assumes that all the locking/unlocking happen in one function, which is not realistic); in addition, it does not showcase a lot of topics we would want to explain (ex: symbols). How about using the SimpleStreamChecker checker that we've presented in the talk? It's not too easy and not too complicated. The source code is available in the repository.

I know that these two issues would make less of your content applicable, but I think it would be better for the final product. What do you think?

---------------  Smaller issues (I've only briefly read this, will have more through comments for the future patches.)--

- We should assume that the reader works with TOT, not the latest released clang version. We'll try to do our best to keep the manual up to date.

ProgramStates:
 - Would be good to visualize the ExplodedGraph. 
 - You could mention the checker writer should not depend on the order in which the nodes are being explored. I think the explanation could be easier to follow if you use DFS, but one would have to try to know for sure.
 - "Since both of the states have a value of ``a`` defined" -> "Since both of the states have a value of ``a`` perfectly constrained"

Manually defining classes:
We should not expose it in the Manual. We have a lot to talk about, so the shorter we keep it the better. We could create a separate internal-use doc (in the docs folder) about it and link to it.

"When using the ``ProgramStateRef``, any action that changes the ``ProgramState`` will, instead of changing the original object, generate a modified copy, which is then assigned to the ``ProgramStateRef``."
ProgramStateRef is just a smart pointer to ProgramState. See r149081.
 
"Detailed Rule Logic"
 Unfortunately, does not belong in a manual.

"When we detect an mistake in the analyzed code (in the Analyzer, this is called a *bug*), we need to report it to the Analyzer so that it can be output." 
Wording is off.

When describing BugReport, you are only describing one of the APIs for its creation. You can mention that this is the case. 

"it is standard to transition the current program state to a *sink node*" This is not standard, but depends on the severity of the bug. See the presentation.

Please, use the LLVM coding standard in the examples (for example "if(blah" -> "if (blah)" ).

To summarize, I think the manual would benefit from the content presented in the following sections:
 - The Structure of a Checker
 - Program States - An Example
 - Extending Program States
 - Reporting Bugs
 - Missing info on testing and additional sources of information

Thanks!
Anna.


On Jan 8, 2013, at 3:57 PM, Anna Zaks <[hidden email]> wrote:


On Jan 7, 2013, at 4:12 PM, Anna Zaks <[hidden email]> wrote:


On Jan 7, 2013, at 4:01 PM, Jordan Rose <[hidden email]> wrote:


On Jan 7, 2013, at 14:00 , Anna Zaks <[hidden email]> wrote:


On Jan 7, 2013, at 12:50 PM, Daniel Dunbar <[hidden email]> wrote:

It makes total sense to me to have the internal documentation browsable on the websites somewhere, and this is what we do for LLVM and Clang and all the other projects, so I see no reason for the analyzer to be different. Note that its only "exposed" as much as there is a URL for it. You can decide if and when you want to link to it from the website.

Makes sense.

This isn't exactly true in Sphinx, because the default mode becomes "include" instead of "exclude".

It probably does make sense to make the "Development" part of the analyzer site become equivalent to LLVM and Clang's "docs".

I am not sure I agree that everything/anything under "Development" tab should be in different format than the rest of the site.

We've discussed this more internally. It makes sense for the analyzer manual to be in the same format as the rest of the site. So unless the rest of the analyzer page gets converted to Sphinx, the manual should stay in HTMl format. 

Unfortunately, we do not have time for full Sphinx conversion. But would be glad if someone else would contribute the effort to do it.

Sam, I am going to review the tutorial in more detail soon.

Thanks,
Anna.


This is where the current Checker Developer Manual lives (and thanks for working on that, Sam; I think I agree with Anna that we'd rather have one merged doc for now), and where some of the current internal stuff could go if it were cleaned up a bit more, but I think we'd want to help moderate that transition. (If you think we should move our text-file notes out of docs/ for now, we could do that too.)

Jordan


_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev

_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev


_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: [RFC] Tutorial for Clang Analyzer Plugins

Sam Handler


We've discussed this internally and decided that we should not advertise "the checker as a plugin" capability until we are ready to support it. Currently, we have not tested it on any platforms other than OS X, so we are not sure, for example, if the plugins are going to work on Windows. We also do not know if the compilers shipped on various platforms have their symbols stripped or not. In addition, there is no effort to guarantee a rigid checker API. The compiler and the plugins are expected to use the same headers. Due to this, the plugins will only work with the compiler they are written for. With the second restriction, in most cases, just adding the checker to the source code of the compiler would be a good option. 

Good points. As I recall, the MSVC compiler produces completely stripped binaries by default.
 
Said that, it would be too bad to loose all the content you've added to explain how the plugins could be used. Maybe we could make it an internal document in the docs directory (not exposed on the website) with a disclaimer that it is an experimental feature.

 I would be okay with this. That part should split out rather easily.
 
The second issue is that the LockUnlockChecker checker is rather weak. It does not solve a real problem (ex: assumes that all the locking/unlocking happen in one function, which is not realistic); in addition, it does not showcase a lot of topics we would want to explain (ex: symbols). How about using the SimpleStreamChecker checker that we've presented in the talk? It's not too easy and not too complicated. The source code is available in the repository.

I freely admit that the LockUnlockChecker does not solve a real problem; it was created to be the absolute simplest possible example of something that requires tracking of state.

I think that the SimpleStreamChecker is the simplest possible checker that solves a "real-world" problem. In addition, the concept is simpler to explain, as the functions used are part of a standard API, instead of being contrived solely for the example. And like you said, it shows the use of symbols, which are a big part of the analyzer.

So overall, I agree that the SimpleStreamChecker would be a better example.

- We should assume that the reader works with TOT, not the latest released clang version. We'll try to do our best to keep the manual up to date.

If the examples can be kept up to date, this is fine. My primary reason for given a specific version was that I have encountered older Clang tutorials on the web that will not compile with TOT.
 
Manually defining classes:
We should not expose it in the Manual. We have a lot to talk about, so the shorter we keep it the better. We could create a separate internal-use doc (in the docs folder) about it and link to it.

I had written the early version of that section before the "convenience macros" for state definition were added. Given that the macros should cover the vast majority of use cases, I see no issue just describing those, and offering additional details in a separate document for those who really need/want to know.



(All of your other comments - I will try to address these in future versions).

Thanks for taking the time to read it though.

To summarize, I think the manual would benefit from the content presented in the following sections:
 - The Structure of a Checker
 - Program States - An Example
 - Extending Program States
 - Reporting Bugs
 - Missing info on testing and additional sources of information

I will try to find time to create an extended version of the Checker Developer Manual with additional material from the tutorial. Which format (HTML or RST) would be better for this?




--
===============================================================================
All opinions expressed in posts from this account are entirely my own, and not
those of any present or former employer. Furthermore, I assert that all works
contributed to the Clang project (1) were developed using no equipment,
supplies, facility or trade secrets of any such employer, (2) were developed
entirely on my own time, and (3) do not result from any work performed for any
such employer.


_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: [RFC] Tutorial for Clang Analyzer Plugins

Oliver Schneider-2
On 2013-01-13 18:23, Sam Handler wrote:

>
>
>     We've discussed this internally and decided that we should not
>     advertise "the checker as a plugin" capability until we are ready
>     to support it. Currently, we have not tested it on any platforms
>     other than OS X, so we are not sure, for example, if the plugins
>     are going to work on Windows. We also do not know if the compilers
>     shipped on various platforms have their symbols stripped or not.
>     In addition, there is no effort to guarantee a rigid checker API.
>     The compiler and the plugins are expected to use the same headers.
>     Due to this, the plugins will only work with the compiler they are
>     written for. With the second restriction, in most cases, just
>     adding the checker to the source code of the compiler would be a
>     good option.
>
>
> Good points. As I recall, the MSVC compiler produces completely
> stripped binaries by default.
Indeed. The old format has been retired mostly, although, depending on
the compiler version used, there are ways to get the old behavior back.
However, debug symbols are usually still created in the form of an
external PDB file. Also keep in mind that until recently (until VS 2012)
MS produced different variations of their compiler, in particular for
the DDKs/WDKs. Not just the ones coming with Visual Studio.

// Oliver

_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev

0xD060B14F.asc (24K) Download Attachment
signature.asc (565 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [RFC] Tutorial for Clang Analyzer Plugins

Anna Zaks
In reply to this post by Sam Handler

On Jan 13, 2013, at 10:23 AM, Sam Handler <[hidden email]> wrote:



We've discussed this internally and decided that we should not advertise "the checker as a plugin" capability until we are ready to support it. Currently, we have not tested it on any platforms other than OS X, so we are not sure, for example, if the plugins are going to work on Windows. We also do not know if the compilers shipped on various platforms have their symbols stripped or not. In addition, there is no effort to guarantee a rigid checker API. The compiler and the plugins are expected to use the same headers. Due to this, the plugins will only work with the compiler they are written for. With the second restriction, in most cases, just adding the checker to the source code of the compiler would be a good option. 

Good points. As I recall, the MSVC compiler produces completely stripped binaries by default.
 
Said that, it would be too bad to loose all the content you've added to explain how the plugins could be used. Maybe we could make it an internal document in the docs directory (not exposed on the website) with a disclaimer that it is an experimental feature.

 I would be okay with this. That part should split out rather easily.
 
The second issue is that the LockUnlockChecker checker is rather weak. It does not solve a real problem (ex: assumes that all the locking/unlocking happen in one function, which is not realistic); in addition, it does not showcase a lot of topics we would want to explain (ex: symbols). How about using the SimpleStreamChecker checker that we've presented in the talk? It's not too easy and not too complicated. The source code is available in the repository.

I freely admit that the LockUnlockChecker does not solve a real problem; it was created to be the absolute simplest possible example of something that requires tracking of state.

I think that the SimpleStreamChecker is the simplest possible checker that solves a "real-world" problem. In addition, the concept is simpler to explain, as the functions used are part of a standard API, instead of being contrived solely for the example. And like you said, it shows the use of symbols, which are a big part of the analyzer.

So overall, I agree that the SimpleStreamChecker would be a better example.

- We should assume that the reader works with TOT, not the latest released clang version. We'll try to do our best to keep the manual up to date.

If the examples can be kept up to date, this is fine. My primary reason for given a specific version was that I have encountered older Clang tutorials on the web that will not compile with TOT.

My hope is that we have more resources and dedication to keep the main analyzer page up to date.

 
Manually defining classes:
We should not expose it in the Manual. We have a lot to talk about, so the shorter we keep it the better. We could create a separate internal-use doc (in the docs folder) about it and link to it.

I had written the early version of that section before the "convenience macros" for state definition were added.

I've guessed that. We've added the macros as we were preparing the presentation.

Given that the macros should cover the vast majority of use cases, I see no issue just describing those, and offering additional details in a separate document for those who really need/want to know.



(All of your other comments - I will try to address these in future versions).

Thanks for taking the time to read it though.

To summarize, I think the manual would benefit from the content presented in the following sections:
 - The Structure of a Checker
 - Program States - An Example
 - Extending Program States
 - Reporting Bugs
 - Missing info on testing and additional sources of information

I will try to find time to create an extended version of the Checker Developer Manual with additional material from the tutorial. Which format (HTML or RST) would be better for this?


Thanks!

I prefer the manual be in the same format as the rest of the site. I don't think anyone subscribed to translate the whole site to Sphinx yet, so it might be easier to go with HTML for now. If you can find a way to use RST but ensure that the style would match the rest of the page (should be doable), that would be good too.




--
===============================================================================
All opinions expressed in posts from this account are entirely my own, and not
those of any present or former employer. Furthermore, I assert that all works
contributed to the Clang project (1) were developed using no equipment,
supplies, facility or trade secrets of any such employer, (2) were developed
entirely on my own time, and (3) do not result from any work performed for any
such employer.



_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: [RFC] Tutorial for Clang Analyzer Plugins

DeepakRajHR
In reply to this post by Sam Handler
Your efforts saved a lot of time of mine. Thanks a lot.
12