[analyzer][RFC] Test all checkers on all tests

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

[analyzer][RFC] Test all checkers on all tests

Artem Dergachev via cfe-dev
Hi,

There is an idea for CSA testing. Currently each checker has it own test
file(s) on which it is tested. Each checker has its own development
history, and many of them were tested on code bases highly different
from each other. That's why one checker may crash (i.e. produce
assertion fail or UNREACHABLE) for code base on which it was never tested.

We tried to launch all checkers on all tests, by replacing `-verify'
flags with `-analyzer-checker=<list of all packages>' through the
lit.local.cfg (using config.substitutions.append method). A lot of
crashes are obtained on our version of CSA (which is somehow different
from upstream, however).

Moreover, we even tried Sema/SemaCXX tests and obtained several more
crashes. Test files usually describe some corner cases, so this method
will allow to test the analyzer on them.

I believe that such testing mode can be useful to improve the analyzer
stability. Does anybody have any ideas, would it be appropriate? Should
we add this mode to the default make check, or create a separate target
for it?

--
Best regards,
Ilya Palachev






_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: [analyzer][RFC] Test all checkers on all tests

Artem Dergachev via cfe-dev
Hi!

I do like the idea of having a target for this but I would not make it part of the make check target. I think it would be a measurable overhead for the runtime of the tests and I am also not sure that we can/want to make that target clean. Maybe in some cases, the return on investment would be too low and we simply have other priorities or the side effect.

Regards,
Gábor

On 18 January 2018 at 13:59, Ilya Palachev <[hidden email]> wrote:
Hi,

There is an idea for CSA testing. Currently each checker has it own test file(s) on which it is tested. Each checker has its own development history, and many of them were tested on code bases highly different from each other. That's why one checker may crash (i.e. produce assertion fail or UNREACHABLE) for code base on which it was never tested.

We tried to launch all checkers on all tests, by replacing `-verify' flags with `-analyzer-checker=<list of all packages>' through the lit.local.cfg (using config.substitutions.append method). A lot of crashes are obtained on our version of CSA (which is somehow different from upstream, however).

Moreover, we even tried Sema/SemaCXX tests and obtained several more crashes. Test files usually describe some corner cases, so this method will allow to test the analyzer on them.

I believe that such testing mode can be useful to improve the analyzer stability. Does anybody have any ideas, would it be appropriate? Should we add this mode to the default make check, or create a separate target for it?

--
Best regards,
Ilya Palachev








_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: [analyzer][RFC] Test all checkers on all tests

Artem Dergachev via cfe-dev
I think this is a great idea! I do think we’ll need to make sure not to run alpha checkers here since those often have assertion failures on purpose for some code constructs since they are work in progress.

I agree with Gabor that we should probably have this be a separate target — but we should have bots run it.

Devin

On Jan 18, 2018, at 7:55 AM, Gábor Horváth <[hidden email]> wrote:

Hi!

I do like the idea of having a target for this but I would not make it part of the make check target. I think it would be a measurable overhead for the runtime of the tests and I am also not sure that we can/want to make that target clean. Maybe in some cases, the return on investment would be too low and we simply have other priorities or the side effect.

Regards,
Gábor

On 18 January 2018 at 13:59, Ilya Palachev <[hidden email]> wrote:
Hi,

There is an idea for CSA testing. Currently each checker has it own test file(s) on which it is tested. Each checker has its own development history, and many of them were tested on code bases highly different from each other. That's why one checker may crash (i.e. produce assertion fail or UNREACHABLE) for code base on which it was never tested.

We tried to launch all checkers on all tests, by replacing `-verify' flags with `-analyzer-checker=<list of all packages>' through the lit.local.cfg (using config.substitutions.append method). A lot of crashes are obtained on our version of CSA (which is somehow different from upstream, however).

Moreover, we even tried Sema/SemaCXX tests and obtained several more crashes. Test files usually describe some corner cases, so this method will allow to test the analyzer on them.

I believe that such testing mode can be useful to improve the analyzer stability. Does anybody have any ideas, would it be appropriate? Should we add this mode to the default make check, or create a separate target for it?

--
Best regards,
Ilya Palachev









_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: [analyzer][RFC] Test all checkers on all tests

Artem Dergachev via cfe-dev
In reply to this post by Artem Dergachev via cfe-dev
Tried this. Found a couple of "Check name is not set properly."
assertion failures! Namely:


   $ bin/clang -cc1 -internal-isystem lib/clang/7.0.0/include
-nostdsysteminc -analyze
-analyzer-checker=core,unix,osx,cplusplus,optin,nullability,valist,llvm
-analyzer-checker=unix.cstring.BadSizeArg -analyzer-store=region
-Wno-strncat-size -Wno-strlcpy-strlcat-size -Wno-sizeof-array-argument
-Wno-sizeof-pointer-memaccess -verify
../llvm/tools/clang/test/Analysis/cstring-syntax.c

   $ bin/clang -cc1 -internal-isystem lib/clang/7.0.0/include
-nostdsysteminc -analyze
-analyzer-checker=core,unix,osx,cplusplus,optin,nullability,valist,llvm
-triple i386-apple-darwin10
-analyzer-checker=security.insecureAPI,security.FloatLoopCounter
../llvm/tools/clang/test/Analysis/security-syntax-checks.m -verify


That was it though, not much crashes on the whole Clang's test suite. I
guess those guys who send us crashes from the GCC test suite did a
really good job^^ (eg. https://reviews.llvm.org/D39800). Also i didn't
enable alpha checkers.

I'd love to have this sort of stress test when i'm working with the AST
that i don't fully understand. I guess it's a good idea for a buildbot
(eg. add to
http://green.lab.llvm.org/green/job/StaticAnalyzerBenchmarks/ or make a
new one?).


On 18/01/2018 7:55 AM, Gábor Horváth wrote:

> Hi!
>
> I do like the idea of having a target for this but I would not make it
> part of the make check target. I think it would be a measurable
> overhead for the runtime of the tests and I am also not sure that we
> can/want to make that target clean. Maybe in some cases, the return on
> investment would be too low and we simply have other priorities or the
> side effect.
>
> Regards,
> Gábor
>
> On 18 January 2018 at 13:59, Ilya Palachev <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     Hi,
>
>     There is an idea for CSA testing. Currently each checker has it
>     own test file(s) on which it is tested. Each checker has its own
>     development history, and many of them were tested on code bases
>     highly different from each other. That's why one checker may crash
>     (i.e. produce assertion fail or UNREACHABLE) for code base on
>     which it was never tested.
>
>     We tried to launch all checkers on all tests, by replacing
>     `-verify' flags with `-analyzer-checker=<list of all packages>'
>     through the lit.local.cfg (using config.substitutions.append
>     method). A lot of crashes are obtained on our version of CSA
>     (which is somehow different from upstream, however).
>
>     Moreover, we even tried Sema/SemaCXX tests and obtained several
>     more crashes. Test files usually describe some corner cases, so
>     this method will allow to test the analyzer on them.
>
>     I believe that such testing mode can be useful to improve the
>     analyzer stability. Does anybody have any ideas, would it be
>     appropriate? Should we add this mode to the default make check, or
>     create a separate target for it?
>
>     --
>     Best regards,
>     Ilya Palachev
>
>
>
>
>
>
>

_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev