Positive tests in the clang test suite?

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Positive tests in the clang test suite?

Richard Smith via cfe-dev
Dear Clang hackers,

I'm trying to understand how Clang is tested,
I've looked at lots of .c and .cpp files in ./test/,
and I see a lot of "// expected-warning" and "// expected-error" annotations,
but I don't see any positive tests checking for expected values?

Other compilers such as rustc uses macros such as assert_eq, e.g.:
#[test]
fn test_extend_specialization() {
    let mut a = BinaryHeap::from(vec![-10, 1, 2, 3, 3]);
    let b = BinaryHeap::from(vec![-20, 5, 43]);

    a.extend(b);

    assert_eq!(a.into_sorted_vec(), [-20, -10, 1, 2, 3, 3, 5, 43]);
}

Is there any similar type of positive testing in Clang, or is the
testing mainly focused on checking for warnings and errors, and just
checking valid code compiles, but not actually checking the computed
resulting values?

Thanks for clarifying.

Best regards,

Joel Jacobson
_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: Positive tests in the clang test suite?

Richard Smith via cfe-dev

> On 18 Jan 2018, at 15:46, Joel Jacobson via cfe-dev <[hidden email]> wrote:
>
> Dear Clang hackers,
>
> I'm trying to understand how Clang is tested,
> I've looked at lots of .c and .cpp files in ./test/,
> and I see a lot of "// expected-warning" and "// expected-error" annotations,
> but I don't see any positive tests checking for expected values?

These are the expected values.  You are looking at Sema checks, which check that the semantic analyser is correctly diagnosing errors in programs.  

>
> Other compilers such as rustc uses macros such as assert_eq, e.g.:
> #[test]
> fn test_extend_specialization() {
>    let mut a = BinaryHeap::from(vec![-10, 1, 2, 3, 3]);
>    let b = BinaryHeap::from(vec![-20, 5, 43]);
>
>    a.extend(b);
>
>    assert_eq!(a.into_sorted_vec(), [-20, -10, 1, 2, 3, 3, 5, 43]);
> }

This looks like an executable test (i.e. one where the output of the compiler must be run).  The clang test suite does not contain these, all of the CodeGen tests check that the correct IR is generated, but it’s then up to LLVM to correctly generate code from this.  In LLVM’s test suite, you will find tests that check that the expected assembly is generated from IR and that the expected binary sequences are generated from assembly.

The advantage of a test suite of this structure is that it can run on any platform (there is no requirement, for example, that the platform that is currently running clang can run code using Windows or DWARF exceptions to test both of those).  The tests are all unit tests and so you can quickly (typically in under 30 seconds) run just the relevant part of the test suite while debugging a new feature.

The down side is that they are not integration tests and so don’t check that the code is correct, only that it is the same as last time.  The LLVM Nightly Test suite contains a number of executable tests, but these typically take much longer to run and so are run by the CI infrastructure on a variety of different platforms.

David

_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: Positive tests in the clang test suite?

Richard Smith via cfe-dev
Also Joel,

If you'd like some documentation on testing for LLVM, you can refer to http://llvm.org/docs/TestingGuide.html.

But of course, David's summary gives you an excellent overview.

On Thu, Jan 18, 2018 at 5:48 PM, David Chisnall via cfe-dev <[hidden email]> wrote:

> On 18 Jan 2018, at 15:46, Joel Jacobson via cfe-dev <[hidden email]> wrote:
>
> Dear Clang hackers,
>
> I'm trying to understand how Clang is tested,
> I've looked at lots of .c and .cpp files in ./test/,
> and I see a lot of "// expected-warning" and "// expected-error" annotations,
> but I don't see any positive tests checking for expected values?

These are the expected values.  You are looking at Sema checks, which check that the semantic analyser is correctly diagnosing errors in programs.

>
> Other compilers such as rustc uses macros such as assert_eq, e.g.:
> #[test]
> fn test_extend_specialization() {
>    let mut a = BinaryHeap::from(vec![-10, 1, 2, 3, 3]);
>    let b = BinaryHeap::from(vec![-20, 5, 43]);
>
>    a.extend(b);
>
>    assert_eq!(a.into_sorted_vec(), [-20, -10, 1, 2, 3, 3, 5, 43]);
> }

This looks like an executable test (i.e. one where the output of the compiler must be run).  The clang test suite does not contain these, all of the CodeGen tests check that the correct IR is generated, but it’s then up to LLVM to correctly generate code from this.  In LLVM’s test suite, you will find tests that check that the expected assembly is generated from IR and that the expected binary sequences are generated from assembly.

The advantage of a test suite of this structure is that it can run on any platform (there is no requirement, for example, that the platform that is currently running clang can run code using Windows or DWARF exceptions to test both of those).  The tests are all unit tests and so you can quickly (typically in under 30 seconds) run just the relevant part of the test suite while debugging a new feature.

The down side is that they are not integration tests and so don’t check that the code is correct, only that it is the same as last time.  The LLVM Nightly Test suite contains a number of executable tests, but these typically take much longer to run and so are run by the CI infrastructure on a variety of different platforms.

David

_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev


_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: Positive tests in the clang test suite?

Richard Smith via cfe-dev
In reply to this post by Richard Smith via cfe-dev
On Thu, Jan 18, 2018 at 12:48 PM, David Chisnall
<[hidden email]> wrote:
> The down side is that they are not integration tests and so don’t check that the code is correct, only that it is the same as last time.  The LLVM Nightly Test suite contains a number of executable tests, but these typically take much longer to run and so are run by the CI infrastructure on a variety of different platforms.

I incorrectly assumed "nightly" referred to some git branch, but I see
no mentioning of "nightly" looking at `git branch -a`, so it must be
located somewhere else?

Googling "LLVM Nightly Test" directs me to
http://llvm.org/docs/lnt/tests.html where it says "LLVM Makefile
test-suite (aka LLVM Nightly Test), Note: The Makefile test-suite is
deprecated.".

Where are these tests located? What git repo/branch to checkout and in
what directory can I find the tests?

My apologies for my stupidity. :)
_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: Positive tests in the clang test suite?

Richard Smith via cfe-dev
From http://llvm.org/docs/TestingGuide.html#test-suite-overview, you can follow the link to http://llvm.org/docs/lnt/quickstart.html which will describe how to set everything up to run the LNT (I think LNT stands for LLVM Nightly Tests) tests (the test-suite).

Since you appear to be using git rather than SVN, you can see where to clone what you need at: http://llvm.org/docs/GettingStarted.html#git-mirror
It suggests the following:
git clone https://git.llvm.org/git/test-suite.git/

On Thu, Jan 18, 2018 at 10:37 PM, Joel Jacobson via cfe-dev <[hidden email]> wrote:
On Thu, Jan 18, 2018 at 12:48 PM, David Chisnall
<[hidden email]> wrote:
> The down side is that they are not integration tests and so don’t check that the code is correct, only that it is the same as last time.  The LLVM Nightly Test suite contains a number of executable tests, but these typically take much longer to run and so are run by the CI infrastructure on a variety of different platforms.

I incorrectly assumed "nightly" referred to some git branch, but I see
no mentioning of "nightly" looking at `git branch -a`, so it must be
located somewhere else?

Googling "LLVM Nightly Test" directs me to
http://llvm.org/docs/lnt/tests.html where it says "LLVM Makefile
test-suite (aka LLVM Nightly Test), Note: The Makefile test-suite is
deprecated.".

Where are these tests located? What git repo/branch to checkout and in
what directory can I find the tests?

My apologies for my stupidity. :)
_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev


_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev