Easy way to generate lit tests for clang?

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Easy way to generate lit tests for clang?

Nathan Ridge via cfe-dev
Hi,

I'm attempting to refactor a large test file and was hoping to see if there was an easier way to automatically update this test to check exact IR output. Perhaps something along the lines of `utils/update_llc_test_checks.py` in llvm, where the checks are automatically generated for each RUN to match IR exactly. Does such a tool exist?

Thanks,
Leonard

_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: Easy way to generate lit tests for clang?

Nathan Ridge via cfe-dev
Leonard Chan via cfe-dev <[hidden email]> writes:

> I'm attempting to refactor a large test file and was hoping to see if
> there was an easier way to automatically update this test to check
> exact IR output. Perhaps something along the lines of
> `utils/update_llc_test_checks.py` in llvm, where the checks are
> automatically generated for each RUN to match IR exactly. Does such a
> tool exist?

Personally, I really would prefer we not go further down that road.
Such tests are very brittle and result in a lot of noise in diffs up for
review.  When I see tests like that in the diff I gloss over them
because they are so big.  I have no idea if the person who updated the
test just ran a script to do so or actually understood the differences
and deemed them correct.

Tests should be small and focused, checking only the very specific thing
they are testing.  We should be striving to change existing tests as
little as possible.

Over time, large tests should be broken up into several smaller tests if
possible.

                     -David
_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: Easy way to generate lit tests for clang?

Nathan Ridge via cfe-dev


> -----Original Message-----
> From: cfe-dev [mailto:[hidden email]] On Behalf Of David
> Greene via cfe-dev
> Sent: Friday, June 14, 2019 4:47 PM
> To: Leonard Chan via cfe-dev
> Cc: Leonard Chan
> Subject: Re: [cfe-dev] Easy way to generate lit tests for clang?
>
> Leonard Chan via cfe-dev <[hidden email]> writes:
>
> > I'm attempting to refactor a large test file and was hoping to see if
> > there was an easier way to automatically update this test to check
> > exact IR output. Perhaps something along the lines of
> > `utils/update_llc_test_checks.py` in llvm, where the checks are
> > automatically generated for each RUN to match IR exactly. Does such a
> > tool exist?
>
> Personally, I really would prefer we not go further down that road.
> Such tests are very brittle and result in a lot of noise in diffs up for
> review.  When I see tests like that in the diff I gloss over them
> because they are so big.  I have no idea if the person who updated the
> test just ran a script to do so or actually understood the differences
> and deemed them correct.

Hear hear. That test tactic does nothing more than use the software
under test as its own oracle and lock down its behavior as a reference.
Basically you are asserting the post-patch behavior is correct, which
would rankle even aside from any preference for test-first development.
I understand the motivation, but the net result is that the valuable
parts of the test are obscured by the irrelevant parts.
--paulr

>
> Tests should be small and focused, checking only the very specific thing
> they are testing.  We should be striving to change existing tests as
> little as possible.
>
> Over time, large tests should be broken up into several smaller tests if
> possible.
>
>                      -David
> _______________________________________________
> cfe-dev mailing list
> [hidden email]
> https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev