RFC: Release qualification criteria

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

RFC: Release qualification criteria

Vassil Vassilev via cfe-dev
Hi,

I'm splitting this discussion off of my RFC for release process
changes.

We currently have no official release qualification criteria.  In
other words, we don't have any blocking tests that need to pass in
order to make a new release.

We do time-based releases, which is not always compatible with having
quality-based criteria for tagging a final release.  So, I think another
way to look at this issue is to talk about what kinds of CI testing we
have for trunk and if there are any additional kinds of
testing (e.g. compile-time performance) that we want to prioritize.

So, for the purposes of this discussion, I see 2 main questions:

1. Should we define some set of blocking tests that need to pass before a release
   can happen?

2. What gaps do we currently have in our CI testing?

_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: [llvm-dev] RFC: Release qualification criteria

Vassil Vassilev via cfe-dev
Hi Tom,

You hit exactly the questions that are also going through my mind. To add to the two questions:
- What is the testing strategy for LLVM (ie. who tests for which bugs when)? 
- What do we expect from the people contributing patches?
- Do we want to also enforce some of these checks/tests on EVERY patch to master or a release branch (ie. enforced pre-merge testing)?

And the meta question:
What's the process to create such an agreement?

Best,
Christian


On Thu, May 21, 2020 at 8:59 PM Tom Stellard via llvm-dev <[hidden email]> wrote:
Hi,

I'm splitting this discussion off of my RFC for release process
changes.

We currently have no official release qualification criteria.  In
other words, we don't have any blocking tests that need to pass in
order to make a new release.

We do time-based releases, which is not always compatible with having
quality-based criteria for tagging a final release.  So, I think another
way to look at this issue is to talk about what kinds of CI testing we
have for trunk and if there are any additional kinds of
testing (e.g. compile-time performance) that we want to prioritize.

So, for the purposes of this discussion, I see 2 main questions:

1. Should we define some set of blocking tests that need to pass before a release
   can happen?

2. What gaps do we currently have in our CI testing?

_______________________________________________
LLVM Developers mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: [Openmp-dev] RFC: Release qualification criteria

Vassil Vassilev via cfe-dev
In reply to this post by Vassil Vassilev via cfe-dev
On Thu, May 21, 2020 at 8:59 PM Tom Stellard via Openmp-dev
<[hidden email]> wrote:

>
> Hi,
>
> I'm splitting this discussion off of my RFC for release process
> changes.
>
> We currently have no official release qualification criteria.  In
> other words, we don't have any blocking tests that need to pass in
> order to make a new release.
>
> We do time-based releases, which is not always compatible with having
> quality-based criteria for tagging a final release.  So, I think another
> way to look at this issue is to talk about what kinds of CI testing we
> have for trunk and if there are any additional kinds of
> testing (e.g. compile-time performance) that we want to prioritize.
>
> So, for the purposes of this discussion, I see 2 main questions:
>
> 1. Should we define some set of blocking tests that need to pass before a release
>    can happen?

I suppose we could have a baseline about clang bootstrap + lit tests
(what test-release.sh does essentially) passes on major platforms.

But the really interesting question for me is really what kind of bugs
we're considering as release blocking. It's the trade-off between
shipping on (or not too much behind schedule) and delaying to
investigate more issues that's tricky..

>
> 2. What gaps do we currently have in our CI testing?

The latest release made it clear we don't track compile time very
well, or at least not well enough to catch the regressions in that
release early enough.

Also I think there's no 32-bit Windows buildbot anymore :/
_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: [Openmp-dev] RFC: Release qualification criteria

Vassil Vassilev via cfe-dev
On 05/25/2020 06:10 AM, Hans Wennborg wrote:

> On Thu, May 21, 2020 at 8:59 PM Tom Stellard via Openmp-dev
> <[hidden email]> wrote:
>>
>> Hi,
>>
>> I'm splitting this discussion off of my RFC for release process
>> changes.
>>
>> We currently have no official release qualification criteria.  In
>> other words, we don't have any blocking tests that need to pass in
>> order to make a new release.
>>
>> We do time-based releases, which is not always compatible with having
>> quality-based criteria for tagging a final release.  So, I think another
>> way to look at this issue is to talk about what kinds of CI testing we
>> have for trunk and if there are any additional kinds of
>> testing (e.g. compile-time performance) that we want to prioritize.
>>
>> So, for the purposes of this discussion, I see 2 main questions:
>>
>> 1. Should we define some set of blocking tests that need to pass before a release
>>    can happen?
>
> I suppose we could have a baseline about clang bootstrap + lit tests
> (what test-release.sh does essentially) passes on major platforms.
>

I think for testing like this that can be easily automated we're almost
better off just adding these as CI tests from trunk rather than having
them gate releases.

-Tom

> But the really interesting question for me is really what kind of bugs
> we're considering as release blocking. It's the trade-off between
> shipping on (or not too much behind schedule) and delaying to
> investigate more issues that's tricky..
>
>>
>> 2. What gaps do we currently have in our CI testing?
>
> The latest release made it clear we don't track compile time very
> well, or at least not well enough to catch the regressions in that
> release early enough.
>
> Also I think there's no 32-bit Windows buildbot anymore :/
>

_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev