[RFC] LLVM bug lifecycle BoF - triaging

classic Classic list List threaded Threaded
23 messages Options
12
Reply | Threaded
Open this post in threaded view
|

[RFC] LLVM bug lifecycle BoF - triaging

David Blaikie via cfe-dev
Hi all,

I’d like to share a few thoughts and analysis results on the LLVM bug life cycle, especially the reporting/triaging part.

As one of the few people creating llvm bugzilla accounts when people request an account, I started to have a feel that many reported bugs, especially by first-time reporters, never get any reply or feedback, let alone be acted on.
If people go through the effort of requesting an account, and then reporting the bug, they show motivation to contribute to the project. However, if then they see zero return on their effort spent, even if it’s just a confirmation of the bug indeed being real or an explanation of what they thought to be a bug isn’t actually a bug, I fear as a community we disincentify a large number of potential long-term contributors.

The above was all based on gut feel, so I tried to gather a bit more data to see if my feel was correct or not.
I scraped the bugs in bugzilla and post-processed them a bit. Below is a chart showing, year by year, how long it takes for a reported bug to get any comment from anyone besides to original reporter. If the bug is still open and didn’t have any reaction after half a year the chart categorizes is as an “infinite” response time.

 
It shows that in recent years the chance of never getting a response to a bug report has been increasing.
For some bugs - e.g. an experienced LLVM developer records a not-that-important bug in bugzilla - that may be just fine.
However, I assume that for people reporting a bug for the first time, the majority may look at least for confirmation that what they reported is actually a bug.
The chart shows (blue bars) that about 50% of first-time bug reporters never get any reply.

I also plotted which components get the most reported bugs that don’t get any reaction and remain open:

The percentage at the top of the bars is the percentage of bugs against that component that never get any reaction. The bar height shows the absolute numbers.


I hope that at the “Lifecycle of LLVM bug reports” BoF at the upcoming dev meeting in San Jose (https://llvmdev18.sched.com/event/H2T3, 17th of October, 10.30am), we can discuss what could be done to improve the experience for first-time reporters and also to reduce the number of bug reports that seemingly get ignored completely.
By sending this email, I hope to trigger discussion before the BoF, both by attendees and non-attendees, so that we have a more fruitful outcome.

At first sight, to me, it seems that the following actions would help:
  • Let’s introduce some form of “triaged” state in bugzilla, to represent that a bug report has been accepted as describing a real problem; able to be acted on (e.g. has a suitable reproducer); and not being a duplicate of another bug report. Looking at https://bugzilla.readthedocs.io/en/5.0/using/editing.html#life-cycle-of-a-bug, maybe the best way to achieve this would be for newly raised bugs to by default go to an “UNCONFIRMED” state instead of “NEW”? Moving the status to “NEW” or “CONFIRMED” would indicate the bug has been triaged.
  • Would it help to have one or multiple people per component that volunteer to triage new bugs?
  • With the majority of developers being part of a team working on a product based on LLVM, I would assume that it is in the interest of most that reported bugs at least get evaluated/triaged? What is stopping those developers to find the time to do some triaging? Would a better notification mechanism be useful to notify when new bugs on a specific component come in that you could triage? Maybe per component try to have a few people on the “default CC list”, which seems easy to set up as a bugzilla administrator.
  • Should we get rid of the "new-bugs/new bugs” component if we won’t have people triaging them?
  • Should we have some description of what a reasonable triage of a bug looks like? If we write such a page, we could also use that page to describe what we think should get recorded when closing bugs.

Thanks,

Kristof


_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: [lldb-dev] [RFC] LLVM bug lifecycle BoF - triaging

David Blaikie via cfe-dev
On 10/04/2018 02:55 AM, Kristof Beyls via lldb-dev wrote:

> Hi all,
>
> I’d like to share a few thoughts and analysis results on the LLVM bug life cycle, especially the reporting/triaging part.
>
> As one of the few people creating llvm bugzilla accounts when people request an account, I started to have a feel that many reported bugs, especially by first-time reporters, never get any reply or feedback, let alone be acted on.
> If people go through the effort of requesting an account, and then reporting the bug, they show motivation to contribute to the project. However, if then they see zero return on their effort spent, even if it’s just a confirmation of the bug indeed being real or an explanation of what they thought to be a bug isn’t actually a bug, I fear as a community we disincentify a large number of potential long-term contributors.
>
> The above was all based on gut feel, so I tried to gather a bit more data to see if my feel was correct or not.
> I scraped the bugs in bugzilla and post-processed them a bit. Below is a chart showing, year by year, how long it takes for a reported bug to get any comment from anyone besides to original reporter. If the bug is still open and didn’t have any reaction after half a year the chart categorizes is as an “infinite” response time.
>
>  
> It shows that in recent years the chance of never getting a response to a bug report has been increasing.
> For some bugs - e.g. an experienced LLVM developer records a not-that-important bug in bugzilla - that may be just fine.
> However, I assume that for people reporting a bug for the first time, the majority may look at least for confirmation that what they reported is actually a bug.
> The chart shows (blue bars) that about 50% of first-time bug reporters never get any reply.
>
> I also plotted which components get the most reported bugs that don’t get any reaction and remain open:
>
> The percentage at the top of the bars is the percentage of bugs against that component that never get any reaction. The bar height shows the absolute numbers.
>
>
> I hope that at the “Lifecycle of LLVM bug reports” BoF at the upcoming dev meeting in San Jose (https://llvmdev18.sched.com/event/H2T3, 17th of October, 10.30am), we can discuss what could be done to improve the experience for first-time reporters and also to reduce the number of bug reports that seemingly get ignored completely.
> By sending this email, I hope to trigger discussion before the BoF, both by attendees and non-attendees, so that we have a more fruitful outcome.
>
> At first sight, to me, it seems that the following actions would help:
>
>   * Let’s introduce some form of “triaged” state in bugzilla, to represent that a bug report has been accepted as describing a real problem; able to be acted on (e.g. has a suitable reproducer); and not being a duplicate of another bug report. Looking at https://bugzilla.readthedocs.io/en/5.0/using/editing.html#life-cycle-of-a-bug, maybe the best way to achieve this would be for newly raised bugs to by default go to an “UNCONFIRMED” state instead of “NEW”? Moving the status to “NEW” or “CONFIRMED” would indicate the bug has been triaged.
>   * Would it help to have one or multiple people per component that volunteer to triage new bugs?
>   * With the majority of developers being part of a team working on a product based on LLVM, I would assume that it is in the interest of most that reported bugs at least get evaluated/triaged? What is stopping those developers to find the time to do some triaging? Would a better notification mechanism be useful to notify when new bugs on a specific component come in that you could triage? Maybe per component try to have a few people on the “default CC list”, which seems easy to set up as a bugzilla administrator.

I think adding relevant people to the "default CC list" is a really
good idea.  I tend to get pretty good response rates on bugs
when I add people to the CC list.  Even with old bugs that have
been dormant for a while.

-Tom


>   * Should we get rid of the "new-bugs/new bugs” component if we won’t have people triaging them?
>   * Should we have some description of what a reasonable triage of a bug looks like? If we write such a page, we could also use that page to describe what we think should get recorded when closing bugs.
>
>
> Thanks,
>
> Kristof
>
>
>
> _______________________________________________
> lldb-dev mailing list
> [hidden email]
> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>

_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: [RFC] LLVM bug lifecycle BoF - triaging

David Blaikie via cfe-dev
In reply to this post by David Blaikie via cfe-dev


> On 4 Oct 2018, at 19:55, Kristof Beyls via cfe-dev <[hidden email]> wrote:
>
> Hi all,
>
> I’d like to share a few thoughts and analysis results on the LLVM bug life cycle, especially the reporting/triaging part.
>
> As one of the few people creating llvm bugzilla accounts when people request an account, I started to have a feel that many reported bugs, especially by first-time reporters, never get any reply or feedback, let alone be acted on.
> If people go through the effort of requesting an account, and then reporting the bug, they show motivation to contribute to the project. However, if then they see zero return on their effort spent, even if it’s just a confirmation of the bug indeed being real or an explanation of what they thought to be a bug isn’t actually a bug, I fear as a community we disincentify a large number of potential long-term contributors.
>
> The above was all based on gut feel, so I tried to gather a bit more data to see if my feel was correct or not.
> I scraped the bugs in bugzilla and post-processed them a bit. Below is a chart showing, year by year, how long it takes for a reported bug to get any comment from anyone besides to original reporter. If the bug is still open and didn’t have any reaction after half a year the chart categorizes is as an “infinite” response time.
>
>  <llvmbugs_triage_response_time.png>
> It shows that in recent years the chance of never getting a response to a bug report has been increasing.
> For some bugs - e.g. an experienced LLVM developer records a not-that-important bug in bugzilla - that may be just fine.
> However, I assume that for people reporting a bug for the first time, the majority may look at least for confirmation that what they reported is actually a bug.
> The chart shows (blue bars) that about 50% of first-time bug reporters never get any reply.
>
> I also plotted which components get the most reported bugs that don’t get any reaction and remain open:
> <llvmbugs_component_response_rate.png>
> The percentage at the top of the bars is the percentage of bugs against that component that never get any reaction. The bar height shows the absolute numbers.
>
>
> I hope that at the “Lifecycle of LLVM bug reports” BoF at the upcoming dev meeting in San Jose (https://llvmdev18.sched.com/event/H2T3, 17th of October, 10.30am), we can discuss what could be done to improve the experience for first-time reporters and also to reduce the number of bug reports that seemingly get ignored completely.
> By sending this email, I hope to trigger discussion before the BoF, both by attendees and non-attendees, so that we have a more fruitful outcome.
>
> At first sight, to me, it seems that the following actions would help:
> • Let’s introduce some form of “triaged” state in bugzilla, to represent that a bug report has been accepted as describing a real problem; able to be acted on (e.g. has a suitable reproducer); and not being a duplicate of another bug report. Looking at https://bugzilla.readthedocs.io/en/5.0/using/editing.html#life-cycle-of-a-bug, maybe the best way to achieve this would be for newly raised bugs to by default go to an “UNCONFIRMED” state instead of “NEW”? Moving the status to “NEW” or “CONFIRMED” would indicate the bug has been triaged.
> • Would it help to have one or multiple people per component that volunteer to triage new bugs?
> • With the majority of developers being part of a team working on a product based on LLVM, I would assume that it is in the interest of most that reported bugs at least get evaluated/triaged? What is stopping those developers to find the time to do some triaging? Would a better notification mechanism be useful to notify when new bugs on a specific component come in that you could triage? Maybe per component try to have a few people on the “default CC list”, which seems easy to set up as a bugzilla administrator.

I set something up this way for XRay, and I try to address those as much as I can. However, sometimes reports go to a different component, and I have to look at the bugs which might be mentioning the components I work on.

This would be great if there were willing volunteers to triage all the bugs that come in. I would do this but unfortunately there’s only so much time in a day/week to make progress on other things as well. If people do step up to do this kind of work, you will have my eternal thanks and will do my part to help out by being as responsive as possible.


> • Should we get rid of the "new-bugs/new bugs” component if we won’t have people triaging them?
> • Should we have some description of what a reasonable triage of a bug looks like? If we write such a page, we could also use that page to describe what we think should get recorded when closing bugs.

Codifying the standards for what a good bug report should look like will definitely help. In my experience especially with open source projects, I’ve found that not everyone — even the experienced contributors — know how much detail is helpful in conveying the problem (this taking more time back-and-forth on getting more details). The default for bug reports seem to be “provide the minimum information as possible” with the hopes that these get a reaction.

If we somehow are able to give an expectation of responsiveness to bug reporters, then maybe we’ll get higher quality bug reports too. Instead of just saying “I tried X and it broke, fix it” maybe we’ll get more reports that say “I did steps 1, 2, 3 on my machine configured as X, Y, Z and got these results A, B, C instead of my expectations which are …”.

Bug report templates, I’ve found, have also been really helpful in this regard. If we don’t have those yet, maybe having a good starting template instead of a blank box to write a bug report in might prompt the reporter better. That also makes triage at least more enjoyable because it leaves very little to wonder when you do get a bug report.

>
> Thanks,
>

Thank you for starting this conversation! I look forward to the results of the BoF discussion summarised as well.

Cheers

-- Dean

_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: [RFC] LLVM bug lifecycle BoF - triaging

David Blaikie via cfe-dev
In reply to this post by David Blaikie via cfe-dev
Hi all,

Just a short feedback about my first impression of the llvm bugzilla. I just requested an account for bugzilla and I just did some browsing the bugs. I checked static analyzer related bugs and as I see almost all bugs are assigned, which is a bit strange to me. Also most of the assigned bugs were assigned to two assignees, so I expect that these two people don't actually work on that ~600 bugs.

So I'm a bit confused now what assigning means in llvm bugzilla. In general for me assigning means the bug is "locked", somebody is working on that issue, so I should not pick it up for working on it. Which means that by now almost every static analyzer related bugs are locked in bugzilla, so I need to find a task somewhere else.

In LibreOffice project we also use bugzilla and only assign a bug if the assignee is actually working on that issue or he/she is planning to work on it soon. Also we reset assignee back to "non" after some months of inactivity. Which means that most of the bugs are unassinged so new contributors can pick them up (also can filter for unassigned bugs). If a bug is related to an area which has an "owner" or expert, we add the expert to the "CC" list so he/she get notified.

I did not find any information about that what assigning means in llvm bugzilla, so I don't know whether it have a different meaning what I expected and described above.

Best Regards,
Tamás Zolnai


Kristof Beyls via cfe-dev <[hidden email]> ezt írta (időpont: 2018. okt. 4., Cs, 11:55):
Hi all,

I’d like to share a few thoughts and analysis results on the LLVM bug life cycle, especially the reporting/triaging part.

As one of the few people creating llvm bugzilla accounts when people request an account, I started to have a feel that many reported bugs, especially by first-time reporters, never get any reply or feedback, let alone be acted on.
If people go through the effort of requesting an account, and then reporting the bug, they show motivation to contribute to the project. However, if then they see zero return on their effort spent, even if it’s just a confirmation of the bug indeed being real or an explanation of what they thought to be a bug isn’t actually a bug, I fear as a community we disincentify a large number of potential long-term contributors.

The above was all based on gut feel, so I tried to gather a bit more data to see if my feel was correct or not.
I scraped the bugs in bugzilla and post-processed them a bit. Below is a chart showing, year by year, how long it takes for a reported bug to get any comment from anyone besides to original reporter. If the bug is still open and didn’t have any reaction after half a year the chart categorizes is as an “infinite” response time.

 
It shows that in recent years the chance of never getting a response to a bug report has been increasing.
For some bugs - e.g. an experienced LLVM developer records a not-that-important bug in bugzilla - that may be just fine.
However, I assume that for people reporting a bug for the first time, the majority may look at least for confirmation that what they reported is actually a bug.
The chart shows (blue bars) that about 50% of first-time bug reporters never get any reply.

I also plotted which components get the most reported bugs that don’t get any reaction and remain open:

The percentage at the top of the bars is the percentage of bugs against that component that never get any reaction. The bar height shows the absolute numbers.


I hope that at the “Lifecycle of LLVM bug reports” BoF at the upcoming dev meeting in San Jose (https://llvmdev18.sched.com/event/H2T3, 17th of October, 10.30am), we can discuss what could be done to improve the experience for first-time reporters and also to reduce the number of bug reports that seemingly get ignored completely.
By sending this email, I hope to trigger discussion before the BoF, both by attendees and non-attendees, so that we have a more fruitful outcome.

At first sight, to me, it seems that the following actions would help:
  • Let’s introduce some form of “triaged” state in bugzilla, to represent that a bug report has been accepted as describing a real problem; able to be acted on (e.g. has a suitable reproducer); and not being a duplicate of another bug report. Looking at https://bugzilla.readthedocs.io/en/5.0/using/editing.html#life-cycle-of-a-bug, maybe the best way to achieve this would be for newly raised bugs to by default go to an “UNCONFIRMED” state instead of “NEW”? Moving the status to “NEW” or “CONFIRMED” would indicate the bug has been triaged.
  • Would it help to have one or multiple people per component that volunteer to triage new bugs?
  • With the majority of developers being part of a team working on a product based on LLVM, I would assume that it is in the interest of most that reported bugs at least get evaluated/triaged? What is stopping those developers to find the time to do some triaging? Would a better notification mechanism be useful to notify when new bugs on a specific component come in that you could triage? Maybe per component try to have a few people on the “default CC list”, which seems easy to set up as a bugzilla administrator.
  • Should we get rid of the "new-bugs/new bugs” component if we won’t have people triaging them?
  • Should we have some description of what a reasonable triage of a bug looks like? If we write such a page, we could also use that page to describe what we think should get recorded when closing bugs.

Thanks,

Kristof

_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev

_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: [RFC] LLVM bug lifecycle BoF - triaging

David Blaikie via cfe-dev
In reply to this post by David Blaikie via cfe-dev
Hello,

I am not a frequent poster on the LLVM mailing lists, but I happened to notice this thread and thought I'd weigh in.

Over 2 years ago, I reported this bug: https://bugs.llvm.org/show_bug.cgi?id=29102

We had to add a pretty ugly workaround in Mono to deal with that, and the workaround is still to this day written to apply to *all* Clang versions on ARM64 because we've gotten no response to the bug. This is what we're doing currently: https://github.com/mono/mono/blob/master/mono/utils/atomic.h#L209

I think this looks to be a pretty serious bug that shouldn't have gone unacknowledged for so long. If there had been any kind of response to the bug, I would've even been happy to cook up a patch. But, frankly, without any confirmation that a bug is valid, very few potential contributors are going to put in the time and effort to write and submit a patch and risk that it gets rejected because the issue it's trying to address isn't even considered a bug by the project maintainers.

Don't get me wrong, though - I understand from experience that "triage all the bugs" is much easier said than done, especially in an open source project. I just wanted to back up Kristof's feeling that the project is losing potential contributions with a concrete example of such, for what it's worth.

Regards,
Alex

On Thu, Oct 4, 2018 at 11:55 AM Kristof Beyls via cfe-dev <[hidden email]> wrote:
Hi all,

I’d like to share a few thoughts and analysis results on the LLVM bug life cycle, especially the reporting/triaging part.

As one of the few people creating llvm bugzilla accounts when people request an account, I started to have a feel that many reported bugs, especially by first-time reporters, never get any reply or feedback, let alone be acted on.
If people go through the effort of requesting an account, and then reporting the bug, they show motivation to contribute to the project. However, if then they see zero return on their effort spent, even if it’s just a confirmation of the bug indeed being real or an explanation of what they thought to be a bug isn’t actually a bug, I fear as a community we disincentify a large number of potential long-term contributors.

The above was all based on gut feel, so I tried to gather a bit more data to see if my feel was correct or not.
I scraped the bugs in bugzilla and post-processed them a bit. Below is a chart showing, year by year, how long it takes for a reported bug to get any comment from anyone besides to original reporter. If the bug is still open and didn’t have any reaction after half a year the chart categorizes is as an “infinite” response time.

 
It shows that in recent years the chance of never getting a response to a bug report has been increasing.
For some bugs - e.g. an experienced LLVM developer records a not-that-important bug in bugzilla - that may be just fine.
However, I assume that for people reporting a bug for the first time, the majority may look at least for confirmation that what they reported is actually a bug.
The chart shows (blue bars) that about 50% of first-time bug reporters never get any reply.

I also plotted which components get the most reported bugs that don’t get any reaction and remain open:

The percentage at the top of the bars is the percentage of bugs against that component that never get any reaction. The bar height shows the absolute numbers.


I hope that at the “Lifecycle of LLVM bug reports” BoF at the upcoming dev meeting in San Jose (https://llvmdev18.sched.com/event/H2T3, 17th of October, 10.30am), we can discuss what could be done to improve the experience for first-time reporters and also to reduce the number of bug reports that seemingly get ignored completely.
By sending this email, I hope to trigger discussion before the BoF, both by attendees and non-attendees, so that we have a more fruitful outcome.

At first sight, to me, it seems that the following actions would help:
  • Let’s introduce some form of “triaged” state in bugzilla, to represent that a bug report has been accepted as describing a real problem; able to be acted on (e.g. has a suitable reproducer); and not being a duplicate of another bug report. Looking at https://bugzilla.readthedocs.io/en/5.0/using/editing.html#life-cycle-of-a-bug, maybe the best way to achieve this would be for newly raised bugs to by default go to an “UNCONFIRMED” state instead of “NEW”? Moving the status to “NEW” or “CONFIRMED” would indicate the bug has been triaged.
  • Would it help to have one or multiple people per component that volunteer to triage new bugs?
  • With the majority of developers being part of a team working on a product based on LLVM, I would assume that it is in the interest of most that reported bugs at least get evaluated/triaged? What is stopping those developers to find the time to do some triaging? Would a better notification mechanism be useful to notify when new bugs on a specific component come in that you could triage? Maybe per component try to have a few people on the “default CC list”, which seems easy to set up as a bugzilla administrator.
  • Should we get rid of the "new-bugs/new bugs” component if we won’t have people triaging them?
  • Should we have some description of what a reasonable triage of a bug looks like? If we write such a page, we could also use that page to describe what we think should get recorded when closing bugs.

Thanks,

Kristof

_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev

_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev

llvmbugs_triage_response_time.png (40K) Download Attachment
llvmbugs_component_response_rate.png (61K) Download Attachment
llvmbugs_component_response_rate.png (61K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [RFC] LLVM bug lifecycle BoF - triaging

David Blaikie via cfe-dev

A couple of additional data points to consider…

 

Approximately 30% of all reported bugs are still open.

The number of open bugs grows by roughly 28 per week. This has been consistent for the past 6 years, when I started tracking it.  I've reported this to the mailing list a few times as an FYI.

So, we do okay—70% of bugs get closed even though we have no defined process—but clearly we can do better.

 

Personally I think anything that raises bug-awareness in the community can only help.  All of the ideas so far have sounded great. Replacing the "new bugs" category with UNCONFIRMED or something like that should be good; making sure that everything at least gets looked at is important. Looking forward to the BoF.

--paulr

 

From: Alex Rønne Petersen [mailto:[hidden email]]
Sent: Saturday, October 06, 2018 5:50 PM
To: [hidden email]
Cc: [hidden email]; [hidden email]; [hidden email]; [hidden email]; [hidden email]; Robinson, Paul
Subject: Re: [cfe-dev] [RFC] LLVM bug lifecycle BoF - triaging

 

Hello,

 

I am not a frequent poster on the LLVM mailing lists, but I happened to notice this thread and thought I'd weigh in.

 

Over 2 years ago, I reported this bug: https://bugs.llvm.org/show_bug.cgi?id=29102

 

We had to add a pretty ugly workaround in Mono to deal with that, and the workaround is still to this day written to apply to *all* Clang versions on ARM64 because we've gotten no response to the bug. This is what we're doing currently: https://github.com/mono/mono/blob/master/mono/utils/atomic.h#L209

 

I think this looks to be a pretty serious bug that shouldn't have gone unacknowledged for so long. If there had been any kind of response to the bug, I would've even been happy to cook up a patch. But, frankly, without any confirmation that a bug is valid, very few potential contributors are going to put in the time and effort to write and submit a patch and risk that it gets rejected because the issue it's trying to address isn't even considered a bug by the project maintainers.

 

Don't get me wrong, though - I understand from experience that "triage all the bugs" is much easier said than done, especially in an open source project. I just wanted to back up Kristof's feeling that the project is losing potential contributions with a concrete example of such, for what it's worth.

 

Regards,

Alex

 

On Thu, Oct 4, 2018 at 11:55 AM Kristof Beyls via cfe-dev <[hidden email]> wrote:

Hi all,


I’d like to share a few thoughts and analysis results on the LLVM bug life cycle, especially the reporting/triaging part.

As one of the few people creating llvm bugzilla accounts when people request an account, I started to have a feel that many reported bugs, especially by first-time reporters, never get any reply or feedback, let alone be acted on.
If people go through the effort of requesting an account, and then reporting the bug, they show motivation to contribute to the project. However, if then they see zero return on their effort spent, even if it’s just a confirmation of the bug indeed being real or an explanation of what they thought to be a bug isn’t actually a bug, I fear as a community we disincentify a large number of potential long-term contributors.

The above was all based on gut feel, so I tried to gather a bit more data to see if my feel was correct or not.
I scraped the bugs in bugzilla and post-processed them a bit. Below is a chart showing, year by year, how long it takes for a reported bug to get any comment from anyone besides to original reporter. If the bug is still open and didn’t have any reaction after half a year the chart categorizes is as an “infinite” response time.

 
It shows that in recent years the chance of never getting a response to a bug report has been increasing.
For some bugs - e.g. an experienced LLVM developer records a not-that-important bug in bugzilla - that may be just fine.
However, I assume that for people reporting a bug for the first time, the majority may look at least for confirmation that what they reported is actually a bug.
The chart shows (blue bars) that about 50% of first-time bug reporters never get any reply.

I also plotted which components get the most reported bugs that don’t get any reaction and remain open:

The percentage at the top of the bars is the percentage of bugs against that component that never get any reaction. The bar height shows the absolute numbers.


I hope that at the “Lifecycle of LLVM bug reports” BoF at the upcoming dev meeting in San Jose (https://llvmdev18.sched.com/event/H2T3, 17th of October, 10.30am), we can discuss what could be done to improve the experience for first-time reporters and also to reduce the number of bug reports that seemingly get ignored completely.
By sending this email, I hope to trigger discussion before the BoF, both by attendees and non-attendees, so that we have a more fruitful outcome.

At first sight, to me, it seems that the following actions would help:

  • Let’s introduce some form of “triaged” state in bugzilla, to represent that a bug report has been accepted as describing a real problem; able to be acted on (e.g. has a suitable reproducer); and not being a duplicate of another bug report. Looking at https://bugzilla.readthedocs.io/en/5.0/using/editing.html#life-cycle-of-a-bug, maybe the best way to achieve this would be for newly raised bugs to by default go to an “UNCONFIRMED” state instead of “NEW”? Moving the status to “NEW” or “CONFIRMED” would indicate the bug has been triaged.
  • Would it help to have one or multiple people per component that volunteer to triage new bugs?
  • With the majority of developers being part of a team working on a product based on LLVM, I would assume that it is in the interest of most that reported bugs at least get evaluated/triaged? What is stopping those developers to find the time to do some triaging? Would a better notification mechanism be useful to notify when new bugs on a specific component come in that you could triage? Maybe per component try to have a few people on the “default CC list”, which seems easy to set up as a bugzilla administrator.
  • Should we get rid of the "new-bugs/new bugs” component if we won’t have people triaging them?
  • Should we have some description of what a reasonable triage of a bug looks like? If we write such a page, we could also use that page to describe what we think should get recorded when closing bugs.


Thanks,

Kristof

 

_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev


_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: [RFC] LLVM bug lifecycle BoF - triaging

David Blaikie via cfe-dev
In reply to this post by David Blaikie via cfe-dev
Hi Tamas,

Thanks for highlighting this - it shows that at least we should have a description of what it means for a bug to be assigned to someone. Your interpretation of a bug being “locked” doesn’t sound unreasonable to me without any other description being available.

For the clang static analyser component, it is configured in bugzilla that when a new bug is raised against it, there is a default assignee. My guess is that most bugs reported against that component just keep on having that default assignee.
I think it’d be an improvement to move default assignees (for the components that have them) to the “default cc” list. That way, the same people still get notified when a bug is raised, but bugs don’t get automatically assigned to them. It’ll take an actual conscious action to assign a bug - so hopefully a bug being assigned to someone can become more meaningful. Or in short, a setup that is somewhat similar to what you describe the LibreOffice project has indeed seems better.

Thanks!

Kristof


On 6 Oct 2018, at 22:53, Tamás Zolnai <[hidden email]> wrote:

Hi all,

Just a short feedback about my first impression of the llvm bugzilla. I just requested an account for bugzilla and I just did some browsing the bugs. I checked static analyzer related bugs and as I see almost all bugs are assigned, which is a bit strange to me. Also most of the assigned bugs were assigned to two assignees, so I expect that these two people don't actually work on that ~600 bugs.

So I'm a bit confused now what assigning means in llvm bugzilla. In general for me assigning means the bug is "locked", somebody is working on that issue, so I should not pick it up for working on it. Which means that by now almost every static analyzer related bugs are locked in bugzilla, so I need to find a task somewhere else.

In LibreOffice project we also use bugzilla and only assign a bug if the assignee is actually working on that issue or he/she is planning to work on it soon. Also we reset assignee back to "non" after some months of inactivity. Which means that most of the bugs are unassinged so new contributors can pick them up (also can filter for unassigned bugs). If a bug is related to an area which has an "owner" or expert, we add the expert to the "CC" list so he/she get notified.

I did not find any information about that what assigning means in llvm bugzilla, so I don't know whether it have a different meaning what I expected and described above.

Best Regards,
Tamás Zolnai


Kristof Beyls via cfe-dev <[hidden email]> ezt írta (időpont: 2018. okt. 4., Cs, 11:55):
Hi all,

I’d like to share a few thoughts and analysis results on the LLVM bug life cycle, especially the reporting/triaging part.

As one of the few people creating llvm bugzilla accounts when people request an account, I started to have a feel that many reported bugs, especially by first-time reporters, never get any reply or feedback, let alone be acted on.
If people go through the effort of requesting an account, and then reporting the bug, they show motivation to contribute to the project. However, if then they see zero return on their effort spent, even if it’s just a confirmation of the bug indeed being real or an explanation of what they thought to be a bug isn’t actually a bug, I fear as a community we disincentify a large number of potential long-term contributors.

The above was all based on gut feel, so I tried to gather a bit more data to see if my feel was correct or not.
I scraped the bugs in bugzilla and post-processed them a bit. Below is a chart showing, year by year, how long it takes for a reported bug to get any comment from anyone besides to original reporter. If the bug is still open and didn’t have any reaction after half a year the chart categorizes is as an “infinite” response time.

 
It shows that in recent years the chance of never getting a response to a bug report has been increasing.
For some bugs - e.g. an experienced LLVM developer records a not-that-important bug in bugzilla - that may be just fine.
However, I assume that for people reporting a bug for the first time, the majority may look at least for confirmation that what they reported is actually a bug.
The chart shows (blue bars) that about 50% of first-time bug reporters never get any reply.

I also plotted which components get the most reported bugs that don’t get any reaction and remain open:

The percentage at the top of the bars is the percentage of bugs against that component that never get any reaction. The bar height shows the absolute numbers.


I hope that at the “Lifecycle of LLVM bug reports” BoF at the upcoming dev meeting in San Jose (https://llvmdev18.sched.com/event/H2T3, 17th of October, 10.30am), we can discuss what could be done to improve the experience for first-time reporters and also to reduce the number of bug reports that seemingly get ignored completely.
By sending this email, I hope to trigger discussion before the BoF, both by attendees and non-attendees, so that we have a more fruitful outcome.

At first sight, to me, it seems that the following actions would help:
  • Let’s introduce some form of “triaged” state in bugzilla, to represent that a bug report has been accepted as describing a real problem; able to be acted on (e.g. has a suitable reproducer); and not being a duplicate of another bug report. Looking at https://bugzilla.readthedocs.io/en/5.0/using/editing.html#life-cycle-of-a-bug, maybe the best way to achieve this would be for newly raised bugs to by default go to an “UNCONFIRMED” state instead of “NEW”? Moving the status to “NEW” or “CONFIRMED” would indicate the bug has been triaged.
  • Would it help to have one or multiple people per component that volunteer to triage new bugs?
  • With the majority of developers being part of a team working on a product based on LLVM, I would assume that it is in the interest of most that reported bugs at least get evaluated/triaged? What is stopping those developers to find the time to do some triaging? Would a better notification mechanism be useful to notify when new bugs on a specific component come in that you could triage? Maybe per component try to have a few people on the “default CC list”, which seems easy to set up as a bugzilla administrator.
  • Should we get rid of the "new-bugs/new bugs” component if we won’t have people triaging them?
  • Should we have some description of what a reasonable triage of a bug looks like? If we write such a page, we could also use that page to describe what we think should get recorded when closing bugs.

Thanks,

Kristof

_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev


_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: [RFC] LLVM bug lifecycle BoF - triaging

David Blaikie via cfe-dev
In reply to this post by David Blaikie via cfe-dev


On 6 Oct 2018, at 23:50, Alex Rønne Petersen <[hidden email]> wrote:

Hello,

I am not a frequent poster on the LLVM mailing lists, but I happened to notice this thread and thought I'd weigh in.

Over 2 years ago, I reported this bug: https://bugs.llvm.org/show_bug.cgi?id=29102

We had to add a pretty ugly workaround in Mono to deal with that, and the workaround is still to this day written to apply to *all* Clang versions on ARM64 because we've gotten no response to the bug. This is what we're doing currently: https://github.com/mono/mono/blob/master/mono/utils/atomic.h#L209

I think this looks to be a pretty serious bug that shouldn't have gone unacknowledged for so long. If there had been any kind of response to the bug, I would've even been happy to cook up a patch. But, frankly, without any confirmation that a bug is valid, very few potential contributors are going to put in the time and effort to write and submit a patch and risk that it gets rejected because the issue it's trying to address isn't even considered a bug by the project maintainers.

Don't get me wrong, though - I understand from experience that "triage all the bugs" is much easier said than done, especially in an open source project. I just wanted to back up Kristof's feeling that the project is losing potential contributions with a concrete example of such, for what it's worth.

Thank you very much, Alex. I assume that most people who reported a bug and never got a reaction on it may not have seen this mail thread. It’s comforting to know that at least some people are in a situation like you describe above. And therefore, improving the bug lifecycle should increase getting contributions from “fresh blood”.


Regards,
Alex

On Thu, Oct 4, 2018 at 11:55 AM Kristof Beyls via cfe-dev <[hidden email]> wrote:
Hi all,

I’d like to share a few thoughts and analysis results on the LLVM bug life cycle, especially the reporting/triaging part.

As one of the few people creating llvm bugzilla accounts when people request an account, I started to have a feel that many reported bugs, especially by first-time reporters, never get any reply or feedback, let alone be acted on.
If people go through the effort of requesting an account, and then reporting the bug, they show motivation to contribute to the project. However, if then they see zero return on their effort spent, even if it’s just a confirmation of the bug indeed being real or an explanation of what they thought to be a bug isn’t actually a bug, I fear as a community we disincentify a large number of potential long-term contributors.

The above was all based on gut feel, so I tried to gather a bit more data to see if my feel was correct or not.
I scraped the bugs in bugzilla and post-processed them a bit. Below is a chart showing, year by year, how long it takes for a reported bug to get any comment from anyone besides to original reporter. If the bug is still open and didn’t have any reaction after half a year the chart categorizes is as an “infinite” response time.

 
It shows that in recent years the chance of never getting a response to a bug report has been increasing.
For some bugs - e.g. an experienced LLVM developer records a not-that-important bug in bugzilla - that may be just fine.
However, I assume that for people reporting a bug for the first time, the majority may look at least for confirmation that what they reported is actually a bug.
The chart shows (blue bars) that about 50% of first-time bug reporters never get any reply.

I also plotted which components get the most reported bugs that don’t get any reaction and remain open:

The percentage at the top of the bars is the percentage of bugs against that component that never get any reaction. The bar height shows the absolute numbers.


I hope that at the “Lifecycle of LLVM bug reports” BoF at the upcoming dev meeting in San Jose (https://llvmdev18.sched.com/event/H2T3, 17th of October, 10.30am), we can discuss what could be done to improve the experience for first-time reporters and also to reduce the number of bug reports that seemingly get ignored completely.
By sending this email, I hope to trigger discussion before the BoF, both by attendees and non-attendees, so that we have a more fruitful outcome.

At first sight, to me, it seems that the following actions would help:
  • Let’s introduce some form of “triaged” state in bugzilla, to represent that a bug report has been accepted as describing a real problem; able to be acted on (e.g. has a suitable reproducer); and not being a duplicate of another bug report. Looking at https://bugzilla.readthedocs.io/en/5.0/using/editing.html#life-cycle-of-a-bug, maybe the best way to achieve this would be for newly raised bugs to by default go to an “UNCONFIRMED” state instead of “NEW”? Moving the status to “NEW” or “CONFIRMED” would indicate the bug has been triaged.
  • Would it help to have one or multiple people per component that volunteer to triage new bugs?
  • With the majority of developers being part of a team working on a product based on LLVM, I would assume that it is in the interest of most that reported bugs at least get evaluated/triaged? What is stopping those developers to find the time to do some triaging? Would a better notification mechanism be useful to notify when new bugs on a specific component come in that you could triage? Maybe per component try to have a few people on the “default CC list”, which seems easy to set up as a bugzilla administrator.
  • Should we get rid of the "new-bugs/new bugs” component if we won’t have people triaging them?
  • Should we have some description of what a reasonable triage of a bug looks like? If we write such a page, we could also use that page to describe what we think should get recorded when closing bugs.

Thanks,

Kristof

_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
<llvmbugs_triage_response_time.png><llvmbugs_component_response_rate.png><llvmbugs_component_response_rate.png>


_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: [llvm-dev] [RFC] LLVM bug lifecycle BoF - triaging

David Blaikie via cfe-dev
In reply to this post by David Blaikie via cfe-dev

Kristof + Paul, thank you for organising this BoF!

A cleanup of the components would be welcome, we have multiple components that get used as dumping grounds (new bugs, clang/new bugs, clang/llvm codegen for instance), just merging those into a single "new bugs"/"unconfirmed" component would at least mean bugs are in the same place.

We have components for things that don't exist any more (old backends ....), and we are missing components for some major parts of the codebase (various tools, should vectorizers be put in a single component? etc.) - it's all scope for bugs getting filed, lost and forgotten.

Embedding more information in the bugzilla main page as well - for instance we barely use keywords (including the beginner tag which still holds promise), adding keyword links (or just the https://bugs.llvm.org/describekeywords.cgi table) to the bottom of the main page could be trivial.

We're also including links to godbolt (sorry, Compiler Explorer) pages a lot more in bugs, there are probably things we can do to make that more integral to the reporting/triage process - I keep wondering if there'd be ways to make that part of web-based reduction and bisection processes.

Finally, finding ways to properly triage the oss-fuzz reports that get sent to [llvm-bugs] - their signal:noise is poor (fuzz tests....) and I have the feeling that most of those go completely unchecked and many 'fixes' are by pure chance.

Simon.

On 10/10/2018 19:17, via llvm-dev wrote:

A couple of additional data points to consider…

 

Approximately 30% of all reported bugs are still open.

The number of open bugs grows by roughly 28 per week. This has been consistent for the past 6 years, when I started tracking it.  I've reported this to the mailing list a few times as an FYI.

So, we do okay—70% of bugs get closed even though we have no defined process—but clearly we can do better.

 

Personally I think anything that raises bug-awareness in the community can only help.  All of the ideas so far have sounded great. Replacing the "new bugs" category with UNCONFIRMED or something like that should be good; making sure that everything at least gets looked at is important. Looking forward to the BoF.

--paulr

 

From: Alex Rønne Petersen [[hidden email]]
Sent: Saturday, October 06, 2018 5:50 PM
To: [hidden email]
Cc: [hidden email]; [hidden email]; [hidden email]; [hidden email]; [hidden email]; Robinson, Paul
Subject: Re: [cfe-dev] [RFC] LLVM bug lifecycle BoF - triaging

 

Hello,

 

I am not a frequent poster on the LLVM mailing lists, but I happened to notice this thread and thought I'd weigh in.

 

Over 2 years ago, I reported this bug: https://bugs.llvm.org/show_bug.cgi?id=29102

 

We had to add a pretty ugly workaround in Mono to deal with that, and the workaround is still to this day written to apply to *all* Clang versions on ARM64 because we've gotten no response to the bug. This is what we're doing currently: https://github.com/mono/mono/blob/master/mono/utils/atomic.h#L209

 

I think this looks to be a pretty serious bug that shouldn't have gone unacknowledged for so long. If there had been any kind of response to the bug, I would've even been happy to cook up a patch. But, frankly, without any confirmation that a bug is valid, very few potential contributors are going to put in the time and effort to write and submit a patch and risk that it gets rejected because the issue it's trying to address isn't even considered a bug by the project maintainers.

 

Don't get me wrong, though - I understand from experience that "triage all the bugs" is much easier said than done, especially in an open source project. I just wanted to back up Kristof's feeling that the project is losing potential contributions with a concrete example of such, for what it's worth.

 

Regards,

Alex

 

On Thu, Oct 4, 2018 at 11:55 AM Kristof Beyls via cfe-dev <[hidden email]> wrote:

Hi all,


I’d like to share a few thoughts and analysis results on the LLVM bug life cycle, especially the reporting/triaging part.

As one of the few people creating llvm bugzilla accounts when people request an account, I started to have a feel that many reported bugs, especially by first-time reporters, never get any reply or feedback, let alone be acted on.
If people go through the effort of requesting an account, and then reporting the bug, they show motivation to contribute to the project. However, if then they see zero return on their effort spent, even if it’s just a confirmation of the bug indeed being real or an explanation of what they thought to be a bug isn’t actually a bug, I fear as a community we disincentify a large number of potential long-term contributors.

The above was all based on gut feel, so I tried to gather a bit more data to see if my feel was correct or not.
I scraped the bugs in bugzilla and post-processed them a bit. Below is a chart showing, year by year, how long it takes for a reported bug to get any comment from anyone besides to original reporter. If the bug is still open and didn’t have any reaction after half a year the chart categorizes is as an “infinite” response time.

 
It shows that in recent years the chance of never getting a response to a bug report has been increasing.
For some bugs - e.g. an experienced LLVM developer records a not-that-important bug in bugzilla - that may be just fine.
However, I assume that for people reporting a bug for the first time, the majority may look at least for confirmation that what they reported is actually a bug.
The chart shows (blue bars) that about 50% of first-time bug reporters never get any reply.

I also plotted which components get the most reported bugs that don’t get any reaction and remain open:

The percentage at the top of the bars is the percentage of bugs against that component that never get any reaction. The bar height shows the absolute numbers.


I hope that at the “Lifecycle of LLVM bug reports” BoF at the upcoming dev meeting in San Jose (https://llvmdev18.sched.com/event/H2T3, 17th of October, 10.30am), we can discuss what could be done to improve the experience for first-time reporters and also to reduce the number of bug reports that seemingly get ignored completely.
By sending this email, I hope to trigger discussion before the BoF, both by attendees and non-attendees, so that we have a more fruitful outcome.

At first sight, to me, it seems that the following actions would help:

  • Let’s introduce some form of “triaged” state in bugzilla, to represent that a bug report has been accepted as describing a real problem; able to be acted on (e.g. has a suitable reproducer); and not being a duplicate of another bug report. Looking at https://bugzilla.readthedocs.io/en/5.0/using/editing.html#life-cycle-of-a-bug, maybe the best way to achieve this would be for newly raised bugs to by default go to an “UNCONFIRMED” state instead of “NEW”? Moving the status to “NEW” or “CONFIRMED” would indicate the bug has been triaged.
  • Would it help to have one or multiple people per component that volunteer to triage new bugs?
  • With the majority of developers being part of a team working on a product based on LLVM, I would assume that it is in the interest of most that reported bugs at least get evaluated/triaged? What is stopping those developers to find the time to do some triaging? Would a better notification mechanism be useful to notify when new bugs on a specific component come in that you could triage? Maybe per component try to have a few people on the “default CC list”, which seems easy to set up as a bugzilla administrator.
  • Should we get rid of the "new-bugs/new bugs” component if we won’t have people triaging them?
  • Should we have some description of what a reasonable triage of a bug looks like? If we write such a page, we could also use that page to describe what we think should get recorded when closing bugs.


Thanks,

Kristof

 

_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev



_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev


_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: [RFC] LLVM bug lifecycle BoF - triaging

David Blaikie via cfe-dev
In reply to this post by David Blaikie via cfe-dev


On 5 Oct 2018, at 07:04, Dean Michael Berris <[hidden email]> wrote:

Thank you for starting this conversation! I look forward to the results of the BoF discussion summarised as well.

Dean, all,

There was a lively discussion at the BoF; we’ve tried to take notes at https://etherpad.net/p/LLVMBugLifeCycleBoF but probably have failed to capture all the points. The slides used to kick start the discussion can be found at https://docs.google.com/presentation/d/1ERB9IQAjSwaNpEnlbchQzd_V9IQlLOojgo9J0_NXvYA/edit

Both at the BoF and in the mail thread, there have been many suggestions for improvements. So many that if we’d want to introduce all of them at once, we’d probably get stuck and not introduce any. To try and make progress on the ones I myself feel are most useful, I’ve volunteered for 2 actions:

1. Write up a proposal for documentation on what to do during bug triaging/closing/etc. I’ve just done so and put it up for review at https://reviews.llvm.org/D53691.
2. Write an email to the mailing lists to ask for volunteers for being on the “default-cc” list for components, implying you’re willing to triage bugs reported against those components. I’ve decided to first try and get consensus on what is expected when triaging a bug (see point above) before actively searching for volunteers for all components. That being said, both at the dev meeting and in the days after, I already received many requests from people to be added to the default-cc list for specific components. Of course, I’m very happy to add people volunteering to default-cc lists, so if you don’t want to wait to get added to a default-cc list, please email [hidden email] or raise it as a ticket in bugs.llvm.org under “Bugzilla Admin”/“Products”.

Furthermore, since the BoF, I’ve seen a quite a few requests to clean up and introduce new components in Bugzilla. We’ve implemented the changes quickly and will aim to continue to have a quick response time in the future. Please file a ticket in bugs.llvm.org under “Bugzilla Admin”/“Products” if you want to request a specific change.

For most of the other points that were raised: I don’t currently plan on acting on them immediately myself and hope to first see an impact of the above actions.

Thanks,

Kristof


_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: [RFC] LLVM bug lifecycle BoF - triaging

David Blaikie via cfe-dev
On Thu, 25 Oct 2018 at 05:10, Kristof Beyls via cfe-dev <[hidden email]> wrote:
On 5 Oct 2018, at 07:04, Dean Michael Berris <[hidden email]> wrote:

Thank you for starting this conversation! I look forward to the results of the BoF discussion summarised as well.

Dean, all,

There was a lively discussion at the BoF; we’ve tried to take notes at https://etherpad.net/p/LLVMBugLifeCycleBoF but probably have failed to capture all the points. The slides used to kick start the discussion can be found at https://docs.google.com/presentation/d/1ERB9IQAjSwaNpEnlbchQzd_V9IQlLOojgo9J0_NXvYA/edit

Both at the BoF and in the mail thread, there have been many suggestions for improvements. So many that if we’d want to introduce all of them at once, we’d probably get stuck and not introduce any. To try and make progress on the ones I myself feel are most useful, I’ve volunteered for 2 actions:

1. Write up a proposal for documentation on what to do during bug triaging/closing/etc. I’ve just done so and put it up for review at https://reviews.llvm.org/D53691.
2. Write an email to the mailing lists to ask for volunteers for being on the “default-cc” list for components, implying you’re willing to triage bugs reported against those components. I’ve decided to first try and get consensus on what is expected when triaging a bug (see point above) before actively searching for volunteers for all components. That being said, both at the dev meeting and in the days after, I already received many requests from people to be added to the default-cc list for specific components. Of course, I’m very happy to add people volunteering to default-cc lists, so if you don’t want to wait to get added to a default-cc list, please email [hidden email] or raise it as a ticket in bugs.llvm.org under “Bugzilla Admin”/“Products”.

Furthermore, since the BoF, I’ve seen a quite a few requests to clean up and introduce new components in Bugzilla. We’ve implemented the changes quickly and will aim to continue to have a quick response time in the future. Please file a ticket in bugs.llvm.org under “Bugzilla Admin”/“Products” if you want to request a specific change.

For most of the other points that were raised: I don’t currently plan on acting on them immediately myself and hope to first see an impact of the above actions.

In the original post, there was a suggestion to bring back the "UNCONFIRMED" status. I think that'd be a great idea, as it both makes it easy to search for untriaged bugs and to give feedback to a reporter that their bug is real and acknowledged. Is that planned?

Also, a big problem with bugzilla as we have it configured today is that commenting on an existing bug often sends mail to literally no-one. Can we reconfigure this so that llvmbugs gets mail for comments on bugs, not just for opening and closing bugs?

_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: [RFC] LLVM bug lifecycle BoF - triaging

David Blaikie via cfe-dev


On 26 Oct 2018, at 04:26, Richard Smith <[hidden email]> wrote:

On Thu, 25 Oct 2018 at 05:10, Kristof Beyls via cfe-dev <[hidden email]> wrote:
On 5 Oct 2018, at 07:04, Dean Michael Berris <[hidden email]> wrote:

Thank you for starting this conversation! I look forward to the results of the BoF discussion summarised as well.

Dean, all,

There was a lively discussion at the BoF; we’ve tried to take notes at https://etherpad.net/p/LLVMBugLifeCycleBoF but probably have failed to capture all the points. The slides used to kick start the discussion can be found at https://docs.google.com/presentation/d/1ERB9IQAjSwaNpEnlbchQzd_V9IQlLOojgo9J0_NXvYA/edit

Both at the BoF and in the mail thread, there have been many suggestions for improvements. So many that if we’d want to introduce all of them at once, we’d probably get stuck and not introduce any. To try and make progress on the ones I myself feel are most useful, I’ve volunteered for 2 actions:

1. Write up a proposal for documentation on what to do during bug triaging/closing/etc. I’ve just done so and put it up for review at https://reviews.llvm.org/D53691.
2. Write an email to the mailing lists to ask for volunteers for being on the “default-cc” list for components, implying you’re willing to triage bugs reported against those components. I’ve decided to first try and get consensus on what is expected when triaging a bug (see point above) before actively searching for volunteers for all components. That being said, both at the dev meeting and in the days after, I already received many requests from people to be added to the default-cc list for specific components. Of course, I’m very happy to add people volunteering to default-cc lists, so if you don’t want to wait to get added to a default-cc list, please email [hidden email] or raise it as a ticket in bugs.llvm.org under “Bugzilla Admin”/“Products”.

Furthermore, since the BoF, I’ve seen a quite a few requests to clean up and introduce new components in Bugzilla. We’ve implemented the changes quickly and will aim to continue to have a quick response time in the future. Please file a ticket in bugs.llvm.org under “Bugzilla Admin”/“Products” if you want to request a specific change.

For most of the other points that were raised: I don’t currently plan on acting on them immediately myself and hope to first see an impact of the above actions.

In the original post, there was a suggestion to bring back the "UNCONFIRMED" status. I think that'd be a great idea, as it both makes it easy to search for untriaged bugs and to give feedback to a reporter that their bug is real and acknowledged. Is that planned?

I hadn’t seen too much feedback on the idea for introducing (reintroducing?) the “UNCONFIRMED” status, so hadn’t planned on making changes to that now.
However, I also think it’s a good idea, so I’ll investigate in more detail now if it wouldn’t be overly hard on the Bugzilla admin side to do so (e.g. I believe I’ll have to give all existing users the rights to be able to confirm bugs). If the “UNCONFIRMED” status can be introduced relatively easily, I now plan to do so, and will adapt the proposed documentation at https://reviews.llvm.org/D53691 accordingly.
Thanks for pointing to this!

Also, a big problem with bugzilla as we have it configured today is that commenting on an existing bug often sends mail to literally no-one. Can we reconfigure this so that llvmbugs gets mail for comments on bugs, not just for opening and closing bugs?

I see you’ve created https://bugs.llvm.org/show_bug.cgi?id=39445 for this in the mean time, and that a bit of discussion has started there - including whether sending an email on all comments wouldn’t result in too much spam on the llvmbugs mailing list. I don’t have a strong opinion either way myself. I hope that making sure that there is at least someone on the default-cc list for every component will result in comments on bugs being emailed to at least one person.

_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: [RFC] LLVM bug lifecycle BoF - triaging

David Blaikie via cfe-dev

As an llvm-bugs subscriber, I would prefer *not* to have email for every comment to every bug.  That's what the CC list is for.

If the admins guarantee that there is at least one auto-cc (who promises to pay attention) for each component, I think that is sufficient.

 

Also +1 for UNCONFIRMED.

--paulr

 

From: Kristof Beyls [mailto:[hidden email]]
Sent: Friday, October 26, 2018 10:26 AM
To: Richard Smith
Cc: [hidden email]; llvm-dev; LLDB Dev; Clang Dev; Tanya Lattner; nd; Robinson, Paul
Subject: Re: [cfe-dev] [RFC] LLVM bug lifecycle BoF - triaging

 

 



On 26 Oct 2018, at 04:26, Richard Smith <[hidden email]> wrote:

 

On Thu, 25 Oct 2018 at 05:10, Kristof Beyls via cfe-dev <[hidden email]> wrote:

On 5 Oct 2018, at 07:04, Dean Michael Berris <[hidden email]> wrote:


Thank you for starting this conversation! I look forward to the results of the BoF discussion summarised as well.

 

Dean, all,

 

There was a lively discussion at the BoF; we’ve tried to take notes at https://etherpad.net/p/LLVMBugLifeCycleBoF but probably have failed to capture all the points. The slides used to kick start the discussion can be found at https://docs.google.com/presentation/d/1ERB9IQAjSwaNpEnlbchQzd_V9IQlLOojgo9J0_NXvYA/edit

 

Both at the BoF and in the mail thread, there have been many suggestions for improvements. So many that if we’d want to introduce all of them at once, we’d probably get stuck and not introduce any. To try and make progress on the ones I myself feel are most useful, I’ve volunteered for 2 actions:

 

1. Write up a proposal for documentation on what to do during bug triaging/closing/etc. I’ve just done so and put it up for review at https://reviews.llvm.org/D53691.

2. Write an email to the mailing lists to ask for volunteers for being on the “default-cc” list for components, implying you’re willing to triage bugs reported against those components. I’ve decided to first try and get consensus on what is expected when triaging a bug (see point above) before actively searching for volunteers for all components. That being said, both at the dev meeting and in the days after, I already received many requests from people to be added to the default-cc list for specific components. Of course, I’m very happy to add people volunteering to default-cc lists, so if you don’t want to wait to get added to a default-cc list, please email [hidden email] or raise it as a ticket in bugs.llvm.org under “Bugzilla Admin”/“Products”.

 

Furthermore, since the BoF, I’ve seen a quite a few requests to clean up and introduce new components in Bugzilla. We’ve implemented the changes quickly and will aim to continue to have a quick response time in the future. Please file a ticket in bugs.llvm.org under “Bugzilla Admin”/“Products” if you want to request a specific change.

 

For most of the other points that were raised: I don’t currently plan on acting on them immediately myself and hope to first see an impact of the above actions.

 

In the original post, there was a suggestion to bring back the "UNCONFIRMED" status. I think that'd be a great idea, as it both makes it easy to search for untriaged bugs and to give feedback to a reporter that their bug is real and acknowledged. Is that planned?

 

I hadn’t seen too much feedback on the idea for introducing (reintroducing?) the “UNCONFIRMED” status, so hadn’t planned on making changes to that now.

However, I also think it’s a good idea, so I’ll investigate in more detail now if it wouldn’t be overly hard on the Bugzilla admin side to do so (e.g. I believe I’ll have to give all existing users the rights to be able to confirm bugs). If the “UNCONFIRMED” status can be introduced relatively easily, I now plan to do so, and will adapt the proposed documentation at https://reviews.llvm.org/D53691 accordingly.

Thanks for pointing to this!



Also, a big problem with bugzilla as we have it configured today is that commenting on an existing bug often sends mail to literally no-one. Can we reconfigure this so that llvmbugs gets mail for comments on bugs, not just for opening and closing bugs?

 

I see you’ve created https://bugs.llvm.org/show_bug.cgi?id=39445 for this in the mean time, and that a bit of discussion has started there - including whether sending an email on all comments wouldn’t result in too much spam on the llvmbugs mailing list. I don’t have a strong opinion either way myself. I hope that making sure that there is at least someone on the default-cc list for every component will result in comments on bugs being emailed to at least one person.


_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: [llvm-dev] [RFC] LLVM bug lifecycle BoF - triaging

David Blaikie via cfe-dev
In reply to this post by David Blaikie via cfe-dev


On 26 Oct 2018, at 16:25, Kristof Beyls via llvm-dev <[hidden email]> wrote:



On 26 Oct 2018, at 04:26, Richard Smith <[hidden email]> wrote:

On Thu, 25 Oct 2018 at 05:10, Kristof Beyls via cfe-dev <[hidden email]> wrote:
On 5 Oct 2018, at 07:04, Dean Michael Berris <[hidden email]> wrote:

Thank you for starting this conversation! I look forward to the results of the BoF discussion summarised as well.

Dean, all,

There was a lively discussion at the BoF; we’ve tried to take notes at https://etherpad.net/p/LLVMBugLifeCycleBoF but probably have failed to capture all the points. The slides used to kick start the discussion can be found at https://docs.google.com/presentation/d/1ERB9IQAjSwaNpEnlbchQzd_V9IQlLOojgo9J0_NXvYA/edit

Both at the BoF and in the mail thread, there have been many suggestions for improvements. So many that if we’d want to introduce all of them at once, we’d probably get stuck and not introduce any. To try and make progress on the ones I myself feel are most useful, I’ve volunteered for 2 actions:

1. Write up a proposal for documentation on what to do during bug triaging/closing/etc. I’ve just done so and put it up for review at https://reviews.llvm.org/D53691.
2. Write an email to the mailing lists to ask for volunteers for being on the “default-cc” list for components, implying you’re willing to triage bugs reported against those components. I’ve decided to first try and get consensus on what is expected when triaging a bug (see point above) before actively searching for volunteers for all components. That being said, both at the dev meeting and in the days after, I already received many requests from people to be added to the default-cc list for specific components. Of course, I’m very happy to add people volunteering to default-cc lists, so if you don’t want to wait to get added to a default-cc list, please email [hidden email] or raise it as a ticket in bugs.llvm.org under “Bugzilla Admin”/“Products”.

Furthermore, since the BoF, I’ve seen a quite a few requests to clean up and introduce new components in Bugzilla. We’ve implemented the changes quickly and will aim to continue to have a quick response time in the future. Please file a ticket in bugs.llvm.org under “Bugzilla Admin”/“Products” if you want to request a specific change.

For most of the other points that were raised: I don’t currently plan on acting on them immediately myself and hope to first see an impact of the above actions.

In the original post, there was a suggestion to bring back the "UNCONFIRMED" status. I think that'd be a great idea, as it both makes it easy to search for untriaged bugs and to give feedback to a reporter that their bug is real and acknowledged. Is that planned?

I hadn’t seen too much feedback on the idea for introducing (reintroducing?) the “UNCONFIRMED” status, so hadn’t planned on making changes to that now.
However, I also think it’s a good idea, so I’ll investigate in more detail now if it wouldn’t be overly hard on the Bugzilla admin side to do so (e.g. I believe I’ll have to give all existing users the rights to be able to confirm bugs). If the “UNCONFIRMED” status can be introduced relatively easily, I now plan to do so, and will adapt the proposed documentation at https://reviews.llvm.org/D53691 accordingly.
Thanks for pointing to this!

Just a status update on investigations to enable the UNCONFIRMED/CONFIRMED states: it seems that bugzilla by-default will put bugs in the “CONFIRMED” state when creating new bugs, if the reporter has “can-confirm” rights. I believe our de facto policy is for everyone with an account to be able to confirm bugs. Also, you need an account to be able to report a bug. The result is that all new bugs by default will go to status “CONFIRMED”, unless the reporter carefully remembers to change the default and select “UNCONFIRMED”. (Also see https://bugzilla.mozilla.org/show_bug.cgi?id=915682 which suggests this behaviour is not configurable).

Unless that can be solved, I fear that many bugs will be reported with the default “CONFIRMED” status even though they aren’t triaged yet. I believe that is worse than the current default “NEW” state.

The only work around for this behaviour where we still get a “to be triaged” state I can think of at the moment is to introduce a custom “CONFIRMED” status, not using bugzilla’s built-in special “UNCONFIRMED” support and have a flow that would allow something like:

NEW -> CONFIRMED -> ASSIGNED -> RESOLVED -> …

The advantage is we’d have separate states to represent “this bug was raised” (NEW) vs “this bug was triaged & confirmed” (CONFIRMED).
The disadvantage is that we’ll start deviating from the default bugzilla workflows.

Overall, I think it’s still a win to implement the above idea. I’ll look into that next week and am all ears for better solutions.

Thanks,

Kristof

_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: [llvm-dev] [RFC] LLVM bug lifecycle BoF - triaging

David Blaikie via cfe-dev


On 26 Oct 2018, at 17:26, Kristof Beyls <[hidden email]> wrote:



On 26 Oct 2018, at 16:25, Kristof Beyls via llvm-dev <[hidden email]> wrote:



On 26 Oct 2018, at 04:26, Richard Smith <[hidden email]> wrote:

On Thu, 25 Oct 2018 at 05:10, Kristof Beyls via cfe-dev <[hidden email]> wrote:
On 5 Oct 2018, at 07:04, Dean Michael Berris <[hidden email]> wrote:

Thank you for starting this conversation! I look forward to the results of the BoF discussion summarised as well.

Dean, all,

There was a lively discussion at the BoF; we’ve tried to take notes at https://etherpad.net/p/LLVMBugLifeCycleBoF but probably have failed to capture all the points. The slides used to kick start the discussion can be found at https://docs.google.com/presentation/d/1ERB9IQAjSwaNpEnlbchQzd_V9IQlLOojgo9J0_NXvYA/edit

Both at the BoF and in the mail thread, there have been many suggestions for improvements. So many that if we’d want to introduce all of them at once, we’d probably get stuck and not introduce any. To try and make progress on the ones I myself feel are most useful, I’ve volunteered for 2 actions:

1. Write up a proposal for documentation on what to do during bug triaging/closing/etc. I’ve just done so and put it up for review at https://reviews.llvm.org/D53691.
2. Write an email to the mailing lists to ask for volunteers for being on the “default-cc” list for components, implying you’re willing to triage bugs reported against those components. I’ve decided to first try and get consensus on what is expected when triaging a bug (see point above) before actively searching for volunteers for all components. That being said, both at the dev meeting and in the days after, I already received many requests from people to be added to the default-cc list for specific components. Of course, I’m very happy to add people volunteering to default-cc lists, so if you don’t want to wait to get added to a default-cc list, please email [hidden email] or raise it as a ticket in bugs.llvm.org under “Bugzilla Admin”/“Products”.

Furthermore, since the BoF, I’ve seen a quite a few requests to clean up and introduce new components in Bugzilla. We’ve implemented the changes quickly and will aim to continue to have a quick response time in the future. Please file a ticket in bugs.llvm.org under “Bugzilla Admin”/“Products” if you want to request a specific change.

For most of the other points that were raised: I don’t currently plan on acting on them immediately myself and hope to first see an impact of the above actions.

In the original post, there was a suggestion to bring back the "UNCONFIRMED" status. I think that'd be a great idea, as it both makes it easy to search for untriaged bugs and to give feedback to a reporter that their bug is real and acknowledged. Is that planned?

I hadn’t seen too much feedback on the idea for introducing (reintroducing?) the “UNCONFIRMED” status, so hadn’t planned on making changes to that now.
However, I also think it’s a good idea, so I’ll investigate in more detail now if it wouldn’t be overly hard on the Bugzilla admin side to do so (e.g. I believe I’ll have to give all existing users the rights to be able to confirm bugs). If the “UNCONFIRMED” status can be introduced relatively easily, I now plan to do so, and will adapt the proposed documentation at https://reviews.llvm.org/D53691 accordingly.
Thanks for pointing to this!

Just a status update on investigations to enable the UNCONFIRMED/CONFIRMED states: it seems that bugzilla by-default will put bugs in the “CONFIRMED” state when creating new bugs, if the reporter has “can-confirm” rights. I believe our de facto policy is for everyone with an account to be able to confirm bugs. Also, you need an account to be able to report a bug. The result is that all new bugs by default will go to status “CONFIRMED”, unless the reporter carefully remembers to change the default and select “UNCONFIRMED”. (Also see https://bugzilla.mozilla.org/show_bug.cgi?id=915682 which suggests this behaviour is not configurable).

Unless that can be solved, I fear that many bugs will be reported with the default “CONFIRMED” status even though they aren’t triaged yet. I believe that is worse than the current default “NEW” state.

The only work around for this behaviour where we still get a “to be triaged” state I can think of at the moment is to introduce a custom “CONFIRMED” status, not using bugzilla’s built-in special “UNCONFIRMED” support and have a flow that would allow something like:

NEW -> CONFIRMED -> ASSIGNED -> RESOLVED -> …

The advantage is we’d have separate states to represent “this bug was raised” (NEW) vs “this bug was triaged & confirmed” (CONFIRMED).
The disadvantage is that we’ll start deviating from the default bugzilla workflows.

Overall, I think it’s still a win to implement the above idea. I’ll look into that next week and am all ears for better solutions.

Thanks,

Kristof

After some further discussion on https://reviews.llvm.org/D53691 and after having looked into bugzilla’s abilities to adapt workflow/statuses, I think we should simplify the workflow to the following:

Start status will remain “NEW”.
After triaging happened, an issue moves to status “CONFIRMED”. This is a new status becoming available.
When an issue is finalised, it moves to status “RESOLVED”.
Status “REOPENED” remains available to reflect that a resolved issue got reopened.

When an issue is actively being worked on, that is indicated by the “Assigned to” field pointing to a real person rather than the “unassigned” pseudo account. 

That means I propose to drop the following currently available statuses:
  • VERIFIED, CLOSED: we didn’t make much/any distinction between “final” statuses RESOLVED, VERIFIED and CLOSED. Therefore, it just simplifies things to only have 1 final state. Bugzilla treats the RESOLVED status special and requires it to always be present. That makes it easy to decide that RESOLVED is the “final” state to keep.
  • ASSIGNED: bugzilla requires the “Assigned to” field to always be filled in. As a result, we have an “unassigned” pseudo account to indicate that an issue is not assigned. Conversely, when the “Assigned to” field contains someone else than this “unassigned” pseudo account, it means the issue is assigned to that specific person. A result of that is that whether an issue is assigned or not is already fully represented in the “Assigned to” field. The ASSIGNED status doesn’t add value on top of this, so let’s just remove it.
    The 64 bugs (see https://bugs.llvm.org/buglist.cgi?bug_status=ASSIGNED&email1=unassigned&emailassigned_to1=1&emailtype1=substring&list_id=148699&query_format=advanced&resolution=---) currently in status ASSIGNED with “Assigned to” pointing to the “unassigned” pseudo account indicates that it has negative value to be able to indicate “assignedness” in two different ways. 
In an attempt to make forward progress: I intend to make the above changes & commit the documentation for bug life cycle at https://reviews.llvm.org/D53691 next week, unless I hear major opposition by then.

Thanks,

Kristof



_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: [llvm-dev] [RFC] LLVM bug lifecycle BoF - triaging

David Blaikie via cfe-dev
On 10/31/2018 06:27 AM, Kristof Beyls wrote:

>
>
>> On 26 Oct 2018, at 17:26, Kristof Beyls <[hidden email] <mailto:[hidden email]>> wrote:
>>
>>
>>
>>> On 26 Oct 2018, at 16:25, Kristof Beyls via llvm-dev <[hidden email] <mailto:[hidden email]>> wrote:
>>>
>>>
>>>
>>>> On 26 Oct 2018, at 04:26, Richard Smith <[hidden email] <mailto:[hidden email]>> wrote:
>>>>
>>>> On Thu, 25 Oct 2018 at 05:10, Kristof Beyls via cfe-dev <[hidden email] <mailto:[hidden email]>> wrote:
>>>>
>>>>>     On 5 Oct 2018, at 07:04, Dean Michael Berris <[hidden email] <mailto:[hidden email]>> wrote:
>>>>>
>>>>>     Thank you for starting this conversation! I look forward to the results of the BoF discussion summarised as well.
>>>>
>>>>     Dean, all,
>>>>
>>>>     There was a lively discussion at the BoF; we’ve tried to take notes at https://etherpad.net/p/LLVMBugLifeCycleBoF but probably have failed to capture all the points. The slides used to kick start the discussion can be found at https://docs.google.com/presentation/d/1ERB9IQAjSwaNpEnlbchQzd_V9IQlLOojgo9J0_NXvYA/edit
>>>>
>>>>     Both at the BoF and in the mail thread, there have been many suggestions for improvements. So many that if we’d want to introduce all of them at once, we’d probably get stuck and not introduce any. To try and make progress on the ones I myself feel are most useful, I’ve volunteered for 2 actions:
>>>>
>>>>     1. Write up a proposal for documentation on what to do during bug triaging/closing/etc. I’ve just done so and put it up for review at https://reviews.llvm.org/D53691.
>>>>     2. Write an email to the mailing lists to ask for volunteers for being on the “default-cc” list for components, implying you’re willing to triage bugs reported against those components. I’ve decided to first try and get consensus on what is expected when triaging a bug (see point above) before actively searching for volunteers for all components. That being said, both at the dev meeting and in the days after, I already received many requests from people to be added to the default-cc list for specific components. Of course, I’m very happy to add people volunteering to default-cc lists, so if you don’t want to wait to get added to a default-cc list, please email [hidden email] <mailto:[hidden email]> or raise it as a ticket in bugs.llvm.org <http://bugs.llvm.org/> under “Bugzilla Admin”/“Products”.
>>>>
>>>>     Furthermore, since the BoF, I’ve seen a quite a few requests to clean up and introduce new components in Bugzilla. We’ve implemented the changes quickly and will aim to continue to have a quick response time in the future. Please file a ticket in bugs.llvm.org <http://bugs.llvm.org/> under “Bugzilla Admin”/“Products” if you want to request a specific change.
>>>>
>>>>     For most of the other points that were raised: I don’t currently plan on acting on them immediately myself and hope to first see an impact of the above actions.
>>>>
>>>>
>>>> In the original post, there was a suggestion to bring back the "UNCONFIRMED" status. I think that'd be a great idea, as it both makes it easy to search for untriaged bugs and to give feedback to a reporter that their bug is real and acknowledged. Is that planned?
>>>
>>> I hadn’t seen too much feedback on the idea for introducing (reintroducing?) the “UNCONFIRMED” status, so hadn’t planned on making changes to that now.
>>> However, I also think it’s a good idea, so I’ll investigate in more detail now if it wouldn’t be overly hard on the Bugzilla admin side to do so (e.g. I believe I’ll have to give all existing users the rights to be able to confirm bugs). If the “UNCONFIRMED” status can be introduced relatively easily, I now plan to do so, and will adapt the proposed documentation at https://reviews.llvm.org/D53691 accordingly.
>>> Thanks for pointing to this!
>>
>> Just a status update on investigations to enable the UNCONFIRMED/CONFIRMED states: it seems that bugzilla by-default will put bugs in the “CONFIRMED” state when creating new bugs, if the reporter has “can-confirm” rights. I believe our de facto policy is for everyone with an account to be able to confirm bugs. Also, you need an account to be able to report a bug. The result is that all new bugs by default will go to status “CONFIRMED”, unless the reporter carefully remembers to change the default and select “UNCONFIRMED”. (Also see https://bugzilla.mozilla.org/show_bug.cgi?id=915682 which suggests this behaviour is not configurable).
>>
>> Unless that can be solved, I fear that many bugs will be reported with the default “CONFIRMED” status even though they aren’t triaged yet. I believe that is worse than the current default “NEW” state.
>>
>> The only work around for this behaviour where we still get a “to be triaged” state I can think of at the moment is to introduce a custom “CONFIRMED” status, not using bugzilla’s built-in special “UNCONFIRMED” support and have a flow that would allow something like:
>>
>> NEW -> CONFIRMED -> ASSIGNED -> RESOLVED -> …
>>
>> The advantage is we’d have separate states to represent “this bug was raised” (NEW) vs “this bug was triaged & confirmed” (CONFIRMED).
>> The disadvantage is that we’ll start deviating from the default bugzilla workflows.
>>
>> Overall, I think it’s still a win to implement the above idea. I’ll look into that next week and am all ears for better solutions.
>>
>> Thanks,
>>
>> Kristof
>
> After some further discussion on https://reviews.llvm.org/D53691 and after having looked into bugzilla’s abilities to adapt workflow/statuses, I think we should simplify the workflow to the following:
>
> Start status will remain “NEW”.
> After triaging happened, an issue moves to status “CONFIRMED”. This is a new status becoming available.
> When an issue is finalised, it moves to status “RESOLVED”.
> Status “REOPENED” remains available to reflect that a resolved issue got reopened.
>
> When an issue is actively being worked on, that is indicated by the “Assigned to” field pointing to a real person rather than the “unassigned” pseudo account.
>
> That means I propose to drop the following currently available statuses:
>
>   * VERIFIED, CLOSED: we didn’t make much/any distinction between “final” statuses RESOLVED, VERIFIED and CLOSED. Therefore, it just simplifies things to only have 1 final state. Bugzilla treats the RESOLVED status special and requires it to always be present. That makes it easy to decide that RESOLVED is the “final” state to keep.
>   * ASSIGNED: bugzilla requires the “Assigned to” field to always be filled in. As a result, we have an “unassigned” pseudo account to indicate that an issue is not assigned. Conversely, when the “Assigned to” field contains someone else than this “unassigned” pseudo account, it means the issue is assigned to that specific person. A result of that is that whether an issue is assigned or not is already fully represented in the “Assigned to” field. The ASSIGNED status doesn’t add value on top of this, so let’s just remove it.
>     The 64 bugs (see https://bugs.llvm.org/buglist.cgi?bug_status=ASSIGNED&email1=unassigned&emailassigned_to1=1&emailtype1=substring&list_id=148699&query_format=advanced&resolution=---) currently in status ASSIGNED with “Assigned to” pointing to the “unassigned” pseudo account indicates that it has negative value to be able to indicate “assignedness” in two different ways.
>
> In an attempt to make forward progress: I intend to make the above changes & commit the documentation for bug life cycle at https://reviews.llvm.org/D53691 next week, unless I hear major opposition by then.
>

This all sounds good to me.  I really like having a more simple process.
Thanks for working on this.

-Tom

> Thanks,
>
> Kristof
>
>

_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: [llvm-dev] [RFC] LLVM bug lifecycle BoF - triaging

David Blaikie via cfe-dev
In reply to this post by David Blaikie via cfe-dev
On Fri, 26 Oct 2018, 08:12 via llvm-dev <[hidden email] wrote:

As an llvm-bugs subscriber, I would prefer *not* to have email for every comment to every bug.  That's what the CC list is for.

If the admins guarantee that there is at least one auto-cc (who promises to pay attention) for each component, I think that is sufficient.

I don't agree. That is the status quo and it doesn't work. Sending email to only one person when a bug is updated is really not much better than sending email to no-one. It's unreasonable to expect a single person to triage all bugs in a component or product.

Auto-cc also makes it impractical to track particular bugs one is actually especially interested in. It's just not a good solution.

Having one bugs list per product, that receives email for all bug updates in that product, would seem like a decent solution. That way the various maintainers for that product can all subscribe and use their normal mail filters to classify the mail.

In fact, I think it'd be entirely reasonable to subscribe cfe-dev to all clang bugs (fully subscribe -- email on all updates!). I don't see any reason whatsoever why a bug update should get *less* attention than non-bug development discussion.

Also +1 for UNCONFIRMED.

--paulr

 

From: Kristof Beyls [mailto:[hidden email]]
Sent: Friday, October 26, 2018 10:26 AM
To: Richard Smith
Cc: [hidden email]; llvm-dev; LLDB Dev; Clang Dev; Tanya Lattner; nd; Robinson, Paul
Subject: Re: [cfe-dev] [RFC] LLVM bug lifecycle BoF - triaging

 

 



On 26 Oct 2018, at 04:26, Richard Smith <[hidden email]> wrote:

 

On Thu, 25 Oct 2018 at 05:10, Kristof Beyls via cfe-dev <[hidden email]> wrote:

On 5 Oct 2018, at 07:04, Dean Michael Berris <[hidden email]> wrote:


Thank you for starting this conversation! I look forward to the results of the BoF discussion summarised as well.

 

Dean, all,

 

There was a lively discussion at the BoF; we’ve tried to take notes at https://etherpad.net/p/LLVMBugLifeCycleBoF but probably have failed to capture all the points. The slides used to kick start the discussion can be found at https://docs.google.com/presentation/d/1ERB9IQAjSwaNpEnlbchQzd_V9IQlLOojgo9J0_NXvYA/edit

 

Both at the BoF and in the mail thread, there have been many suggestions for improvements. So many that if we’d want to introduce all of them at once, we’d probably get stuck and not introduce any. To try and make progress on the ones I myself feel are most useful, I’ve volunteered for 2 actions:

 

1. Write up a proposal for documentation on what to do during bug triaging/closing/etc. I’ve just done so and put it up for review at https://reviews.llvm.org/D53691.

2. Write an email to the mailing lists to ask for volunteers for being on the “default-cc” list for components, implying you’re willing to triage bugs reported against those components. I’ve decided to first try and get consensus on what is expected when triaging a bug (see point above) before actively searching for volunteers for all components. That being said, both at the dev meeting and in the days after, I already received many requests from people to be added to the default-cc list for specific components. Of course, I’m very happy to add people volunteering to default-cc lists, so if you don’t want to wait to get added to a default-cc list, please email [hidden email] or raise it as a ticket in bugs.llvm.org under “Bugzilla Admin”/“Products”.

 

Furthermore, since the BoF, I’ve seen a quite a few requests to clean up and introduce new components in Bugzilla. We’ve implemented the changes quickly and will aim to continue to have a quick response time in the future. Please file a ticket in bugs.llvm.org under “Bugzilla Admin”/“Products” if you want to request a specific change.

 

For most of the other points that were raised: I don’t currently plan on acting on them immediately myself and hope to first see an impact of the above actions.

 

In the original post, there was a suggestion to bring back the "UNCONFIRMED" status. I think that'd be a great idea, as it both makes it easy to search for untriaged bugs and to give feedback to a reporter that their bug is real and acknowledged. Is that planned?

 

I hadn’t seen too much feedback on the idea for introducing (reintroducing?) the “UNCONFIRMED” status, so hadn’t planned on making changes to that now.

However, I also think it’s a good idea, so I’ll investigate in more detail now if it wouldn’t be overly hard on the Bugzilla admin side to do so (e.g. I believe I’ll have to give all existing users the rights to be able to confirm bugs). If the “UNCONFIRMED” status can be introduced relatively easily, I now plan to do so, and will adapt the proposed documentation at https://reviews.llvm.org/D53691 accordingly.

Thanks for pointing to this!



Also, a big problem with bugzilla as we have it configured today is that commenting on an existing bug often sends mail to literally no-one. Can we reconfigure this so that llvmbugs gets mail for comments on bugs, not just for opening and closing bugs?

 

I see you’ve created https://bugs.llvm.org/show_bug.cgi?id=39445 for this in the mean time, and that a bit of discussion has started there - including whether sending an email on all comments wouldn’t result in too much spam on the llvmbugs mailing list. I don’t have a strong opinion either way myself. I hope that making sure that there is at least someone on the default-cc list for every component will result in comments on bugs being emailed to at least one person.

_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: [llvm-dev] [RFC] LLVM bug lifecycle BoF - triaging

David Blaikie via cfe-dev
Richard Smith via cfe-dev <[hidden email]> writes:

> In fact, I think it'd be entirely reasonable to subscribe cfe-dev to
> all clang bugs (fully subscribe -- email on all updates!). I don't see
> any reason whatsoever why a bug update should get *less* attention
> than non-bug development discussion.

Some of us are on space-limited machines (I'm thinking of personal
equipment, not corporate infrastructure) and getting all bug updates for
components could put a real squeeze on things.

I agree that cfe-bugs, for example, should get copied on all updates but
those updates should be opt-in.

                         -David
_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: [llvm-dev] [RFC] LLVM bug lifecycle BoF - triaging

David Blaikie via cfe-dev
On Wed, 31 Oct 2018, 10:47 David Greene via cfe-dev <[hidden email] wrote:
Richard Smith via cfe-dev <[hidden email]> writes:

> In fact, I think it'd be entirely reasonable to subscribe cfe-dev to
> all clang bugs (fully subscribe -- email on all updates!). I don't see
> any reason whatsoever why a bug update should get *less* attention
> than non-bug development discussion.

Some of us are on space-limited machines (I'm thinking of personal
equipment, not corporate infrastructure) and getting all bug updates for
components could put a real squeeze on things.

I agree that cfe-bugs, for example, should get copied on all updates but
those updates should be opt-in.

Assuming we go that way, do you think it's reasonable for someone to want to subscribe to cfe-dev but not cfe-bugs? What's the use case for that? If it's email volume, that choice would prioritize the discussion of "I'm not sure this is a bug" or "what's going on here?" plus general dev discussion and announcements (cfe-dev) over the discussion of "I'm confident that this is a bug" (cfe-bugs).

Perhaps we should have a separate cfe-announce list for people who want to stay informed but not drink from the firehose of development discussion (current cfe-dev plus clang bug updates).

                         -David
_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev

_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: [llvm-dev] [RFC] LLVM bug lifecycle BoF - triaging

David Blaikie via cfe-dev

If the admins guarantee that there is at least one auto-cc (who promises to pay attention) for each component, I think that is sufficient.

I don't agree. That is the status quo and it doesn't work.

 

No, it's not the status quo, because we've only started soliciting auto-cc subscribers in the past week. We don't know how it's working yet.  The status quo is people willing to subscribe to llvm-bugs, and as for myself, I probably care about 1% of the bugs ever filed.  In the bug BoF, very few people present subscribed to llvm-bugs, with at least one non-subscriber proclaiming he didn't want the extra traffic.

 

I agree that cfe-bugs, for example, should get copied on all updates but
those updates should be opt-in.

 

Assuming we go that way, do you think it's reasonable for someone to want to subscribe to cfe-dev but not cfe-bugs? What's the use case for that?

 

The use case is people who are doing their own projects, not working on the Clang front-end itself.  There's clearly a non-trivial percentage of dev subscribers in that category.  I feel a need to keep up with what's happening but it's extremely rarely that I'll ever be moved to try to fix a Clang bug.

--paulr

 


_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
12