making -ftrivial-auto-var-init=zero a first-class option

classic Classic list List threaded Threaded
41 messages Options
123
Reply | Threaded
Open this post in threaded view
|

making -ftrivial-auto-var-init=zero a first-class option

Fangrui Song via cfe-dev
Hi,

tl;dr: I'd like to revisit making -ftrivial-auto-var-init=zero an expressly
supported option. To do this, I think we need to either entirely remove
"-enable-trivial-auto-var-init-zero-knowing-it-will-be-removed-from-clang"
or rename it to something more directly reflecting the issue, like
"-enable-trivial-auto-var-init-zero-knowing-it-forks-the-language".

This is currently open as https://bugs.llvm.org/show_bug.cgi?id=45497

Here is the situation: -ftrivial-auto-var-init=pattern is great for
debugging, but -ftrivial-auto-var-init=zero is needed for production
systems for mainly two reasons, each of which I will try to express context
for:

1) performance and size

As measured by various Google folks across a few projects and in
various places, there's a fairly significant performance impact of
using pattern-init over zero-init. I can let other folks chime in
with their exact numbers, but I can at least share some measurements
Alexander Potapenko made with the Linux kernel (see "Performance costs"):
https://clangbuiltlinux.github.io/CBL-meetup-2020-slides/glider/Fighting_uninitialized_memory_%40_CBL_Meetup_2020.pdf
tl;dr: zero-init tended to be half the cost of pattern-init, though it
varied based on workload, and binary size impact fell over 95% going
from pattern-init to zero-init.

2) security

Another driving factor (see below from various vendors/projects), is the
security stance. Putting non-zero values into most variables types ends
up making them arguably more dangerous than if they were zero-filled.
Most notably, sizes and indexes and less likely to be used out of bounds
if they are zero-initialized. The same holds for bool values that tend
to indicate success instead of failing safe with a false value. While
pointers in the non-canonical range are nice, zero tends to be just
as good. There are certainly exceptions here, but the bulk of the
historical record on how "uninitialized" variables have been used in
real world exploitation involve their being non-zero, and analysis of
those bugs support that conclusion.


Various positions from vendors and projects:

Google (Android, Chrome OS)

Both Android and Chrome OS initially started using pattern-init, but due
to each of: the performance characteristics, the binary size changes, and
the less robust security stance, both projects have recently committed
to switching to zero-init.


Microsoft (Windows)

I'm repeating what Joe Bialek has told me, so he can clarify if I'm not
representing this correctly... While not using Clang/LLVM, Microsoft is
part of the larger C/C++ ecosystem and has implemented both zero-init
(for production builds) and pattern-init (for debug builds) in their
compiler too. They also chose zero-init for production expressly due
to the security benefits.

Some details of their work:
https://github.com/microsoft/MSRC-Security-Research/blob/master/presentations/2019_09_CppCon/CppCon2019%20-%20Killing%20Uninitialized%20Memory.pdf


Upstream Linux kernel

Linus Torvalds has directly stated that he wants zero-init:
"So I'd like the zeroing of local variables to be a native compiler
option..."
"This, btw, is why I also think that the "initialize with poison" is
pointless and wrong."
https://lore.kernel.org/lkml/CAHk-=wgTM+cN7zyUZacGQDv3DuuoA4LORNPWgb1Y_Z1p4iedNQ@.../
Unsurprisingly, I strongly agree. ;)


GrapheneOS is using zero-init (rather than patching Clang as it used to, to get
the same result):
https://twitter.com/DanielMicay/status/1248384468181643272


GCC
There's been mostly silence on the entire topic of automatic variable
initialization, though there have been patches proposed in the past for
zero-init:
https://gcc.gnu.org/legacy-ml/gcc-patches/2014-06/msg00615.html


Apple

I can't speak meaningfully here, but I've heard rumors that they are
depending on zero-init as well. Perhaps someone there can clarify how
they are using these features?



So, while I understand the earlier objections to zero-init from a
"language fork" concern, I think this isn't a position that can really
stand up to the reality of how many projects are using the feature (even
via non-Clang compilers). Given that so much code is going to be built
using zero-init, what's the best way for Clang to adapt here? I would
prefer to just drop the -enable... option entirely, but I think just
renaming it would be fine too.

Thoughts/flames? ;)

--
Kees Cook
_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: making -ftrivial-auto-var-init=zero a first-class option

Fangrui Song via cfe-dev
+a few of the more actively involved folks from the original discussion ( https://lists.llvm.org/pipermail/cfe-dev/2018-November/060172.html )

On Tue, Apr 21, 2020 at 2:20 PM Kees Cook via cfe-dev <[hidden email]> wrote:
Hi,

tl;dr: I'd like to revisit making -ftrivial-auto-var-init=zero an expressly
supported option. To do this, I think we need to either entirely remove
"-enable-trivial-auto-var-init-zero-knowing-it-will-be-removed-from-clang"
or rename it to something more directly reflecting the issue, like
"-enable-trivial-auto-var-init-zero-knowing-it-forks-the-language".

This is currently open as https://bugs.llvm.org/show_bug.cgi?id=45497

Here is the situation: -ftrivial-auto-var-init=pattern is great for
debugging, but -ftrivial-auto-var-init=zero is needed for production
systems for mainly two reasons, each of which I will try to express context
for:

1) performance and size

As measured by various Google folks across a few projects and in
various places, there's a fairly significant performance impact of
using pattern-init over zero-init. I can let other folks chime in
with their exact numbers, but I can at least share some measurements
Alexander Potapenko made with the Linux kernel (see "Performance costs"):
https://clangbuiltlinux.github.io/CBL-meetup-2020-slides/glider/Fighting_uninitialized_memory_%40_CBL_Meetup_2020.pdf
tl;dr: zero-init tended to be half the cost of pattern-init, though it
varied based on workload, and binary size impact fell over 95% going
from pattern-init to zero-init.

2) security

Another driving factor (see below from various vendors/projects), is the
security stance. Putting non-zero values into most variables types ends
up making them arguably more dangerous than if they were zero-filled.
Most notably, sizes and indexes and less likely to be used out of bounds
if they are zero-initialized. The same holds for bool values that tend
to indicate success instead of failing safe with a false value. While
pointers in the non-canonical range are nice, zero tends to be just
as good. There are certainly exceptions here, but the bulk of the
historical record on how "uninitialized" variables have been used in
real world exploitation involve their being non-zero, and analysis of
those bugs support that conclusion.


Various positions from vendors and projects:

Google (Android, Chrome OS)

Both Android and Chrome OS initially started using pattern-init, but due
to each of: the performance characteristics, the binary size changes, and
the less robust security stance, both projects have recently committed
to switching to zero-init.


Microsoft (Windows)

I'm repeating what Joe Bialek has told me, so he can clarify if I'm not
representing this correctly... While not using Clang/LLVM, Microsoft is
part of the larger C/C++ ecosystem and has implemented both zero-init
(for production builds) and pattern-init (for debug builds) in their
compiler too. They also chose zero-init for production expressly due
to the security benefits.

Some details of their work:
https://github.com/microsoft/MSRC-Security-Research/blob/master/presentations/2019_09_CppCon/CppCon2019%20-%20Killing%20Uninitialized%20Memory.pdf


Upstream Linux kernel

Linus Torvalds has directly stated that he wants zero-init:
"So I'd like the zeroing of local variables to be a native compiler
option..."
"This, btw, is why I also think that the "initialize with poison" is
pointless and wrong."
https://lore.kernel.org/lkml/CAHk-=wgTM+cN7zyUZacGQDv3DuuoA4LORNPWgb1Y_Z1p4iedNQ@.../
Unsurprisingly, I strongly agree. ;)


GrapheneOS is using zero-init (rather than patching Clang as it used to, to get
the same result):
https://twitter.com/DanielMicay/status/1248384468181643272


GCC
There's been mostly silence on the entire topic of automatic variable
initialization, though there have been patches proposed in the past for
zero-init:
https://gcc.gnu.org/legacy-ml/gcc-patches/2014-06/msg00615.html


Apple

I can't speak meaningfully here, but I've heard rumors that they are
depending on zero-init as well. Perhaps someone there can clarify how
they are using these features?



So, while I understand the earlier objections to zero-init from a
"language fork" concern, I think this isn't a position that can really
stand up to the reality of how many projects are using the feature (even
via non-Clang compilers). Given that so much code is going to be built
using zero-init, what's the best way for Clang to adapt here? I would
prefer to just drop the -enable... option entirely, but I think just
renaming it would be fine too.

Thoughts/flames? ;)

--
Kees Cook
_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev

_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: making -ftrivial-auto-var-init=zero a first-class option

Fangrui Song via cfe-dev
In reply to this post by Fangrui Song via cfe-dev
What you're proposing is, without question, a language extension. Our policy on language extensions is documented here: http://clang.llvm.org/get_involved.html

Right now, this fails at point 4. We do not want to create or encourage the creation of language dialects and non-portable code, so the place to have this discussion is in the C and C++ committees. Both committees have processes for specifying optional features these days, and they might be amenable to using those processes to standardize the behavior you're asking for. (I mean, maybe not, but our policy requires that you at least try.)

However, there is a variant on what you're proposing that might fare better: instead of guaranteeing zero-initialization, we could guarantee that any observation of an uninitialized variable *either* gives produces zero or results in a trap. That is: it's still undefined to read from uninitialized variables -- we still do not guarantee what will happen if you do, and will warn on uninitialized uses and so on -- but we would bound the damage that can result from such accesses. You would get the security hardening benefits with the modest binary size impact. That approach would not introduce the risk of creating a language dialect (at least, not to the same extent), so our policy on avoiding language extensions would not apply.

On Tue, 21 Apr 2020 at 14:21, Kees Cook via cfe-dev <[hidden email]> wrote:
Hi,

tl;dr: I'd like to revisit making -ftrivial-auto-var-init=zero an expressly
supported option. To do this, I think we need to either entirely remove
"-enable-trivial-auto-var-init-zero-knowing-it-will-be-removed-from-clang"
or rename it to something more directly reflecting the issue, like
"-enable-trivial-auto-var-init-zero-knowing-it-forks-the-language".

This is currently open as https://bugs.llvm.org/show_bug.cgi?id=45497

Here is the situation: -ftrivial-auto-var-init=pattern is great for
debugging, but -ftrivial-auto-var-init=zero is needed for production
systems for mainly two reasons, each of which I will try to express context
for:

1) performance and size

As measured by various Google folks across a few projects and in
various places, there's a fairly significant performance impact of
using pattern-init over zero-init. I can let other folks chime in
with their exact numbers, but I can at least share some measurements
Alexander Potapenko made with the Linux kernel (see "Performance costs"):
https://clangbuiltlinux.github.io/CBL-meetup-2020-slides/glider/Fighting_uninitialized_memory_%40_CBL_Meetup_2020.pdf
tl;dr: zero-init tended to be half the cost of pattern-init, though it
varied based on workload, and binary size impact fell over 95% going
from pattern-init to zero-init.

2) security

Another driving factor (see below from various vendors/projects), is the
security stance. Putting non-zero values into most variables types ends
up making them arguably more dangerous than if they were zero-filled.
Most notably, sizes and indexes and less likely to be used out of bounds
if they are zero-initialized. The same holds for bool values that tend
to indicate success instead of failing safe with a false value. While
pointers in the non-canonical range are nice, zero tends to be just
as good. There are certainly exceptions here, but the bulk of the
historical record on how "uninitialized" variables have been used in
real world exploitation involve their being non-zero, and analysis of
those bugs support that conclusion.


Various positions from vendors and projects:

Google (Android, Chrome OS)

Both Android and Chrome OS initially started using pattern-init, but due
to each of: the performance characteristics, the binary size changes, and
the less robust security stance, both projects have recently committed
to switching to zero-init.


Microsoft (Windows)

I'm repeating what Joe Bialek has told me, so he can clarify if I'm not
representing this correctly... While not using Clang/LLVM, Microsoft is
part of the larger C/C++ ecosystem and has implemented both zero-init
(for production builds) and pattern-init (for debug builds) in their
compiler too. They also chose zero-init for production expressly due
to the security benefits.

Some details of their work:
https://github.com/microsoft/MSRC-Security-Research/blob/master/presentations/2019_09_CppCon/CppCon2019%20-%20Killing%20Uninitialized%20Memory.pdf


Upstream Linux kernel

Linus Torvalds has directly stated that he wants zero-init:
"So I'd like the zeroing of local variables to be a native compiler
option..."
"This, btw, is why I also think that the "initialize with poison" is
pointless and wrong."
https://lore.kernel.org/lkml/CAHk-=wgTM+cN7zyUZacGQDv3DuuoA4LORNPWgb1Y_Z1p4iedNQ@.../
Unsurprisingly, I strongly agree. ;)


GrapheneOS is using zero-init (rather than patching Clang as it used to, to get
the same result):
https://twitter.com/DanielMicay/status/1248384468181643272


GCC
There's been mostly silence on the entire topic of automatic variable
initialization, though there have been patches proposed in the past for
zero-init:
https://gcc.gnu.org/legacy-ml/gcc-patches/2014-06/msg00615.html


Apple

I can't speak meaningfully here, but I've heard rumors that they are
depending on zero-init as well. Perhaps someone there can clarify how
they are using these features?



So, while I understand the earlier objections to zero-init from a
"language fork" concern, I think this isn't a position that can really
stand up to the reality of how many projects are using the feature (even
via non-Clang compilers). Given that so much code is going to be built
using zero-init, what's the best way for Clang to adapt here? I would
prefer to just drop the -enable... option entirely, but I think just
renaming it would be fine too.

Thoughts/flames? ;)

--
Kees Cook
_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev

_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: making -ftrivial-auto-var-init=zero a first-class option

Fangrui Song via cfe-dev
In reply to this post by Fangrui Song via cfe-dev
On Tue, Apr 21, 2020 at 5:20 PM Kees Cook via cfe-dev <[hidden email]> wrote:
Hi,

tl;dr: I'd like to revisit making -ftrivial-auto-var-init=zero an expressly
supported option. To do this, I think we need to either entirely remove
"-enable-trivial-auto-var-init-zero-knowing-it-will-be-removed-from-clang"
or rename it to something more directly reflecting the issue, like
"-enable-trivial-auto-var-init-zero-knowing-it-forks-the-language".

This is currently open as https://bugs.llvm.org/show_bug.cgi?id=45497

Here is the situation: -ftrivial-auto-var-init=pattern is great for
debugging, but -ftrivial-auto-var-init=zero is needed for production
systems for mainly two reasons, each of which I will try to express context
for:

1) performance and size

As measured by various Google folks across a few projects and in
various places, there's a fairly significant performance impact of
using pattern-init over zero-init. I can let other folks chime in
with their exact numbers, but I can at least share some measurements
Alexander Potapenko made with the Linux kernel (see "Performance costs"):
https://clangbuiltlinux.github.io/CBL-meetup-2020-slides/glider/Fighting_uninitialized_memory_%40_CBL_Meetup_2020.pdf
tl;dr: zero-init tended to be half the cost of pattern-init, though it
varied based on workload, and binary size impact fell over 95% going
from pattern-init to zero-init.
This does not seem to indicate why zero-init is preferred over a default of using no explicit policy in production.
 

2) security

Another driving factor (see below from various vendors/projects), is the
security stance. Putting non-zero values into most variables types ends
up making them arguably more dangerous than if they were zero-filled.
Most notably, sizes and indexes and less likely to be used out of bounds
if they are zero-initialized. The same holds for bool values that tend
to indicate success instead of failing safe with a false value. While
pointers in the non-canonical range are nice, zero tends to be just
as good. There are certainly exceptions here, but the bulk of the
historical record on how "uninitialized" variables have been used in
Maybe an explanation of the scare quotes around "uninitialized" would help clarify your position.
 
real world exploitation involve their being non-zero, and analysis of
those bugs support that conclusion.


Various positions from vendors and projects:

Google (Android, Chrome OS)

Both Android and Chrome OS initially started using pattern-init, but due
to each of: the performance characteristics, the binary size changes, and
the less robust security stance, both projects have recently committed
to switching to zero-init.
I'm not sure that this is clear in terms of whether the statements apply to debug/development or production. I don't think pattern-init is meant to be a tool for production builds, which leads me to think that the above statement is about debug builds, at which point I'm thinking that using zero-init only serves to hide problems.
 


Microsoft (Windows)

I'm repeating what Joe Bialek has told me, so he can clarify if I'm not
representing this correctly... While not using Clang/LLVM, Microsoft is
part of the larger C/C++ ecosystem and has implemented both zero-init
(for production builds) and pattern-init (for debug builds) in their
compiler too. They also chose zero-init for production expressly due
to the security benefits.

Some details of their work:
https://github.com/microsoft/MSRC-Security-Research/blob/master/presentations/2019_09_CppCon/CppCon2019%20-%20Killing%20Uninitialized%20Memory.pdf


Upstream Linux kernel

Linus Torvalds has directly stated that he wants zero-init:
"So I'd like the zeroing of local variables to be a native compiler
option..."
"This, btw, is why I also think that the "initialize with poison" is
pointless and wrong."
https://lore.kernel.org/lkml/CAHk-=wgTM+cN7zyUZacGQDv3DuuoA4LORNPWgb1Y_Z1p4iedNQ@.../
Unsurprisingly, I strongly agree. ;)
I don't see why claiming that pattern-init is bad helps make the case for zero-init.
 


GrapheneOS is using zero-init (rather than patching Clang as it used to, to get
the same result):
https://twitter.com/DanielMicay/status/1248384468181643272


GCC
There's been mostly silence on the entire topic of automatic variable
initialization, though there have been patches proposed in the past for
zero-init:
https://gcc.gnu.org/legacy-ml/gcc-patches/2014-06/msg00615.html


Apple

I can't speak meaningfully here, but I've heard rumors that they are
depending on zero-init as well. Perhaps someone there can clarify how
they are using these features?
There's a difference between "depending on zero-init" (as in, the group in question is okay with relying on implicit zeroing on code reviews, etc.) and the use of zero-init as some sort of defence-in-depth approach. Are these rumours clear as to which?
 



So, while I understand the earlier objections to zero-init from a
"language fork" concern, I think this isn't a position that can really
stand up to the reality of how many projects are using the feature (even
via non-Clang compilers). Given that so much code is going to be built
using zero-init, what's the best way for Clang to adapt here?
It happens that there is zero-init and it's at least close enough to what these projects want, but it is actually what they need?
 
I would
prefer to just drop the -enable... option entirely, but I think just
renaming it would be fine too.

Thoughts/flames? ;)

--
Kees Cook
_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev

_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: making -ftrivial-auto-var-init=zero a first-class option

Fangrui Song via cfe-dev
In reply to this post by Fangrui Song via cfe-dev
On Tue, Apr 21, 2020 at 6:12 PM Richard Smith via cfe-dev <[hidden email]> wrote:
What you're proposing is, without question, a language extension. Our policy on language extensions is documented here: http://clang.llvm.org/get_involved.html

Right now, this fails at point 4. We do not want to create or encourage the creation of language dialects and non-portable code, so the place to have this discussion is in the C and C++ committees. Both committees have processes for specifying optional features these days, and they might be amenable to using those processes to standardize the behavior you're asking for. (I mean, maybe not, but our policy requires that you at least try.)

However, there is a variant on what you're proposing that might fare better: instead of guaranteeing zero-initialization, we could guarantee that any observation of an uninitialized variable *either* gives produces zero or results in a trap. That is: it's still undefined to read from uninitialized variables -- we still do not guarantee what will happen if you do, and will warn on uninitialized uses and so on -- but we would bound the damage that can result from such accesses. You would get the security hardening benefits with the modest binary size impact. That approach would not introduce the risk of creating a language dialect (at least, not to the same extent), so our policy on avoiding language extensions would not apply.

I dont understand the point you're making here.  You're saying that if Clang provides reliable behavior, that's a language extension and therefore impossible; but if Clang provides behavior that unpredictably switches between only two possible alternatives (zero or trap), then that's not a language extension anymore and therefore is possible?

I suspect many of the people quoted in the original quote-fest would not be happy with "zero or trap" as the two behaviors.
What if you made it "zero or one"?  That is, whenever you access an uninitialized variable, you are guaranteed to get either all-bits-zero or else all-bits-zero-except-for-the-last-bit-which-is-1?  Would that selection of two behaviors leave matters sufficiently unspecified so as to dodge this "language extension" nonsense?

Alternatively, what about "all-bits-zero or else we explode the physical computer," or "all-bits-zero or else we output a proof of Goldbach's conjecture"?

–Arthur

_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: making -ftrivial-auto-var-init=zero a first-class option

Fangrui Song via cfe-dev
In reply to this post by Fangrui Song via cfe-dev
On Tue, Apr 21, 2020 at 06:29:07PM -0400, Hubert Tong wrote:
> On Tue, Apr 21, 2020 at 5:20 PM Kees Cook via cfe-dev <[hidden email]> wrote:
> > https://clangbuiltlinux.github.io/CBL-meetup-2020-slides/glider/Fighting_uninitialized_memory_%40_CBL_Meetup_2020.pdf
> > tl;dr: zero-init tended to be half the cost of pattern-init, though it
> > varied based on workload, and binary size impact fell over 95% going
> > from pattern-init to zero-init.
> >
> This does not seem to indicate why zero-init is preferred over a default of
> using no explicit policy in production.

Do you mean "leaving things uninitialized" when you say "no explicit
policy"? Maybe I've misunderstood. Google's goal of using auto-init is
to eliminate uninitialized variables in production as a security
defense. When examining zero-init vs pattern-init, there is a clear
advantage on performance and size for zero-init.

> > as good. There are certainly exceptions here, but the bulk of the
> > historical record on how "uninitialized" variables have been used in
> >
> Maybe an explanation of the scare quotes around "uninitialized" would help
> clarify your position.

Ah, sorry, I always use quotes (they are not intended to scare but to
clarify) when discussing uninitialized variables in real-world contexts,
because they are, of course, not uninitialized in the sense of them
not having a value. The RAM contents have a value. Many people without
compiler backgrounds think of such variables as being uncontrollable
or meaningless, when in fact they are usually highly controllable by an
attacker, etc.

> > Google (Android, Chrome OS)
> >
> > Both Android and Chrome OS initially started using pattern-init, but due
> > to each of: the performance characteristics, the binary size changes, and
> > the less robust security stance, both projects have recently committed
> > to switching to zero-init.
> >
> I'm not sure that this is clear in terms of whether the statements apply to
> debug/development or production. I don't think pattern-init is meant to be
> a tool for production builds, which leads me to think that the above
> statement is about debug builds, at which point I'm thinking that using
> zero-init only serves to hide problems.

The context for Google's use of zero-init was meant here to be about
production builds.

> > Upstream Linux kernel
> >
> > Linus Torvalds has directly stated that he wants zero-init:
> > "So I'd like the zeroing of local variables to be a native compiler
> > option..."
> > "This, btw, is why I also think that the "initialize with poison" is
> > pointless and wrong."
> >
> > https://lore.kernel.org/lkml/CAHk-=wgTM+cN7zyUZacGQDv3DuuoA4LORNPWgb1Y_Z1p4iedNQ@.../
> > Unsurprisingly, I strongly agree. ;)
> >
> I don't see why claiming that pattern-init is bad helps make the case for
> zero-init.

Perhaps I did not express it well enough, but both have meaningful and
important uses. My goal here is to illustrate how zero-init is being
used (or preferred) in many real situations, as an argument for why it
should not be hidden behind what some might see as a scary enable flag.

> > Apple
> >
> > I can't speak meaningfully here, but I've heard rumors that they are
> > depending on zero-init as well. Perhaps someone there can clarify how
> > they are using these features?
> >
> There's a difference between "depending on zero-init" (as in, the group in
> question is okay with relying on implicit zeroing on code reviews, etc.)
> and the use of zero-init as some sort of defence-in-depth approach. Are
> these rumours clear as to which?

My understanding was the latter, but I hope to find out for real via
this thread! :) It's not clear to me either.

> > So, while I understand the earlier objections to zero-init from a
> > "language fork" concern, I think this isn't a position that can really
> > stand up to the reality of how many projects are using the feature (even
> > via non-Clang compilers). Given that so much code is going to be built
> > using zero-init, what's the best way for Clang to adapt here?
>
> It happens that there is zero-init and it's at least close enough to what
> these projects want, but it is actually what they need?

Yes, it's expressly what is desired from a security perspective. (And
quite to the relief of that same community, comes with the least
performance impact, which is an unfortunately uncommon scenario in
security flaw mitigations.) I tried to detail that earlier in my email
where it's directly what is indicated as a meaningful defense against
the long history of real-world "uninitialized" variable attacks: setting
everything to zero is the best defense for the entire class of flaws.

--
Kees Cook
_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: [EXTERNAL] Re: making -ftrivial-auto-var-init=zero a first-class option

Fangrui Song via cfe-dev
To add in, we (Microsoft) currently use zero initialization technology in Visual Studio in a large amount of production code we ship to customers (all kernel components, a number of user-mode components). This code is both C and C++.

We already have had multiple vulnerabilities killed because we shipped this technology in production. We received bug reports with repros that worked on older versions of Windows without the mitigation and new versions of Windows that do have it. The new versions don't repro, the old ones do.

Using this sort of technology only in development (and not in production) is not sufficient. Some of these bugs will never produce crashes, the uninitialized data is copied across a trust boundary (i.e. from the kernel in to a untrusted user-mode process). This will never result in a crash but does result in a security issue. This is why shipping in production is a requirement even if you had perfect test coverage that exercises all code paths (which nobody has).

We do enable pattern initialization for debug builds and internal retail builds (using a developer mode in the build environment). We do this to help prevent "forking of the language" and also to force non-determinism. If your code relies on the zero-init, it won't work when we do pattern init. If your code only works with a non-zero value but doesn't care what that value is (Booleans, certain bit tests, etc.), it won't work with zero-init. Developers cannot depend on the automatic initialization for program correctness.

Joe

-----Original Message-----
From: Kees Cook <[hidden email]>
Sent: Tuesday, April 21, 2020 4:20 PM
To: Hubert Tong <[hidden email]>
Cc: Clang Dev <[hidden email]>; Joe Bialek <[hidden email]>
Subject: [EXTERNAL] Re: [cfe-dev] making -ftrivial-auto-var-init=zero a first-class option

On Tue, Apr 21, 2020 at 06:29:07PM -0400, Hubert Tong wrote:

> On Tue, Apr 21, 2020 at 5:20 PM Kees Cook via cfe-dev <[hidden email]> wrote:
> > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcl
> > angbuiltlinux.github.io%2FCBL-meetup-2020-slides%2Fglider%2FFighting
> > _uninitialized_memory_%2540_CBL_Meetup_2020.pdf&amp;data=02%7C01%7Cj
> > obialek%40microsoft.com%7C4bce6c76554b4dcf4c2b08d7e64a7848%7C72f988b
> > f86f141af91ab2d7cd011db47%7C1%7C0%7C637231080235713420&amp;sdata=uMZ
> > PAiQvnjfLxORQNOAdUHGLY8czk8Mlsxc8dXpLSYg%3D&amp;reserved=0
> > tl;dr: zero-init tended to be half the cost of pattern-init, though
> > it varied based on workload, and binary size impact fell over 95%
> > going from pattern-init to zero-init.
> >
> This does not seem to indicate why zero-init is preferred over a
> default of using no explicit policy in production.

Do you mean "leaving things uninitialized" when you say "no explicit policy"? Maybe I've misunderstood. Google's goal of using auto-init is to eliminate uninitialized variables in production as a security defense. When examining zero-init vs pattern-init, there is a clear advantage on performance and size for zero-init.

> > as good. There are certainly exceptions here, but the bulk of the
> > historical record on how "uninitialized" variables have been used in
> >
> Maybe an explanation of the scare quotes around "uninitialized" would
> help clarify your position.

Ah, sorry, I always use quotes (they are not intended to scare but to
clarify) when discussing uninitialized variables in real-world contexts, because they are, of course, not uninitialized in the sense of them not having a value. The RAM contents have a value. Many people without compiler backgrounds think of such variables as being uncontrollable or meaningless, when in fact they are usually highly controllable by an attacker, etc.

> > Google (Android, Chrome OS)
> >
> > Both Android and Chrome OS initially started using pattern-init, but
> > due to each of: the performance characteristics, the binary size
> > changes, and the less robust security stance, both projects have
> > recently committed to switching to zero-init.
> >
> I'm not sure that this is clear in terms of whether the statements
> apply to debug/development or production. I don't think pattern-init
> is meant to be a tool for production builds, which leads me to think
> that the above statement is about debug builds, at which point I'm
> thinking that using zero-init only serves to hide problems.

The context for Google's use of zero-init was meant here to be about production builds.

> > Upstream Linux kernel
> >
> > Linus Torvalds has directly stated that he wants zero-init:
> > "So I'd like the zeroing of local variables to be a native compiler
> > option..."
> > "This, btw, is why I also think that the "initialize with poison" is
> > pointless and wrong."
> >
> > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flo
> > re.kernel.org%2Flkml%2FCAHk-%3DwgTM%2BcN7zyUZacGQDv3DuuoA4LORNPWgb1Y
> > _Z1p4iedNQ%40mail.gmail.com%2F&amp;data=02%7C01%7Cjobialek%40microso
> > ft.com%7C4bce6c76554b4dcf4c2b08d7e64a7848%7C72f988bf86f141af91ab2d7c
> > d011db47%7C1%7C0%7C637231080235723408&amp;sdata=256CyAUusLIf8IetQfyd
> > 3KNAlIqVwV8uvjuPc6daP14%3D&amp;reserved=0
> > Unsurprisingly, I strongly agree. ;)
> >
> I don't see why claiming that pattern-init is bad helps make the case
> for zero-init.

Perhaps I did not express it well enough, but both have meaningful and important uses. My goal here is to illustrate how zero-init is being used (or preferred) in many real situations, as an argument for why it should not be hidden behind what some might see as a scary enable flag.

> > Apple
> >
> > I can't speak meaningfully here, but I've heard rumors that they are
> > depending on zero-init as well. Perhaps someone there can clarify
> > how they are using these features?
> >
> There's a difference between "depending on zero-init" (as in, the
> group in question is okay with relying on implicit zeroing on code
> reviews, etc.) and the use of zero-init as some sort of
> defence-in-depth approach. Are these rumours clear as to which?

My understanding was the latter, but I hope to find out for real via this thread! :) It's not clear to me either.

> > So, while I understand the earlier objections to zero-init from a
> > "language fork" concern, I think this isn't a position that can
> > really stand up to the reality of how many projects are using the
> > feature (even via non-Clang compilers). Given that so much code is
> > going to be built using zero-init, what's the best way for Clang to adapt here?
>
> It happens that there is zero-init and it's at least close enough to
> what these projects want, but it is actually what they need?

Yes, it's expressly what is desired from a security perspective. (And quite to the relief of that same community, comes with the least performance impact, which is an unfortunately uncommon scenario in security flaw mitigations.) I tried to detail that earlier in my email where it's directly what is indicated as a meaningful defense against the long history of real-world "uninitialized" variable attacks: setting everything to zero is the best defense for the entire class of flaws.

--
Kees Cook
_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: making -ftrivial-auto-var-init=zero a first-class option

Fangrui Song via cfe-dev
In reply to this post by Fangrui Song via cfe-dev
On Tue, Apr 21, 2020 at 06:44:44PM -0400, Arthur O'Dwyer wrote:

> On Tue, Apr 21, 2020 at 6:12 PM Richard Smith via cfe-dev <[hidden email]> wrote:
> > What you're proposing is, without question, a language extension. Our
> > policy on language extensions is documented here:
> > http://clang.llvm.org/get_involved.html
> >
> > Right now, this fails at point 4. We do not want to create or encourage
> > the creation of language dialects and non-portable code, so the place to
> > have this discussion is in the C and C++ committees. Both committees have
> > processes for specifying optional features these days, and they might be
> > amenable to using those processes to standardize the behavior you're asking
> > for. (I mean, maybe not, but our policy requires that you at least try.)

Well, I wasn't intending to re-discuss the presence of the feature. I was
trying to point out that it does exist in Clang already, and it exists
in other compilers too (and is being rapidly embraced by projects). So,
language extension or not, this behavior is present in released C/C++
binaries. Do you mean that _because_ it's present, it should be brought
to the committee for standardization? If so, what's the right path there?

> > However, there is a variant on what you're proposing that might fare
> > better: instead of guaranteeing zero-initialization, we could guarantee
> > that any observation of an uninitialized variable *either* gives produces
> > zero or results in a trap. That is: it's still undefined to read from
> > uninitialized variables -- we still do not guarantee what will happen if
> > you do, and will warn on uninitialized uses and so on -- but we would bound
> > the damage that can result from such accesses. You would get the security
> > hardening benefits with the modest binary size impact. That approach would
> > not introduce the risk of creating a language dialect (at least, not to the
> > same extent), so our policy on avoiding language extensions would not apply.

While I like the trap idea, I must say that I don't have a lot of
confidence that it can be done in a way that would actually provide the
same coverage benefits. How would such detection work? There are entire
suites of tools (e.g. KMSan) that exist for this kind of thing and are not
really suited for production use because it's so expensive to implement.

> I dont understand the point you're making here.  You're saying that if
> Clang provides reliable behavior, that's a language extension and therefore
> impossible; but if Clang provides behavior that unpredictably switches
> between only two possible alternatives (zero or trap), then that's not a
> language extension anymore and therefore is possible?

Er, is that what was meant? I mean, don't we kind of already have this
state in Clang already with "pattern" or "zero"? I'm just hoping to drop
the -enable... flag.

> I suspect many of the people quoted in the original quote-fest would not be
> happy with "zero *or trap*" as the two behaviors.
> What if you made it "zero *or one*"?  That is, whenever you access an
> uninitialized variable, you are guaranteed to get either all-bits-zero or
> else all-bits-zero-except-for-the-last-bit-which-is-1?  Would that
> selection of two behaviors leave matters sufficiently unspecified so as to
> dodge this "language extension" nonsense?

Speaking with my kernel security flaw mitigation hat on, no, zero-init is
the desired feature unless trap is equally or less expensive, in which
case trap is great because then the kernel will warn and then set it
to zero.

--
Kees Cook
_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: making -ftrivial-auto-var-init=zero a first-class option

Fangrui Song via cfe-dev
In reply to this post by Fangrui Song via cfe-dev
On Tue, 21 Apr 2020 at 15:46, Arthur O'Dwyer via cfe-dev <[hidden email]> wrote:
On Tue, Apr 21, 2020 at 6:12 PM Richard Smith via cfe-dev <[hidden email]> wrote:
What you're proposing is, without question, a language extension. Our policy on language extensions is documented here: http://clang.llvm.org/get_involved.html

Right now, this fails at point 4. We do not want to create or encourage the creation of language dialects and non-portable code, so the place to have this discussion is in the C and C++ committees. Both committees have processes for specifying optional features these days, and they might be amenable to using those processes to standardize the behavior you're asking for. (I mean, maybe not, but our policy requires that you at least try.)

However, there is a variant on what you're proposing that might fare better: instead of guaranteeing zero-initialization, we could guarantee that any observation of an uninitialized variable *either* gives produces zero or results in a trap. That is: it's still undefined to read from uninitialized variables -- we still do not guarantee what will happen if you do, and will warn on uninitialized uses and so on -- but we would bound the damage that can result from such accesses. You would get the security hardening benefits with the modest binary size impact. That approach would not introduce the risk of creating a language dialect (at least, not to the same extent), so our policy on avoiding language extensions would not apply.

I dont understand the point you're making here.  You're saying that if Clang provides reliable behavior, that's a language extension and therefore impossible; but if Clang provides behavior that unpredictably switches between only two possible alternatives (zero or trap), then that's not a language extension anymore and therefore is possible?

A major part of the purpose of point 4 of the policy is to prevent creation of language dialects, such as have unfortunately been created by (for example) the -fno-exceptions and -fno-rtti flags. If people can rely on uninitialized variables behaving as if initialized to zero, then they will write code that assumes that to be the case, and such code will not work in compilers / compilation modes that don't provide that language dialect. If people cannot rely on uninitialized variables behaving as if initialized to zero, then we mitigate the risk of creating a language dialect.
 
I suspect many of the people quoted in the original quote-fest would not be happy with "zero or trap" as the two behaviors.

They are, of course, welcome to speak up and express that opinion. But note that one reason that people want zero-init to be used from a security perspective is to give a high chance that use of the uninitialized value (especially when it is a pointer) will crash, and issuing a trap instead is consistent with that. (Another reason is that zero is very unlikely to be an out-of-bounds array index, but issuing a trap in that case is likely more desirable than producing a wrong-but-in-bounds result.)

What if you made it "zero or one"?  That is, whenever you access an uninitialized variable, you are guaranteed to get either all-bits-zero or else all-bits-zero-except-for-the-last-bit-which-is-1?  Would that selection of two behaviors leave matters sufficiently unspecified so as to dodge this "language extension" nonsense?

Please try to keep a respectful tone. Our policy in this area exists for a reason and is not "nonsense".

Any reasonable alternative that practically mitigates the risk of creating a language dialect, while still addressing the presented use cases, would seem in scope for discussion.

Alternatively, what about "all-bits-zero or else we explode the physical computer," or "all-bits-zero or else we output a proof of Goldbach's conjecture"?

–Arthur
_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev

_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: making -ftrivial-auto-var-init=zero a first-class option

Fangrui Song via cfe-dev
In reply to this post by Fangrui Song via cfe-dev


On 4/21/20 3:11 PM, Richard Smith via cfe-dev wrote:
What you're proposing is, without question, a language extension. Our policy on language extensions is documented here: http://clang.llvm.org/get_involved.html

Right now, this fails at point 4. We do not want to create or encourage the creation of language dialects and non-portable code, so the place to have this discussion is in the C and C++ committees. Both committees have processes for specifying optional features these days, and they might be amenable to using those processes to standardize the behavior you're asking for. (I mean, maybe not, but our policy requires that you at least try.)

However, there is a variant on what you're proposing that might fare better: instead of guaranteeing zero-initialization, we could guarantee that any observation of an uninitialized variable *either* gives produces zero or results in a trap. That is: it's still undefined to read from uninitialized variables -- we still do not guarantee what will happen if you do, and will warn on uninitialized uses and so on -- but we would bound the damage that can result from such accesses. You would get the security hardening benefits with the modest binary size impact. That approach would not introduce the risk of creating a language dialect (at least, not to the same extent), so our policy on avoiding language extensions would not apply.

Richard, just to check here, it sounds to me like you're raising more a point of specification then of implementation right?  That is, you're not stating that the actual implementation must sometimes trap (when producing a zero wouldn't), but that the specification of the flags and docs must leave the possibility there of?

If I'm complete misinterpreting, please just say so.  I don't want to start a tangent discussion here, I just spotted what sound like it could be a "quick fix" which lets to OP achieve their objective and wanted to call it out if in fact I'd read correctly.

Philip


On Tue, 21 Apr 2020 at 14:21, Kees Cook via cfe-dev <[hidden email]> wrote:
Hi,

tl;dr: I'd like to revisit making -ftrivial-auto-var-init=zero an expressly
supported option. To do this, I think we need to either entirely remove
"-enable-trivial-auto-var-init-zero-knowing-it-will-be-removed-from-clang"
or rename it to something more directly reflecting the issue, like
"-enable-trivial-auto-var-init-zero-knowing-it-forks-the-language".

This is currently open as https://bugs.llvm.org/show_bug.cgi?id=45497

Here is the situation: -ftrivial-auto-var-init=pattern is great for
debugging, but -ftrivial-auto-var-init=zero is needed for production
systems for mainly two reasons, each of which I will try to express context
for:

1) performance and size

As measured by various Google folks across a few projects and in
various places, there's a fairly significant performance impact of
using pattern-init over zero-init. I can let other folks chime in
with their exact numbers, but I can at least share some measurements
Alexander Potapenko made with the Linux kernel (see "Performance costs"):
https://clangbuiltlinux.github.io/CBL-meetup-2020-slides/glider/Fighting_uninitialized_memory_%40_CBL_Meetup_2020.pdf
tl;dr: zero-init tended to be half the cost of pattern-init, though it
varied based on workload, and binary size impact fell over 95% going
from pattern-init to zero-init.

2) security

Another driving factor (see below from various vendors/projects), is the
security stance. Putting non-zero values into most variables types ends
up making them arguably more dangerous than if they were zero-filled.
Most notably, sizes and indexes and less likely to be used out of bounds
if they are zero-initialized. The same holds for bool values that tend
to indicate success instead of failing safe with a false value. While
pointers in the non-canonical range are nice, zero tends to be just
as good. There are certainly exceptions here, but the bulk of the
historical record on how "uninitialized" variables have been used in
real world exploitation involve their being non-zero, and analysis of
those bugs support that conclusion.


Various positions from vendors and projects:

Google (Android, Chrome OS)

Both Android and Chrome OS initially started using pattern-init, but due
to each of: the performance characteristics, the binary size changes, and
the less robust security stance, both projects have recently committed
to switching to zero-init.


Microsoft (Windows)

I'm repeating what Joe Bialek has told me, so he can clarify if I'm not
representing this correctly... While not using Clang/LLVM, Microsoft is
part of the larger C/C++ ecosystem and has implemented both zero-init
(for production builds) and pattern-init (for debug builds) in their
compiler too. They also chose zero-init for production expressly due
to the security benefits.

Some details of their work:
https://github.com/microsoft/MSRC-Security-Research/blob/master/presentations/2019_09_CppCon/CppCon2019%20-%20Killing%20Uninitialized%20Memory.pdf


Upstream Linux kernel

Linus Torvalds has directly stated that he wants zero-init:
"So I'd like the zeroing of local variables to be a native compiler
option..."
"This, btw, is why I also think that the "initialize with poison" is
pointless and wrong."
https://lore.kernel.org/lkml/CAHk-=wgTM+cN7zyUZacGQDv3DuuoA4LORNPWgb1Y_Z1p4iedNQ@.../
Unsurprisingly, I strongly agree. ;)


GrapheneOS is using zero-init (rather than patching Clang as it used to, to get
the same result):
https://twitter.com/DanielMicay/status/1248384468181643272


GCC
There's been mostly silence on the entire topic of automatic variable
initialization, though there have been patches proposed in the past for
zero-init:
https://gcc.gnu.org/legacy-ml/gcc-patches/2014-06/msg00615.html


Apple

I can't speak meaningfully here, but I've heard rumors that they are
depending on zero-init as well. Perhaps someone there can clarify how
they are using these features?



So, while I understand the earlier objections to zero-init from a
"language fork" concern, I think this isn't a position that can really
stand up to the reality of how many projects are using the feature (even
via non-Clang compilers). Given that so much code is going to be built
using zero-init, what's the best way for Clang to adapt here? I would
prefer to just drop the -enable... option entirely, but I think just
renaming it would be fine too.

Thoughts/flames? ;)

--
Kees Cook
_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev

_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev

_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: making -ftrivial-auto-var-init=zero a first-class option

Fangrui Song via cfe-dev
In reply to this post by Fangrui Song via cfe-dev
On Tue, 21 Apr 2020 at 16:39, Kees Cook via cfe-dev <[hidden email]> wrote:
On Tue, Apr 21, 2020 at 06:44:44PM -0400, Arthur O'Dwyer wrote:
> On Tue, Apr 21, 2020 at 6:12 PM Richard Smith via cfe-dev <[hidden email]> wrote:
> > What you're proposing is, without question, a language extension. Our
> > policy on language extensions is documented here:
> > http://clang.llvm.org/get_involved.html
> >
> > Right now, this fails at point 4. We do not want to create or encourage
> > the creation of language dialects and non-portable code, so the place to
> > have this discussion is in the C and C++ committees. Both committees have
> > processes for specifying optional features these days, and they might be
> > amenable to using those processes to standardize the behavior you're asking
> > for. (I mean, maybe not, but our policy requires that you at least try.)

Well, I wasn't intending to re-discuss the presence of the feature. I was
trying to point out that it does exist in Clang already, and it exists
in other compilers too (and is being rapidly embraced by projects). So,
language extension or not, this behavior is present in released C/C++
binaries. Do you mean that _because_ it's present, it should be brought
to the committee for standardization? If so, what's the right path there?

The existence of the --long-ugly-flag-name-that-says-we'll-remove-the-feature is the way we currently try to avoid introducing a language dialect. If we remove that flag as is proposed, then we are effectively relitigating the question of whether to have the feature at all.
 
> > However, there is a variant on what you're proposing that might fare
> > better: instead of guaranteeing zero-initialization, we could guarantee
> > that any observation of an uninitialized variable *either* gives produces
> > zero or results in a trap. That is: it's still undefined to read from
> > uninitialized variables -- we still do not guarantee what will happen if
> > you do, and will warn on uninitialized uses and so on -- but we would bound
> > the damage that can result from such accesses. You would get the security
> > hardening benefits with the modest binary size impact. That approach would
> > not introduce the risk of creating a language dialect (at least, not to the
> > same extent), so our policy on avoiding language extensions would not apply.

While I like the trap idea, I must say that I don't have a lot of
confidence that it can be done in a way that would actually provide the
same coverage benefits. How would such detection work? There are entire
suites of tools (e.g. KMSan) that exist for this kind of thing and are not
really suited for production use because it's so expensive to implement.

The idea is that we would eventually emit exactly the same code as with the zero-init approach, except with some additional IR-level markers that would allow control flow paths that always read an uninitialized value to be optimized into a trap instruction. (For example: instead of initializing with a literal zero, you initialize with a call to an intrinsic function. If you can see that an instruction that's allowed to fault depends on a value produced by such a function, then you replace it with a trap. And at some stage of the optimization pipeline, you replace all such calls to that intrinsic function with literal zeroes.)

We might still produce zeroes instead of traps almost all the time. I think that's OK, so long as we trap enough that "you're using a variable that's uninitialized" receives an "oops" response instead of an "it's OK, the compiler will definitely zero it" response.

And indeed it might even be OK if the initial behavior is that we *always* zero-initialize (as Philip asked), so long as our documentation clearly says that we do not guarantee that the value will be zero (only that we guarantee that *if the program continues*, the value will be zero), and our intent is that we may still produce traps or otherwise abort the computation.
 
> I dont understand the point you're making here.  You're saying that if
> Clang provides reliable behavior, that's a language extension and therefore
> impossible; but if Clang provides behavior that unpredictably switches
> between only two possible alternatives (zero or trap), then that's not a
> language extension anymore and therefore is possible?

Er, is that what was meant? I mean, don't we kind of already have this
state in Clang already with "pattern" or "zero"? I'm just hoping to drop
the -enable... flag.

> I suspect many of the people quoted in the original quote-fest would not be
> happy with "zero *or trap*" as the two behaviors.
> What if you made it "zero *or one*"?  That is, whenever you access an
> uninitialized variable, you are guaranteed to get either all-bits-zero or
> else all-bits-zero-except-for-the-last-bit-which-is-1?  Would that
> selection of two behaviors leave matters sufficiently unspecified so as to
> dodge this "language extension" nonsense?

Speaking with my kernel security flaw mitigation hat on, no, zero-init is
the desired feature unless trap is equally or less expensive, in which
case trap is great because then the kernel will warn and then set it
to zero.

--
Kees Cook
_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev

_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: making -ftrivial-auto-var-init=zero a first-class option

Fangrui Song via cfe-dev
In reply to this post by Fangrui Song via cfe-dev
On Tue, 21 Apr 2020 at 16:49, Philip Reames via cfe-dev <[hidden email]> wrote:

On 4/21/20 3:11 PM, Richard Smith via cfe-dev wrote:

What you're proposing is, without question, a language extension. Our policy on language extensions is documented here: http://clang.llvm.org/get_involved.html

Right now, this fails at point 4. We do not want to create or encourage the creation of language dialects and non-portable code, so the place to have this discussion is in the C and C++ committees. Both committees have processes for specifying optional features these days, and they might be amenable to using those processes to standardize the behavior you're asking for. (I mean, maybe not, but our policy requires that you at least try.)

However, there is a variant on what you're proposing that might fare better: instead of guaranteeing zero-initialization, we could guarantee that any observation of an uninitialized variable *either* gives produces zero or results in a trap. That is: it's still undefined to read from uninitialized variables -- we still do not guarantee what will happen if you do, and will warn on uninitialized uses and so on -- but we would bound the damage that can result from such accesses. You would get the security hardening benefits with the modest binary size impact. That approach would not introduce the risk of creating a language dialect (at least, not to the same extent), so our policy on avoiding language extensions would not apply.

Richard, just to check here, it sounds to me like you're raising more a point of specification then of implementation right?  That is, you're not stating that the actual implementation must sometimes trap (when producing a zero wouldn't), but that the specification of the flags and docs must leave the possibility there of?

Well, I think it's not sufficient to merely say that we might do something like trap, if our intent is that we never will. We would need to reasonably agree that (for example) if someone came forward with a patch that actually implemented said trapping behavior and didn't introduce any significant code size or performance impact, that we would consider such a change to be a quality of implementation improvement. But I don't think we need anyone to have actually committed themselves to producing such a patch, or any timeline or expectation of when (or indeed whether) it would be done. Sorry if this is splitting a hair, but I think it's an important hair to split. 

If I'm complete misinterpreting, please just say so.  I don't want to start a tangent discussion here, I just spotted what sound like it could be a "quick fix" which lets to OP achieve their objective and wanted to call it out if in fact I'd read correctly.

Philip


On Tue, 21 Apr 2020 at 14:21, Kees Cook via cfe-dev <[hidden email]> wrote:
Hi,

tl;dr: I'd like to revisit making -ftrivial-auto-var-init=zero an expressly
supported option. To do this, I think we need to either entirely remove
"-enable-trivial-auto-var-init-zero-knowing-it-will-be-removed-from-clang"
or rename it to something more directly reflecting the issue, like
"-enable-trivial-auto-var-init-zero-knowing-it-forks-the-language".

This is currently open as https://bugs.llvm.org/show_bug.cgi?id=45497

Here is the situation: -ftrivial-auto-var-init=pattern is great for
debugging, but -ftrivial-auto-var-init=zero is needed for production
systems for mainly two reasons, each of which I will try to express context
for:

1) performance and size

As measured by various Google folks across a few projects and in
various places, there's a fairly significant performance impact of
using pattern-init over zero-init. I can let other folks chime in
with their exact numbers, but I can at least share some measurements
Alexander Potapenko made with the Linux kernel (see "Performance costs"):
https://clangbuiltlinux.github.io/CBL-meetup-2020-slides/glider/Fighting_uninitialized_memory_%40_CBL_Meetup_2020.pdf
tl;dr: zero-init tended to be half the cost of pattern-init, though it
varied based on workload, and binary size impact fell over 95% going
from pattern-init to zero-init.

2) security

Another driving factor (see below from various vendors/projects), is the
security stance. Putting non-zero values into most variables types ends
up making them arguably more dangerous than if they were zero-filled.
Most notably, sizes and indexes and less likely to be used out of bounds
if they are zero-initialized. The same holds for bool values that tend
to indicate success instead of failing safe with a false value. While
pointers in the non-canonical range are nice, zero tends to be just
as good. There are certainly exceptions here, but the bulk of the
historical record on how "uninitialized" variables have been used in
real world exploitation involve their being non-zero, and analysis of
those bugs support that conclusion.


Various positions from vendors and projects:

Google (Android, Chrome OS)

Both Android and Chrome OS initially started using pattern-init, but due
to each of: the performance characteristics, the binary size changes, and
the less robust security stance, both projects have recently committed
to switching to zero-init.


Microsoft (Windows)

I'm repeating what Joe Bialek has told me, so he can clarify if I'm not
representing this correctly... While not using Clang/LLVM, Microsoft is
part of the larger C/C++ ecosystem and has implemented both zero-init
(for production builds) and pattern-init (for debug builds) in their
compiler too. They also chose zero-init for production expressly due
to the security benefits.

Some details of their work:
https://github.com/microsoft/MSRC-Security-Research/blob/master/presentations/2019_09_CppCon/CppCon2019%20-%20Killing%20Uninitialized%20Memory.pdf


Upstream Linux kernel

Linus Torvalds has directly stated that he wants zero-init:
"So I'd like the zeroing of local variables to be a native compiler
option..."
"This, btw, is why I also think that the "initialize with poison" is
pointless and wrong."
https://lore.kernel.org/lkml/CAHk-=wgTM+cN7zyUZacGQDv3DuuoA4LORNPWgb1Y_Z1p4iedNQ@.../
Unsurprisingly, I strongly agree. ;)


GrapheneOS is using zero-init (rather than patching Clang as it used to, to get
the same result):
https://twitter.com/DanielMicay/status/1248384468181643272


GCC
There's been mostly silence on the entire topic of automatic variable
initialization, though there have been patches proposed in the past for
zero-init:
https://gcc.gnu.org/legacy-ml/gcc-patches/2014-06/msg00615.html


Apple

I can't speak meaningfully here, but I've heard rumors that they are
depending on zero-init as well. Perhaps someone there can clarify how
they are using these features?



So, while I understand the earlier objections to zero-init from a
"language fork" concern, I think this isn't a position that can really
stand up to the reality of how many projects are using the feature (even
via non-Clang compilers). Given that so much code is going to be built
using zero-init, what's the best way for Clang to adapt here? I would
prefer to just drop the -enable... option entirely, but I think just
renaming it would be fine too.

Thoughts/flames? ;)

--
Kees Cook
_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev

_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev

_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: making -ftrivial-auto-var-init=zero a first-class option

Fangrui Song via cfe-dev
In reply to this post by Fangrui Song via cfe-dev
On Tue, Apr 21, 2020 at 7:19 PM Kees Cook <[hidden email]> wrote:
On Tue, Apr 21, 2020 at 06:29:07PM -0400, Hubert Tong wrote:
> On Tue, Apr 21, 2020 at 5:20 PM Kees Cook via cfe-dev <[hidden email]> wrote:
> > https://clangbuiltlinux.github.io/CBL-meetup-2020-slides/glider/Fighting_uninitialized_memory_%40_CBL_Meetup_2020.pdf
> > tl;dr: zero-init tended to be half the cost of pattern-init, though it
> > varied based on workload, and binary size impact fell over 95% going
> > from pattern-init to zero-init.
> >
> This does not seem to indicate why zero-init is preferred over a default of
> using no explicit policy in production.

Do you mean "leaving things uninitialized" when you say "no explicit
policy"?
Yes.

Maybe I've misunderstood. Google's goal of using auto-init is
to eliminate uninitialized variables in production as a security
defense.
This is the missing context that was not stated. The structure of your note indicated that the first reason for using zero-init in production is for performance and size (without indicating that some form of auto-init is presupposed for this production environment).
 
When examining zero-init vs pattern-init, there is a clear
advantage on performance and size for zero-init.
Okay.


> > as good. There are certainly exceptions here, but the bulk of the
> > historical record on how "uninitialized" variables have been used in
> >
> Maybe an explanation of the scare quotes around "uninitialized" would help
> clarify your position.

Ah, sorry, I always use quotes (they are not intended to scare but to
clarify) when discussing uninitialized variables in real-world contexts,
because they are, of course, not uninitialized in the sense of them
not having a value. The RAM contents have a value. Many people without
compiler backgrounds think of such variables as being uncontrollable
or meaningless, when in fact they are usually highly controllable by an
attacker, etc.
Understood; thanks.
 

> > Google (Android, Chrome OS)
> >
> > Both Android and Chrome OS initially started using pattern-init, but due
> > to each of: the performance characteristics, the binary size changes, and
> > the less robust security stance, both projects have recently committed
> > to switching to zero-init.
> >
> I'm not sure that this is clear in terms of whether the statements apply to
> debug/development or production. I don't think pattern-init is meant to be
> a tool for production builds, which leads me to think that the above
> statement is about debug builds, at which point I'm thinking that using
> zero-init only serves to hide problems.

The context for Google's use of zero-init was meant here to be about
production builds.
Got it. Thanks.
 

> > Upstream Linux kernel
> >
> > Linus Torvalds has directly stated that he wants zero-init:
> > "So I'd like the zeroing of local variables to be a native compiler
> > option..."
> > "This, btw, is why I also think that the "initialize with poison" is
> > pointless and wrong."
> >
> > https://lore.kernel.org/lkml/CAHk-=wgTM+cN7zyUZacGQDv3DuuoA4LORNPWgb1Y_Z1p4iedNQ@.../
> > Unsurprisingly, I strongly agree. ;)
> >
> I don't see why claiming that pattern-init is bad helps make the case for
> zero-init.

Perhaps I did not express it well enough, but both have meaningful and
important uses. My goal here is to illustrate how zero-init is being
used (or preferred) in many real situations, as an argument for why it
should not be hidden behind what some might see as a scary enable flag.
Understood.
 

> > Apple
> >
> > I can't speak meaningfully here, but I've heard rumors that they are
> > depending on zero-init as well. Perhaps someone there can clarify how
> > they are using these features?
> >
> There's a difference between "depending on zero-init" (as in, the group in
> question is okay with relying on implicit zeroing on code reviews, etc.)
> and the use of zero-init as some sort of defence-in-depth approach. Are
> these rumours clear as to which?

My understanding was the latter, but I hope to find out for real via
this thread! :) It's not clear to me either.

> > So, while I understand the earlier objections to zero-init from a
> > "language fork" concern, I think this isn't a position that can really
> > stand up to the reality of how many projects are using the feature (even
> > via non-Clang compilers). Given that so much code is going to be built
> > using zero-init, what's the best way for Clang to adapt here?
>
> It happens that there is zero-init and it's at least close enough to what
> these projects want, but it is actually what they need?

Yes, it's expressly what is desired from a security perspective. (And
quite to the relief of that same community, comes with the least
performance impact, which is an unfortunately uncommon scenario in
security flaw mitigations.) I tried to detail that earlier in my email
where it's directly what is indicated as a meaningful defense against
the long history of real-world "uninitialized" variable attacks: setting
everything to zero is the best defense for the entire class of flaws.
I'm intrigued by Richard's suggestion of trying to trap on "easy to identify" cases of reading an uninitialized variable. I applaud the effort being made to improve the status quo of the industry. I'm not sure I have a better idea at this point.
 

--
Kees Cook

_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: making -ftrivial-auto-var-init=zero a first-class option

Fangrui Song via cfe-dev
In reply to this post by Fangrui Song via cfe-dev


On Tue, Apr 21, 2020 at 2:20 PM Kees Cook via cfe-dev <[hidden email]> wrote:
Hi,

tl;dr: I'd like to revisit making -ftrivial-auto-var-init=zero an expressly
supported option. To do this, I think we need to either entirely remove
"-enable-trivial-auto-var-init-zero-knowing-it-will-be-removed-from-clang"
or rename it to something more directly reflecting the issue, like
"-enable-trivial-auto-var-init-zero-knowing-it-forks-the-language".

This is currently open as https://bugs.llvm.org/show_bug.cgi?id=45497

Here is the situation: -ftrivial-auto-var-init=pattern is great for
debugging, but -ftrivial-auto-var-init=zero is needed for production
systems for mainly two reasons, each of which I will try to express context
for:

1) performance and size

As measured by various Google folks across a few projects and in
various places, there's a fairly significant performance impact of
using pattern-init over zero-init. I can let other folks chime in
with their exact numbers, but I can at least share some measurements
Alexander Potapenko made with the Linux kernel (see "Performance costs"):
https://clangbuiltlinux.github.io/CBL-meetup-2020-slides/glider/Fighting_uninitialized_memory_%40_CBL_Meetup_2020.pdf
tl;dr: zero-init tended to be half the cost of pattern-init, though it
varied based on workload, and binary size impact fell over 95% going
from pattern-init to zero-init.

2) security

Another driving factor (see below from various vendors/projects), is the
security stance. Putting non-zero values into most variables types ends
up making them arguably more dangerous than if they were zero-filled.
Most notably, sizes and indexes and less likely to be used out of bounds
if they are zero-initialized. The same holds for bool values that tend
to indicate success instead of failing safe with a false value. While
pointers in the non-canonical range are nice, zero tends to be just
as good. There are certainly exceptions here, but the bulk of the
historical record on how "uninitialized" variables have been used in
real world exploitation involve their being non-zero, and analysis of
those bugs support that conclusion.


Various positions from vendors and projects:

Google (Android, Chrome OS)

Both Android and Chrome OS initially started using pattern-init, but due
to each of: the performance characteristics, the binary size changes, and
the less robust security stance, both projects have recently committed
to switching to zero-init.


Microsoft (Windows)

I'm repeating what Joe Bialek has told me, so he can clarify if I'm not
representing this correctly... While not using Clang/LLVM, Microsoft is
part of the larger C/C++ ecosystem and has implemented both zero-init
(for production builds) and pattern-init (for debug builds) in their
compiler too. They also chose zero-init for production expressly due
to the security benefits.

Some details of their work:
https://github.com/microsoft/MSRC-Security-Research/blob/master/presentations/2019_09_CppCon/CppCon2019%20-%20Killing%20Uninitialized%20Memory.pdf

They mention that "zero is the safest value from security point-of-view", and you mentioned it also in multiple places. Is there a detailed analysis somewhere that explains this? (I'm not knowledgeable but always interested to learn more)

Overall when I read their perf analysis I would try to advocate to kill the flag and make zero-init the default for non-array POD. This would simplify the perf tuning aspect of the compiler (the optimizer heuristics would be always tuned for this) and most software out there could be hand-tuned with respect to this as well. This seems better than yet another mode in the compiler, in particular for a "production oriented" option.
This is also another way to address the "language fork" issue: if the major compiler out there (clang, gcc, MSVC at least) were to implement this behavior by default, this could then in turn help convincing the standard committee to adopt it in a future revision.

When I see -fno-rtti and/or -fno-exception which are actually removing entire features of the language, it seems absolutely not comparable to what is at hand here: adopting zero-init for POD by default does not make any existing software "not work" anymore and isn't making software incompatible with each others. 
It also seems to me that contrary to the other flags, the zero-init behavior is a standard-compliant implementation, which to me does not fit obviously into the "language extensions" category anymore.

Finally there is precedent for such "standard compliant" behavior where the compiler will be more conservative that what the standard consider undefined behavior: strict aliasing. Even though this could provide similar performance gain, this is not our default mode, and this seems like a quite similar case to the zero-init to me.


-- 
Mehdi






Upstream Linux kernel

Linus Torvalds has directly stated that he wants zero-init:
"So I'd like the zeroing of local variables to be a native compiler
option..."
"This, btw, is why I also think that the "initialize with poison" is
pointless and wrong."
https://lore.kernel.org/lkml/CAHk-=wgTM+cN7zyUZacGQDv3DuuoA4LORNPWgb1Y_Z1p4iedNQ@.../
Unsurprisingly, I strongly agree. ;)


GrapheneOS is using zero-init (rather than patching Clang as it used to, to get
the same result):
https://twitter.com/DanielMicay/status/1248384468181643272


GCC
There's been mostly silence on the entire topic of automatic variable
initialization, though there have been patches proposed in the past for
zero-init:
https://gcc.gnu.org/legacy-ml/gcc-patches/2014-06/msg00615.html


Apple

I can't speak meaningfully here, but I've heard rumors that they are
depending on zero-init as well. Perhaps someone there can clarify how
they are using these features?



So, while I understand the earlier objections to zero-init from a
"language fork" concern, I think this isn't a position that can really
stand up to the reality of how many projects are using the feature (even
via non-Clang compilers). Given that so much code is going to be built
using zero-init, what's the best way for Clang to adapt here? I would
prefer to just drop the -enable... option entirely, but I think just
renaming it would be fine too.

Thoughts/flames? ;)

--
Kees Cook
_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev

_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: making -ftrivial-auto-var-init=zero a first-class option

Fangrui Song via cfe-dev
In reply to this post by Fangrui Song via cfe-dev


On Apr 21, 2020, at 3:11 PM, Richard Smith via cfe-dev <[hidden email]> wrote:

What you're proposing is, without question, a language extension.

Is defining UB a language extension? :-)
The standard already lets the compiler define it to whatever it wants.


Our policy on language extensions is documented here: http://clang.llvm.org/get_involved.html

Right now, this fails at point 4. We do not want to create or encourage the creation of language dialects and non-portable code, so the place to have this discussion is in the C and C++ committees. Both committees have processes for specifying optional features these days, and they might be amenable to using those processes to standardize the behavior you're asking for. (I mean, maybe not, but our policy requires that you at least try.)

I agree that both committees should look at this.


However, there is a variant on what you're proposing that might fare better: instead of guaranteeing zero-initialization, we could guarantee that any observation of an uninitialized variable *either* gives produces zero or results in a trap. That is: it's still undefined to read from uninitialized variables -- we still do not guarantee what will happen if you do, and will warn on uninitialized uses and so on -- but we would bound the damage that can result from such accesses. You would get the security hardening benefits with the modest binary size impact. That approach would not introduce the risk of creating a language dialect (at least, not to the same extent), so our policy on avoiding language extensions would not apply.

I agree that this is a nice approach, and I’d like to see it happen. There are many other approaches that I’d like to see explored, many of which would have performance benefits. I would like Kees to come out of the discussion with a clear guidance from the clang community on what’s expected. This is a bar I’d support.


On Tue, 21 Apr 2020 at 14:21, Kees Cook via cfe-dev <[hidden email]> wrote:
Hi,

tl;dr: I'd like to revisit making -ftrivial-auto-var-init=zero an expressly
supported option. To do this, I think we need to either entirely remove
"-enable-trivial-auto-var-init-zero-knowing-it-will-be-removed-from-clang"
or rename it to something more directly reflecting the issue, like
"-enable-trivial-auto-var-init-zero-knowing-it-forks-the-language".

This is currently open as https://bugs.llvm.org/show_bug.cgi?id=45497

Here is the situation: -ftrivial-auto-var-init=pattern is great for
debugging, but -ftrivial-auto-var-init=zero is needed for production
systems for mainly two reasons, each of which I will try to express context
for:

1) performance and size

As measured by various Google folks across a few projects and in
various places, there's a fairly significant performance impact of
using pattern-init over zero-init. I can let other folks chime in
with their exact numbers, but I can at least share some measurements
Alexander Potapenko made with the Linux kernel (see "Performance costs"):
https://clangbuiltlinux.github.io/CBL-meetup-2020-slides/glider/Fighting_uninitialized_memory_%40_CBL_Meetup_2020.pdf
tl;dr: zero-init tended to be half the cost of pattern-init, though it
varied based on workload, and binary size impact fell over 95% going
from pattern-init to zero-init.

2) security

Another driving factor (see below from various vendors/projects), is the
security stance. Putting non-zero values into most variables types ends
up making them arguably more dangerous than if they were zero-filled.
Most notably, sizes and indexes and less likely to be used out of bounds
if they are zero-initialized. The same holds for bool values that tend
to indicate success instead of failing safe with a false value. While
pointers in the non-canonical range are nice, zero tends to be just
as good. There are certainly exceptions here, but the bulk of the
historical record on how "uninitialized" variables have been used in
real world exploitation involve their being non-zero, and analysis of
those bugs support that conclusion.


Various positions from vendors and projects:

Google (Android, Chrome OS)

Both Android and Chrome OS initially started using pattern-init, but due
to each of: the performance characteristics, the binary size changes, and
the less robust security stance, both projects have recently committed
to switching to zero-init.


Microsoft (Windows)

I'm repeating what Joe Bialek has told me, so he can clarify if I'm not
representing this correctly... While not using Clang/LLVM, Microsoft is
part of the larger C/C++ ecosystem and has implemented both zero-init
(for production builds) and pattern-init (for debug builds) in their
compiler too. They also chose zero-init for production expressly due
to the security benefits.

Some details of their work:
https://github.com/microsoft/MSRC-Security-Research/blob/master/presentations/2019_09_CppCon/CppCon2019%20-%20Killing%20Uninitialized%20Memory.pdf


Upstream Linux kernel

Linus Torvalds has directly stated that he wants zero-init:
"So I'd like the zeroing of local variables to be a native compiler
option..."
"This, btw, is why I also think that the "initialize with poison" is
pointless and wrong."
https://lore.kernel.org/lkml/CAHk-=wgTM+cN7zyUZacGQDv3DuuoA4LORNPWgb1Y_Z1p4iedNQ@.../
Unsurprisingly, I strongly agree. ;)


GrapheneOS is using zero-init (rather than patching Clang as it used to, to get
the same result):
https://twitter.com/DanielMicay/status/1248384468181643272


GCC
There's been mostly silence on the entire topic of automatic variable
initialization, though there have been patches proposed in the past for
zero-init:
https://gcc.gnu.org/legacy-ml/gcc-patches/2014-06/msg00615.html


Apple

I can't speak meaningfully here, but I've heard rumors that they are
depending on zero-init as well. Perhaps someone there can clarify how
they are using these features?



So, while I understand the earlier objections to zero-init from a
"language fork" concern, I think this isn't a position that can really
stand up to the reality of how many projects are using the feature (even
via non-Clang compilers). Given that so much code is going to be built
using zero-init, what's the best way for Clang to adapt here? I would
prefer to just drop the -enable... option entirely, but I think just
renaming it would be fine too.

Thoughts/flames? ;)

--
Kees Cook
_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev


_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: making -ftrivial-auto-var-init=zero a first-class option

Fangrui Song via cfe-dev
In reply to this post by Fangrui Song via cfe-dev


On Apr 21, 2020, at 3:29 PM, Hubert Tong via cfe-dev <[hidden email]> wrote:

On Tue, Apr 21, 2020 at 5:20 PM Kees Cook via cfe-dev <[hidden email]> wrote:
Hi,

tl;dr: I'd like to revisit making -ftrivial-auto-var-init=zero an expressly
supported option. To do this, I think we need to either entirely remove
"-enable-trivial-auto-var-init-zero-knowing-it-will-be-removed-from-clang"
or rename it to something more directly reflecting the issue, like
"-enable-trivial-auto-var-init-zero-knowing-it-forks-the-language".

This is currently open as https://bugs.llvm.org/show_bug.cgi?id=45497

Here is the situation: -ftrivial-auto-var-init=pattern is great for
debugging, but -ftrivial-auto-var-init=zero is needed for production
systems for mainly two reasons, each of which I will try to express context
for:

1) performance and size

As measured by various Google folks across a few projects and in
various places, there's a fairly significant performance impact of
using pattern-init over zero-init. I can let other folks chime in
with their exact numbers, but I can at least share some measurements
Alexander Potapenko made with the Linux kernel (see "Performance costs"):
https://clangbuiltlinux.github.io/CBL-meetup-2020-slides/glider/Fighting_uninitialized_memory_%40_CBL_Meetup_2020.pdf
tl;dr: zero-init tended to be half the cost of pattern-init, though it
varied based on workload, and binary size impact fell over 95% going
from pattern-init to zero-init.
This does not seem to indicate why zero-init is preferred over a default of using no explicit policy in production.
 

2) security

Another driving factor (see below from various vendors/projects), is the
security stance. Putting non-zero values into most variables types ends
up making them arguably more dangerous than if they were zero-filled.
Most notably, sizes and indexes and less likely to be used out of bounds
if they are zero-initialized. The same holds for bool values that tend
to indicate success instead of failing safe with a false value. While
pointers in the non-canonical range are nice, zero tends to be just
as good. There are certainly exceptions here, but the bulk of the
historical record on how "uninitialized" variables have been used in
Maybe an explanation of the scare quotes around "uninitialized" would help clarify your position.
 
real world exploitation involve their being non-zero, and analysis of
those bugs support that conclusion.


Various positions from vendors and projects:

Google (Android, Chrome OS)

Both Android and Chrome OS initially started using pattern-init, but due
to each of: the performance characteristics, the binary size changes, and
the less robust security stance, both projects have recently committed
to switching to zero-init.
I'm not sure that this is clear in terms of whether the statements apply to debug/development or production. I don't think pattern-init is meant to be a tool for production builds, which leads me to think that the above statement is about debug builds, at which point I'm thinking that using zero-init only serves to hide problems.

The entire feature (including pattern init) is exactly designed to be a tool for production builds. It is in production at Apple as well as Google (not just their servers, but their devices as well), and I’ve heard the same (privately) from many others, bit and small. I covered why in my LLVM dev meeting talk: https://www.youtube.com/watch?v=I-XUHPimq3o

I’ve also written this out in details in prior discussions. I can repeat here if needed, but I’m not sure more text is helpful at the moment :-)


Microsoft (Windows)

I'm repeating what Joe Bialek has told me, so he can clarify if I'm not
representing this correctly... While not using Clang/LLVM, Microsoft is
part of the larger C/C++ ecosystem and has implemented both zero-init
(for production builds) and pattern-init (for debug builds) in their
compiler too. They also chose zero-init for production expressly due
to the security benefits.

Some details of their work:
https://github.com/microsoft/MSRC-Security-Research/blob/master/presentations/2019_09_CppCon/CppCon2019%20-%20Killing%20Uninitialized%20Memory.pdf


Upstream Linux kernel

Linus Torvalds has directly stated that he wants zero-init:
"So I'd like the zeroing of local variables to be a native compiler
option..."
"This, btw, is why I also think that the "initialize with poison" is
pointless and wrong."
https://lore.kernel.org/lkml/CAHk-=wgTM+cN7zyUZacGQDv3DuuoA4LORNPWgb1Y_Z1p4iedNQ@.../
Unsurprisingly, I strongly agree. ;)
I don't see why claiming that pattern-init is bad helps make the case for zero-init.
 


GrapheneOS is using zero-init (rather than patching Clang as it used to, to get
the same result):
https://twitter.com/DanielMicay/status/1248384468181643272


GCC
There's been mostly silence on the entire topic of automatic variable
initialization, though there have been patches proposed in the past for
zero-init:
https://gcc.gnu.org/legacy-ml/gcc-patches/2014-06/msg00615.html


Apple

I can't speak meaningfully here, but I've heard rumors that they are
depending on zero-init as well. Perhaps someone there can clarify how
they are using these features?
There's a difference between "depending on zero-init" (as in, the group in question is okay with relying on implicit zeroing on code reviews, etc.) and the use of zero-init as some sort of defence-in-depth approach. Are these rumours clear as to which?
 



So, while I understand the earlier objections to zero-init from a
"language fork" concern, I think this isn't a position that can really
stand up to the reality of how many projects are using the feature (even
via non-Clang compilers). Given that so much code is going to be built
using zero-init, what's the best way for Clang to adapt here?
It happens that there is zero-init and it's at least close enough to what these projects want, but it is actually what they need?
 
I would
prefer to just drop the -enable... option entirely, but I think just
renaming it would be fine too.

Thoughts/flames? ;)

-- 
Kees Cook
_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev


_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: making -ftrivial-auto-var-init=zero a first-class option

Fangrui Song via cfe-dev
In reply to this post by Fangrui Song via cfe-dev


On Apr 21, 2020, at 2:20 PM, Kees Cook <[hidden email]> wrote:

Hi,

tl;dr: I'd like to revisit making -ftrivial-auto-var-init=zero an expressly
supported option. To do this, I think we need to either entirely remove
"-enable-trivial-auto-var-init-zero-knowing-it-will-be-removed-from-clang"
or rename it to something more directly reflecting the issue, like
"-enable-trivial-auto-var-init-zero-knowing-it-forks-the-language".

This is currently open as https://bugs.llvm.org/show_bug.cgi?id=45497

Here is the situation: -ftrivial-auto-var-init=pattern is great for
debugging, but -ftrivial-auto-var-init=zero is needed for production
systems for mainly two reasons, each of which I will try to express context
for:

1) performance and size

As measured by various Google folks across a few projects and in
various places, there's a fairly significant performance impact of
using pattern-init over zero-init. I can let other folks chime in
with their exact numbers, but I can at least share some measurements
Alexander Potapenko made with the Linux kernel (see "Performance costs"):
https://clangbuiltlinux.github.io/CBL-meetup-2020-slides/glider/Fighting_uninitialized_memory_%40_CBL_Meetup_2020.pdf
tl;dr: zero-init tended to be half the cost of pattern-init, though it
varied based on workload, and binary size impact fell over 95% going
from pattern-init to zero-init.

The main reason folks were OK with me putting in zero init *at all* was to compare performance between 1) no init 2) zero init 3) pattern init, and *then* tune pattern init to the “floor” set by zero init. In other words: try to get to as low a cost as possible. I and other optimizer folks put significant efforts towards this, and we still are, but we’ve only done so for our binaries, and only on front-end / middle-end / ARM64. There’s still very low-lying fruit for pattern init on x86, and there’s more to be optimized on ARM64 and general middle-end. I’m therefore not sure we’re honoring the spirit of the agreement without putting in more optimization effort. To clarify: the performance got better, but we haven’t tried enough to declare “pattern init will definitely never get acceptably close to zero init”.

At the same time, it’s pretty obvious that synthesizing zero will always be cheaper than synthesizing repeated bytes. The question we’ve still to answer is: does this matter?

I’d like to see the remainder of the optimization work happen. It benefits generic code, not just pattern init. I don’t think it’s a mountain of work.


2) security

Another driving factor (see below from various vendors/projects), is the
security stance. Putting non-zero values into most variables types ends
up making them arguably more dangerous than if they were zero-filled.
Most notably, sizes and indexes and less likely to be used out of bounds
if they are zero-initialized. The same holds for bool values that tend
to indicate success instead of failing safe with a false value. While
pointers in the non-canonical range are nice, zero tends to be just
as good. There are certainly exceptions here, but the bulk of the
historical record on how "uninitialized" variables have been used in
real world exploitation involve their being non-zero, and analysis of
those bugs support that conclusion.


Various positions from vendors and projects:

Google (Android, Chrome OS)

Both Android and Chrome OS initially started using pattern-init, but due
to each of: the performance characteristics, the binary size changes, and
the less robust security stance, both projects have recently committed
to switching to zero-init.


Microsoft (Windows)

I'm repeating what Joe Bialek has told me, so he can clarify if I'm not
representing this correctly... While not using Clang/LLVM, Microsoft is
part of the larger C/C++ ecosystem and has implemented both zero-init
(for production builds) and pattern-init (for debug builds) in their
compiler too. They also chose zero-init for production expressly due
to the security benefits.

Some details of their work:
https://github.com/microsoft/MSRC-Security-Research/blob/master/presentations/2019_09_CppCon/CppCon2019%20-%20Killing%20Uninitialized%20Memory.pdf


Upstream Linux kernel

Linus Torvalds has directly stated that he wants zero-init:
"So I'd like the zeroing of local variables to be a native compiler
option..."
"This, btw, is why I also think that the "initialize with poison" is
pointless and wrong."
https://lore.kernel.org/lkml/CAHk-=[hidden email]/
Unsurprisingly, I strongly agree. ;)


GrapheneOS is using zero-init (rather than patching Clang as it used to, to get
the same result):
https://twitter.com/DanielMicay/status/1248384468181643272


GCC
There's been mostly silence on the entire topic of automatic variable
initialization, though there have been patches proposed in the past for
zero-init:
https://gcc.gnu.org/legacy-ml/gcc-patches/2014-06/msg00615.html


Apple

I can't speak meaningfully here, but I've heard rumors that they are
depending on zero-init as well. Perhaps someone there can clarify how
they are using these features?

What I can easily share is in my LLVM dev meeting talk: https://www.youtube.com/watch?v=I-XUHPimq3o
There I say that we’ve at least deployed it in Secure Enclave code, and XNU. I’ve seen people on the internet find other places we seem to have deployed it :-)


So, while I understand the earlier objections to zero-init from a
"language fork" concern, I think this isn't a position that can really
stand up to the reality of how many projects are using the feature (even
via non-Clang compilers). Given that so much code is going to be built
using zero-init, what's the best way for Clang to adapt here? I would
prefer to just drop the -enable... option entirely, but I think just
renaming it would be fine too.

Thoughts/flames? ;)

--
Kees Cook


_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: making -ftrivial-auto-var-init=zero a first-class option

Fangrui Song via cfe-dev
On Wed, Apr 22, 2020 at 5:43 AM JF Bastien <[hidden email]> wrote:

>
>
>
> On Apr 21, 2020, at 2:20 PM, Kees Cook <[hidden email]> wrote:
>
> Hi,
>
> tl;dr: I'd like to revisit making -ftrivial-auto-var-init=zero an expressly
> supported option. To do this, I think we need to either entirely remove
> "-enable-trivial-auto-var-init-zero-knowing-it-will-be-removed-from-clang"
> or rename it to something more directly reflecting the issue, like
> "-enable-trivial-auto-var-init-zero-knowing-it-forks-the-language".
>
> This is currently open as https://bugs.llvm.org/show_bug.cgi?id=45497
>
> Here is the situation: -ftrivial-auto-var-init=pattern is great for
> debugging, but -ftrivial-auto-var-init=zero is needed for production
> systems for mainly two reasons, each of which I will try to express context
> for:
>
> 1) performance and size
>
> As measured by various Google folks across a few projects and in
> various places, there's a fairly significant performance impact of
> using pattern-init over zero-init. I can let other folks chime in
> with their exact numbers, but I can at least share some measurements
> Alexander Potapenko made with the Linux kernel (see "Performance costs"):
> https://clangbuiltlinux.github.io/CBL-meetup-2020-slides/glider/Fighting_uninitialized_memory_%40_CBL_Meetup_2020.pdf
> tl;dr: zero-init tended to be half the cost of pattern-init, though it
> varied based on workload, and binary size impact fell over 95% going
> from pattern-init to zero-init.
>
>
> The main reason folks were OK with me putting in zero init *at all* was to compare performance between 1) no init 2) zero init 3) pattern init, and *then* tune pattern init to the “floor” set by zero init. In other words: try to get to as low a cost as possible. I and other optimizer folks put significant efforts towards this, and we still are, but we’ve only done so for our binaries, and only on front-end / middle-end / ARM64. There’s still very low-lying fruit for pattern init on x86, and there’s more to be optimized on ARM64 and general middle-end. I’m therefore not sure we’re honoring the spirit of the agreement without putting in more optimization effort. To clarify: the performance got better, but we haven’t tried enough to declare “pattern init will definitely never get acceptably close to zero init”.
>
> At the same time, it’s pretty obvious that synthesizing zero will always be cheaper than synthesizing repeated bytes. The question we’ve still to answer is: does this matter?


It is true that pattern initialization has recently become a lot
faster. Thanks for all the efforts!

But one of the reasons to switch from pattern initialization to zero
initialization is that the former breaks buggy code that happened to
work previously.
I.e. along with making production code more secure (by mitigating
information leaks) we also introduce new crashes, making it less
stable.

> I’d like to see the remainder of the optimization work happen. It benefits generic code, not just pattern init. I don’t think it’s a mountain of work.



--
Alexander Potapenko
Software Engineer

Google Germany GmbH
Erika-Mann-Straße, 33
80636 München

Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg
_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: making -ftrivial-auto-var-init=zero a first-class option

Fangrui Song via cfe-dev


On Apr 22, 2020, at 2:30 AM, Alexander Potapenko <[hidden email]> wrote:

On Wed, Apr 22, 2020 at 5:43 AM JF Bastien <[hidden email]> wrote:



On Apr 21, 2020, at 2:20 PM, Kees Cook <[hidden email]> wrote:

Hi,

tl;dr: I'd like to revisit making -ftrivial-auto-var-init=zero an expressly
supported option. To do this, I think we need to either entirely remove
"-enable-trivial-auto-var-init-zero-knowing-it-will-be-removed-from-clang"
or rename it to something more directly reflecting the issue, like
"-enable-trivial-auto-var-init-zero-knowing-it-forks-the-language".

This is currently open as https://bugs.llvm.org/show_bug.cgi?id=45497

Here is the situation: -ftrivial-auto-var-init=pattern is great for
debugging, but -ftrivial-auto-var-init=zero is needed for production
systems for mainly two reasons, each of which I will try to express context
for:

1) performance and size

As measured by various Google folks across a few projects and in
various places, there's a fairly significant performance impact of
using pattern-init over zero-init. I can let other folks chime in
with their exact numbers, but I can at least share some measurements
Alexander Potapenko made with the Linux kernel (see "Performance costs"):
https://clangbuiltlinux.github.io/CBL-meetup-2020-slides/glider/Fighting_uninitialized_memory_%40_CBL_Meetup_2020.pdf
tl;dr: zero-init tended to be half the cost of pattern-init, though it
varied based on workload, and binary size impact fell over 95% going
from pattern-init to zero-init.


The main reason folks were OK with me putting in zero init *at all* was to compare performance between 1) no init 2) zero init 3) pattern init, and *then* tune pattern init to the “floor” set by zero init. In other words: try to get to as low a cost as possible. I and other optimizer folks put significant efforts towards this, and we still are, but we’ve only done so for our binaries, and only on front-end / middle-end / ARM64. There’s still very low-lying fruit for pattern init on x86, and there’s more to be optimized on ARM64 and general middle-end. I’m therefore not sure we’re honoring the spirit of the agreement without putting in more optimization effort. To clarify: the performance got better, but we haven’t tried enough to declare “pattern init will definitely never get acceptably close to zero init”.

At the same time, it’s pretty obvious that synthesizing zero will always be cheaper than synthesizing repeated bytes. The question we’ve still to answer is: does this matter?


It is true that pattern initialization has recently become a lot
faster. Thanks for all the efforts!

But one of the reasons to switch from pattern initialization to zero
initialization is that the former breaks buggy code that happened to
work previously.
I.e. along with making production code more secure (by mitigating
information leaks) we also introduce new crashes, making it less
stable.

All I’m saying is what the original bar for acceptance was for zero-init: it was added with performance comparison as the justification, and I think folks expect us to deliver in that area (compare performance, and tune until they’re as close as can reasonably be).

What you mention here is new data which I think folks would want to consider. It's a valuable point to make, it would be useful to have data backing it up. However, I don’t think it invalidates the original performance comparison discussion, but it might make it less important.


I’d like to see the remainder of the optimization work happen. It benefits generic code, not just pattern init. I don’t think it’s a mountain of work.



-- 
Alexander Potapenko
Software Engineer

Google Germany GmbH
Erika-Mann-Straße, 33
80636 München

Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg


_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: making -ftrivial-auto-var-init=zero a first-class option

Fangrui Song via cfe-dev
In reply to this post by Fangrui Song via cfe-dev


On 4/21/20 4:59 PM, Richard Smith wrote:
On Tue, 21 Apr 2020 at 16:49, Philip Reames via cfe-dev <[hidden email]> wrote:

On 4/21/20 3:11 PM, Richard Smith via cfe-dev wrote:

What you're proposing is, without question, a language extension. Our policy on language extensions is documented here: http://clang.llvm.org/get_involved.html

Right now, this fails at point 4. We do not want to create or encourage the creation of language dialects and non-portable code, so the place to have this discussion is in the C and C++ committees. Both committees have processes for specifying optional features these days, and they might be amenable to using those processes to standardize the behavior you're asking for. (I mean, maybe not, but our policy requires that you at least try.)

However, there is a variant on what you're proposing that might fare better: instead of guaranteeing zero-initialization, we could guarantee that any observation of an uninitialized variable *either* gives produces zero or results in a trap. That is: it's still undefined to read from uninitialized variables -- we still do not guarantee what will happen if you do, and will warn on uninitialized uses and so on -- but we would bound the damage that can result from such accesses. You would get the security hardening benefits with the modest binary size impact. That approach would not introduce the risk of creating a language dialect (at least, not to the same extent), so our policy on avoiding language extensions would not apply.

Richard, just to check here, it sounds to me like you're raising more a point of specification then of implementation right?  That is, you're not stating that the actual implementation must sometimes trap (when producing a zero wouldn't), but that the specification of the flags and docs must leave the possibility there of?

Well, I think it's not sufficient to merely say that we might do something like trap, if our intent is that we never will. We would need to reasonably agree that (for example) if someone came forward with a patch that actually implemented said trapping behavior and didn't introduce any significant code size or performance impact, that we would consider such a change to be a quality of implementation improvement. But I don't think we need anyone to have actually committed themselves to producing such a patch, or any timeline or expectation of when (or indeed whether) it would be done. Sorry if this is splitting a hair, but I think it's an important hair to split.
Hair successfully split.  I agree it is a key distinction.

If I'm complete misinterpreting, please just say so.  I don't want to start a tangent discussion here, I just spotted what sound like it could be a "quick fix" which lets to OP achieve their objective and wanted to call it out if in fact I'd read correctly.

Philip


On Tue, 21 Apr 2020 at 14:21, Kees Cook via cfe-dev <[hidden email]> wrote:
Hi,

tl;dr: I'd like to revisit making -ftrivial-auto-var-init=zero an expressly
supported option. To do this, I think we need to either entirely remove
"-enable-trivial-auto-var-init-zero-knowing-it-will-be-removed-from-clang"
or rename it to something more directly reflecting the issue, like
"-enable-trivial-auto-var-init-zero-knowing-it-forks-the-language".

This is currently open as https://bugs.llvm.org/show_bug.cgi?id=45497

Here is the situation: -ftrivial-auto-var-init=pattern is great for
debugging, but -ftrivial-auto-var-init=zero is needed for production
systems for mainly two reasons, each of which I will try to express context
for:

1) performance and size

As measured by various Google folks across a few projects and in
various places, there's a fairly significant performance impact of
using pattern-init over zero-init. I can let other folks chime in
with their exact numbers, but I can at least share some measurements
Alexander Potapenko made with the Linux kernel (see "Performance costs"):
https://clangbuiltlinux.github.io/CBL-meetup-2020-slides/glider/Fighting_uninitialized_memory_%40_CBL_Meetup_2020.pdf
tl;dr: zero-init tended to be half the cost of pattern-init, though it
varied based on workload, and binary size impact fell over 95% going
from pattern-init to zero-init.

2) security

Another driving factor (see below from various vendors/projects), is the
security stance. Putting non-zero values into most variables types ends
up making them arguably more dangerous than if they were zero-filled.
Most notably, sizes and indexes and less likely to be used out of bounds
if they are zero-initialized. The same holds for bool values that tend
to indicate success instead of failing safe with a false value. While
pointers in the non-canonical range are nice, zero tends to be just
as good. There are certainly exceptions here, but the bulk of the
historical record on how "uninitialized" variables have been used in
real world exploitation involve their being non-zero, and analysis of
those bugs support that conclusion.


Various positions from vendors and projects:

Google (Android, Chrome OS)

Both Android and Chrome OS initially started using pattern-init, but due
to each of: the performance characteristics, the binary size changes, and
the less robust security stance, both projects have recently committed
to switching to zero-init.


Microsoft (Windows)

I'm repeating what Joe Bialek has told me, so he can clarify if I'm not
representing this correctly... While not using Clang/LLVM, Microsoft is
part of the larger C/C++ ecosystem and has implemented both zero-init
(for production builds) and pattern-init (for debug builds) in their
compiler too. They also chose zero-init for production expressly due
to the security benefits.

Some details of their work:
https://github.com/microsoft/MSRC-Security-Research/blob/master/presentations/2019_09_CppCon/CppCon2019%20-%20Killing%20Uninitialized%20Memory.pdf


Upstream Linux kernel

Linus Torvalds has directly stated that he wants zero-init:
"So I'd like the zeroing of local variables to be a native compiler
option..."
"This, btw, is why I also think that the "initialize with poison" is
pointless and wrong."
https://lore.kernel.org/lkml/CAHk-=wgTM+cN7zyUZacGQDv3DuuoA4LORNPWgb1Y_Z1p4iedNQ@.../
Unsurprisingly, I strongly agree. ;)


GrapheneOS is using zero-init (rather than patching Clang as it used to, to get
the same result):
https://twitter.com/DanielMicay/status/1248384468181643272


GCC
There's been mostly silence on the entire topic of automatic variable
initialization, though there have been patches proposed in the past for
zero-init:
https://gcc.gnu.org/legacy-ml/gcc-patches/2014-06/msg00615.html


Apple

I can't speak meaningfully here, but I've heard rumors that they are
depending on zero-init as well. Perhaps someone there can clarify how
they are using these features?



So, while I understand the earlier objections to zero-init from a
"language fork" concern, I think this isn't a position that can really
stand up to the reality of how many projects are using the feature (even
via non-Clang compilers). Given that so much code is going to be built
using zero-init, what's the best way for Clang to adapt here? I would
prefer to just drop the -enable... option entirely, but I think just
renaming it would be fine too.

Thoughts/flames? ;)

--
Kees Cook
_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev

_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev

_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
123