[libc++] RFC: Add Windows kernel support and partial MSVC 2015 support

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[libc++] RFC: Add Windows kernel support and partial MSVC 2015 support

Richard Pennington via cfe-dev
Abstract:
I would like to add Windows x86 and x64 kernel support to libc++.  My initial
compiler would be MSVC 2015 Update 3, though MSVC 2017 shouldn't be difficult to
add after the fact.

I would like to know if this is a port that the libc++ maintainers are willing
to accept.

Motivation:
C++ is intended to be useful for systems programming, and the kernel is as
"systems" as it gets.  There are a fair number of language features that lean
heavily on the presence of the library.  Initializer lists require library
support.  R-value references require std::move, std::forward, and others, or
require users to duplicate their functionality.  Variadic templates are
extremely cumbersome to work with if std::tuple is not available.  std::atomic
provides a much better interface than compiler intrinsics for working with the
C++ memory model.

A subset of the C++ language can be used (with caution) in the Windows kernel,
but the library remains out of reach.  The C++ library that ships with MSVC
won't compile at all in the kernel, even for headers that ostensibly make sense,
like <atomic>.

Since the vendor's C++ library doesn't work in the kernel, might as well try
and make libc++ work there instead.

There's no reason that the Windows ARM kernel couldn't be addressed as well.
However, my employer has no significant interest in that environment, and I have
no access to devices to test with for that environment.  I would be glad to help
review code to support the Windows ARM kernel though.

Strategy:
I will not be porting the entirety of libc++.  I will have exceptions and RTTI
turned off.  I plan on porting the freestanding headers [17.5.1.3,2 compliance],
minus <exception>, and <typeinfo>.  I also plan on porting the portions of
<algorithm>, <array>, <tuple>, and <utility> that can naturally be expressed in
an exception-free and RTTI-free environment.  I can be convinced that other
headers should also be included.

This means that string, vector, and the non-array containers won't be ported.
Any class or function that requires throwing exceptions to meet their standards
required behavior will be omitted.  That rules out a lot of classes that
allocate memory.

Avoiding allocations allows us to sidestep one other large issue.  In the
kernel, not all memory is equal.  There are several memory pools to choose from,
but the two most common memory pools are the pageable pool and the non-pageable
pool.  There is no clear correct answer for which pool a global operator new
should use, so we simply won't require an allocating new to be present for our
implementation.  Placement new shall remain though.

I don't believe I will need to port libc++abi, libunwind, and compiler-rt.

In some ways, this is a good first step for getting MSVC support.  The
functionality that is most coupled to POSIX-like operating systems is all in
the "hosted" portion of the library.

Prior Work:
Lawrence Crowl and Alberto Ganesh Barbati proposed this paper back in the C++11
time frame with mixed results:
http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2011/n3256.html

My employer has significant experience using C++ in the kernel.  We have been
using a modified version of STLPort for quite some time and learned a lot about
C++ library usage in the kernel, often the hard way.  The big, obvious lesson
is that a lot of the STL is either difficult, or impossible to work with when
exceptions are off and std::terminate is undesired.  There's nothing wrong with
sorting in the kernel though.

Challenges:
* Header partitioning.
    * I don't have an exact list of what functions and classes I will be
      keeping.  I do know that some of the headers I want to bring along have
      parts that I won't be keeping.  For instance, many of the algorithms
      allocate a temporary buffer in order to meet performance requirements.
    * I'll also need to figure out how to not drag along unwanted header
      dependencies.
* I don't know if anyone has ever attempted to get cmake to target a Windows
  kernel environment.  I don't think I will be trying to do this either.  I
  think I will be using a script similar in spirit to buildit.sh.
* Testing.
    * New 'requires' code will be needed to only attempt to run the tests that
      I care about.  I plan on doing this by white listing headers, and having
      the test runner examine the test to see if any unsupported headers are in
      the test.
    * Installing code so that it can be run in the kernel takes several seconds
      for each binary.
    * There is no facility in the Windows kernel for running a program starting
      at "main" in the context of the kernel.
    * The 32-bit Windows kernel requires a default calling convention of
      stdcall, but warns if "main" is not cdecl.
    * A common failure mode for libc++ tests is to crash or assert.  That
      will "blue-screen" a machine.
    * The existing lit test infrastructure assumes flags similar to gcc and
      llvm.  These assumptions are scattered throughout the code, and not in a
      centralized location.
* MSVC compiler feature set
    * No #include_next.  I'll need to use computed include paths, like an
      STL using caveman.
    * Limited expression SFINAE.  Some areas in the code are straightforward
      to fix, and others are not.
* C-runtime
    * The Windows kernel has its own implementation of the C-runtime.  I don't
      know all the details on it.  I suspect (but don't know) that it is
      derived from Dinkumware, but I know that it is not the same C-runtime
      as used in user mode.
     
I have a very early prototype that is poorly tested, and won't work at all
for existing supporting platforms.  Basically, I took the quick and dirty
approach of commenting out / deleting code that got in the way of the Windows
kernel.  It can still be somewhat instructive in the kinds of things I expect
to need to do (eventually) to the libc++ code base.  The prototype can
be found here: https://github.com/ben-craig/libcxx/tree/ntddk .


Thanks,
Ben Craig
_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [libc++] RFC: Add Windows kernel support and partial MSVC 2015 support

Richard Pennington via cfe-dev
You'll probably find VC 2017 to be significantly friendlier to libc++ than 2015 Update 3 was, given that so many compiler features and fixes were implemented for C++17 features, and for making libc++'s tests happier with VC's STL and FE. I would strongly recommend ignoring 2015 Update 3 and starting directly with VC 2017 (they are bin-compatible and built from the same branch of source code).

STL

-----Original Message-----
From: Ben Craig [mailto:[hidden email]]
Sent: Saturday, March 25, 2017 8:18 AM
To: via cfe-dev <[hidden email]>; [hidden email]; [hidden email]; Billy O'Neal (VC LIBS) <[hidden email]>; Stephan T. Lavavej <[hidden email]>; [hidden email]
Subject: [libc++] RFC: Add Windows kernel support and partial MSVC 2015 support

Abstract:
I would like to add Windows x86 and x64 kernel support to libc++.  My initial compiler would be MSVC 2015 Update 3, though MSVC 2017 shouldn't be difficult to add after the fact.

I would like to know if this is a port that the libc++ maintainers are willing to accept.

Motivation:
C++ is intended to be useful for systems programming, and the kernel is
C++ as
"systems" as it gets.  There are a fair number of language features that lean heavily on the presence of the library.  Initializer lists require library support.  R-value references require std::move, std::forward, and others, or require users to duplicate their functionality.  Variadic templates are extremely cumbersome to work with if std::tuple is not available.  std::atomic provides a much better interface than compiler intrinsics for working with the
C++ memory model.

A subset of the C++ language can be used (with caution) in the Windows kernel, but the library remains out of reach.  The C++ library that ships with MSVC won't compile at all in the kernel, even for headers that ostensibly make sense, like <atomic>.

Since the vendor's C++ library doesn't work in the kernel, might as well try and make libc++ work there instead.

There's no reason that the Windows ARM kernel couldn't be addressed as well.
However, my employer has no significant interest in that environment, and I have no access to devices to test with for that environment.  I would be glad to help review code to support the Windows ARM kernel though.

Strategy:
I will not be porting the entirety of libc++.  I will have exceptions and RTTI turned off.  I plan on porting the freestanding headers [17.5.1.3,2 compliance], minus <exception>, and <typeinfo>.  I also plan on porting the portions of <algorithm>, <array>, <tuple>, and <utility> that can naturally be expressed in an exception-free and RTTI-free environment.  I can be convinced that other headers should also be included.

This means that string, vector, and the non-array containers won't be ported.
Any class or function that requires throwing exceptions to meet their standards required behavior will be omitted.  That rules out a lot of classes that allocate memory.

Avoiding allocations allows us to sidestep one other large issue.  In the kernel, not all memory is equal.  There are several memory pools to choose from, but the two most common memory pools are the pageable pool and the non-pageable pool.  There is no clear correct answer for which pool a global operator new should use, so we simply won't require an allocating new to be present for our implementation.  Placement new shall remain though.

I don't believe I will need to port libc++abi, libunwind, and compiler-rt.

In some ways, this is a good first step for getting MSVC support.  The functionality that is most coupled to POSIX-like operating systems is all in the "hosted" portion of the library.

Prior Work:
Lawrence Crowl and Alberto Ganesh Barbati proposed this paper back in the C++11 time frame with mixed results:
https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.open-std.org%2Fjtc1%2Fsc22%2Fwg21%2Fdocs%2Fpapers%2F2011%2Fn3256.html&data=02%7C01%7Cstl%40exchange.microsoft.com%7C832c31de248041e34e6108d4739227c1%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636260518944916138&sdata=wRs3XX8FFwVLqZBaIttvL%2FRouxaE31o9WjqaTOEQ5Ic%3D&reserved=0

My employer has significant experience using C++ in the kernel.  We have been using a modified version of STLPort for quite some time and learned a lot about
C++ library usage in the kernel, often the hard way.  The big, obvious
C++ lesson
is that a lot of the STL is either difficult, or impossible to work with when exceptions are off and std::terminate is undesired.  There's nothing wrong with sorting in the kernel though.

Challenges:
* Header partitioning.
    * I don't have an exact list of what functions and classes I will be
      keeping.  I do know that some of the headers I want to bring along have
      parts that I won't be keeping.  For instance, many of the algorithms
      allocate a temporary buffer in order to meet performance requirements.
    * I'll also need to figure out how to not drag along unwanted header
      dependencies.
* I don't know if anyone has ever attempted to get cmake to target a Windows
  kernel environment.  I don't think I will be trying to do this either.  I
  think I will be using a script similar in spirit to buildit.sh.
* Testing.
    * New 'requires' code will be needed to only attempt to run the tests that
      I care about.  I plan on doing this by white listing headers, and having
      the test runner examine the test to see if any unsupported headers are in
      the test.
    * Installing code so that it can be run in the kernel takes several seconds
      for each binary.
    * There is no facility in the Windows kernel for running a program starting
      at "main" in the context of the kernel.
    * The 32-bit Windows kernel requires a default calling convention of
      stdcall, but warns if "main" is not cdecl.
    * A common failure mode for libc++ tests is to crash or assert.  That
      will "blue-screen" a machine.
    * The existing lit test infrastructure assumes flags similar to gcc and
      llvm.  These assumptions are scattered throughout the code, and not in a
      centralized location.
* MSVC compiler feature set
    * No #include_next.  I'll need to use computed include paths, like an
      STL using caveman.
    * Limited expression SFINAE.  Some areas in the code are straightforward
      to fix, and others are not.
* C-runtime
    * The Windows kernel has its own implementation of the C-runtime.  I don't
      know all the details on it.  I suspect (but don't know) that it is
      derived from Dinkumware, but I know that it is not the same C-runtime
      as used in user mode.
     
I have a very early prototype that is poorly tested, and won't work at all for existing supporting platforms.  Basically, I took the quick and dirty approach of commenting out / deleting code that got in the way of the Windows kernel.  It can still be somewhat instructive in the kinds of things I expect to need to do (eventually) to the libc++ code base.  The prototype can be found here: https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fben-craig%2Flibcxx%2Ftree%2Fntddk&data=02%7C01%7Cstl%40exchange.microsoft.com%7C832c31de248041e34e6108d4739227c1%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636260518944916138&sdata=uX9xJkK0U3yecRp36pY2ECE2zHnsPZo72ZZgF89m7YY%3D&reserved=0 .


Thanks,
Ben Craig
_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [libc++] RFC: Add Windows kernel support and partial MSVC 2015 support

Richard Pennington via cfe-dev
In reply to this post by Richard Pennington via cfe-dev




On Sat, Mar 25, 2017 at 9:18 AM, Ben Craig <[hidden email]> wrote:
Abstract:
I would like to add Windows x86 and x64 kernel support to libc++.  My initial
compiler would be MSVC 2015 Update 3, though MSVC 2017 shouldn't be difficult to
add after the fact.

I would like to know if this is a port that the libc++ maintainers are willing
to accept.

Before responding in depth I need to ask this question: Why not use Clang/C2 or simply Clang+LLVM?
I'm assuming they don't support targeting the kernel?

There seem to be three separate changes going on here, and each of the changes
will benefits and challenges. For that reason I think it's best to consider them
separately. 

(1) Porting to MSVC
(2) Supporting the Windows Kernel C library.
(3) Implementation a Freestanding configuration of libc++
(4) Implementing the Kernel-space compatible configuration described in the original email.

In general I'm happy to accept specific or niche libc++ changes as long as they
aren't detrimental to the overall libc++ quality and implementation complexity.
Changes to better support (1) or (3) would be beneficial to the larger libc++ audience and I would b
happy to upstream them. However I have large concerns regarding the changes required for (2) and (4)
as I suspect they won't satisfy the above requirement.

For example What if the Windows Kernel C library is incomplete, or significantly different from
existing implementations, and supporting it requires re-implementing the missing parts within libc++?
These portions would be virtually untestable outside of the Windows Kernel environment and would
quickly become unmaintained. Having such code in Libc++ could quickly become a detriment.

My main concern with (4) is the limited feature set that has been proposed. Specifically
how it limits the libc++ internals to the same feature set and the changes that would be
needed to support and maintain it.

First Libc++ cannot reasonably limit itself to the proposed language and library feature set since
it must be free to use "restricted" features within the implementation of non-restricted ones whenever
it is beneficial to the implementation. The burden of maintaining a working restricted feature set could
not fall on upstream maintainers.

Second, the changes required to either guard restricted features using #ifdef or remove restricted features
by re-structuring the headers would be vast and would require constant maintenance. Frankly I don't
see how libc++ or its existing users could benefit from upstreaming these changes (For example adding
#ifdef guards for every usage of `operator new`).

I think it would be much better to get as much of libc++ compiling as possible, even if it depends on restricted
features, and then finding another mechanism to restrict the features end-users are allowed to use (Such as clang-tidy).
This would eliminate the need to restructure headers or spam out internal #ifdef guards.

 
This means that string, vector, and the non-array containers won't be ported.
Any class or function that requires throwing exceptions to meet their standards
required behavior will be omitted.  That rules out a lot of classes that
allocate memory.

There are often ways to safely use STL containers w/o exceptions (Especially when
custom allocators are provided).
 

Avoiding allocations allows us to sidestep one other large issue.  In the
kernel, not all memory is equal.  There are several memory pools to choose from,
but the two most common memory pools are the pageable pool and the non-pageable
pool.  There is no clear correct answer for which pool a global operator new
should use, so we simply won't require an allocating new to be present for our
implementation.  Placement new shall remain though.

Containers don't use new/delete directly but instead go through the specified allocator,
allowing containers to change the allocation behavior on a per-object basis. Not 
supporting containers because of global operator new's behavior seems misguided.
 

My employer has significant experience using C++ in the kernel.  We have been
using a modified version of STLPort for quite some time and learned a lot about
C++ library usage in the kernel, often the hard way.  The big, obvious lesson
is that a lot of the STL is either difficult, or impossible to work with when
exceptions are off and std::terminate is undesired.  There's nothing wrong with
sorting in the kernel though. 

Challenges:
* Header partitioning.

Libc++ prefers larger monolithic headers over many well-partitioned headers. The idea is that hitting the filesystem
multiple times is slower than processing the single include file. Any proposed changes should keep this in mind.
 
    * I don't have an exact list of what functions and classes I will be
      keeping.  I do know that some of the headers I want to bring along have
      parts that I won't be keeping.  For instance, many of the algorithms
      allocate a temporary buffer in order to meet performance requirements.

I don't think partitioning <algorithm> into smaller headers would be beneficial
to existing libc++ users (If that's what you're suggesting). 

 
    * I'll also need to figure out how to not drag along unwanted header
      dependencies.

I don't see how libc++ could upstream changes since it requires every header
it currently includes (modulo bugs).

* Testing.
    * Installing code so that it can be run in the kernel takes several seconds
      for each binary. 
    * There is no facility in the Windows kernel for running a program starting
      at "main" in the context of the kernel.

I suspect this will be the largest hurdle to get the test-suite running. My only
idea for handling this would be to write a script to rename main() to some
unique function name and creating a separate test-driver that calls the re-named main.

This could also allow us to combine multiple tests into a single executable, avoiding
the cost of installing every test manually.


 
    * The 32-bit Windows kernel requires a default calling convention of
      stdcall, but warns if "main" is not cdecl.
    * A common failure mode for libc++ tests is to crash or assert.  That
      will "blue-screen" a machine.

I could envision a fix which replaces `<assert.h>` when compiling the tests, but that
would be a hack. Maybe the Windows Kernel C library provides a mechanism for replacing
the assert handler?

 .
* MSVC compiler feature set
    * No #include_next.  I'll need to use computed include paths, like an
      STL using caveman.

Using hard-coded kernel paths is not a change I see being upstream-able. However
we might be able to convince MSVC to implement #include_next if we can provide
strong enough rational for why we need it.

 
    * Limited expression SFINAE.  Some areas in the code are straightforward
      to fix, and others are not.

I'm violently against working around MSVC template bugs. Every time I've seen
it done it makes the implementation worse.

Another point is that libc++ doesn't have an <atomic> implementation for MSVC
so adding MSVC support would required adding one.

 
* C-runtime
    * The Windows kernel has its own implementation of the C-runtime.  I don't
      know all the details on it.  I suspect (but don't know) that it is
      derived from Dinkumware, but I know that it is not the same C-runtime
      as used in user mode.

I'm very curious to know the changes required to support the kernel C runtime.
Hopefully it's similar enough to other supported C libraries that very few changes
are needed.

I hope my response has been helpful, and it hasn't scared you away from using libc++.
These are my initial reactions and are in no way absolute. Please let me know if I can
clarify anything or if I got anything wrong.

/Eric


_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [libc++] RFC: Add Windows kernel support and partial MSVC 2015 support

Richard Pennington via cfe-dev

> There are often ways to safely use STL containers w/o exceptions (Especially when custom allocators are provided).

 

Yes, but those ways generally involve going to terminate() on OOM. That's not OK in a driver.

 

From: [hidden email]
Sent: Monday, March 27, 2017 4:37 PM
To: [hidden email]
Cc: [hidden email]; [hidden email]; [hidden email]; [hidden email]; [hidden email]
Subject: Re: [libc++] RFC: Add Windows kernel support and partial MSVC 2015 support

 

 

 

 

 

On Sat, Mar 25, 2017 at 9:18 AM, Ben Craig <[hidden email]> wrote:

Abstract:
I would like to add Windows x86 and x64 kernel support to libc++.  My initial
compiler would be MSVC 2015 Update 3, though MSVC 2017 shouldn't be difficult to
add after the fact.

I would like to know if this is a port that the libc++ maintainers are willing
to accept.

 

Before responding in depth I need to ask this question: Why not use Clang/C2 or simply Clang+LLVM?

I'm assuming they don't support targeting the kernel?

 

There seem to be three separate changes going on here, and each of the changes

will benefits and challenges. For that reason I think it's best to consider them

separately. 

 

(1) Porting to MSVC

(2) Supporting the Windows Kernel C library.

(3) Implementation a Freestanding configuration of libc++

(4) Implementing the Kernel-space compatible configuration described in the original email.

 

In general I'm happy to accept specific or niche libc++ changes as long as they

aren't detrimental to the overall libc++ quality and implementation complexity.

Changes to better support (1) or (3) would be beneficial to the larger libc++ audience and I would b

happy to upstream them. However I have large concerns regarding the changes required for (2) and (4)

as I suspect they won't satisfy the above requirement.

 

For example What if the Windows Kernel C library is incomplete, or significantly different from

existing implementations, and supporting it requires re-implementing the missing parts within libc++?

These portions would be virtually untestable outside of the Windows Kernel environment and would

quickly become unmaintained. Having such code in Libc++ could quickly become a detriment.

 

My main concern with (4) is the limited feature set that has been proposed. Specifically

how it limits the libc++ internals to the same feature set and the changes that would be

needed to support and maintain it.

 

First Libc++ cannot reasonably limit itself to the proposed language and library feature set since

it must be free to use "restricted" features within the implementation of non-restricted ones whenever

it is beneficial to the implementation. The burden of maintaining a working restricted feature set could

not fall on upstream maintainers.

 

Second, the changes required to either guard restricted features using #ifdef or remove restricted features

by re-structuring the headers would be vast and would require constant maintenance. Frankly I don't

see how libc++ or its existing users could benefit from upstreaming these changes (For example adding

#ifdef guards for every usage of `operator new`).

 

I think it would be much better to get as much of libc++ compiling as possible, even if it depends on restricted

features, and then finding another mechanism to restrict the features end-users are allowed to use (Such as clang-tidy).

This would eliminate the need to restructure headers or spam out internal #ifdef guards.

 

 

This means that string, vector, and the non-array containers won't be ported.
Any class or function that requires throwing exceptions to meet their standards
required behavior will be omitted.  That rules out a lot of classes that
allocate memory.

 

There are often ways to safely use STL containers w/o exceptions (Especially when

custom allocators are provided).

 

 

Avoiding allocations allows us to sidestep one other large issue.  In the
kernel, not all memory is equal.  There are several memory pools to choose from,
but the two most common memory pools are the pageable pool and the non-pageable
pool.  There is no clear correct answer for which pool a global operator new
should use, so we simply won't require an allocating new to be present for our
implementation.  Placement new shall remain though.

 

Containers don't use new/delete directly but instead go through the specified allocator,

allowing containers to change the allocation behavior on a per-object basis. Not 

supporting containers because of global operator new's behavior seems misguided.

 

 

My employer has significant experience using C++ in the kernel.  We have been
using a modified version of STLPort for quite some time and learned a lot about
C++ library usage in the kernel, often the hard way.  The big, obvious lesson
is that a lot of the STL is either difficult, or impossible to work with when
exceptions are off and std::terminate is undesired.  There's nothing wrong with
sorting in the kernel though. 


Challenges:
* Header partitioning.

 

Libc++ prefers larger monolithic headers over many well-partitioned headers. The idea is that hitting the filesystem

multiple times is slower than processing the single include file. Any proposed changes should keep this in mind.

 

    * I don't have an exact list of what functions and classes I will be
      keeping.  I do know that some of the headers I want to bring along have
      parts that I won't be keeping.  For instance, many of the algorithms
      allocate a temporary buffer in order to meet performance requirements.

 

I don't think partitioning <algorithm> into smaller headers would be beneficial

to existing libc++ users (If that's what you're suggesting). 

 

 

    * I'll also need to figure out how to not drag along unwanted header
      dependencies.

 

I don't see how libc++ could upstream changes since it requires every header

it currently includes (modulo bugs).

 

* Testing.
    * Installing code so that it can be run in the kernel takes several seconds
      for each binary. 

    * There is no facility in the Windows kernel for running a program starting
      at "main" in the context of the kernel.

 

I suspect this will be the largest hurdle to get the test-suite running. My only

idea for handling this would be to write a script to rename main() to some

unique function name and creating a separate test-driver that calls the re-named main.

 

This could also allow us to combine multiple tests into a single executable, avoiding

the cost of installing every test manually.

 

 

 

    * The 32-bit Windows kernel requires a default calling convention of
      stdcall, but warns if "main" is not cdecl.
    * A common failure mode for libc++ tests is to crash or assert.  That
      will "blue-screen" a machine.

 

I could envision a fix which replaces `<assert.h>` when compiling the tests, but that

would be a hack. Maybe the Windows Kernel C library provides a mechanism for replacing

the assert handler?

 

 .

* MSVC compiler feature set
    * No #include_next.  I'll need to use computed include paths, like an
      STL using caveman.

 

Using hard-coded kernel paths is not a change I see being upstream-able. However

we might be able to convince MSVC to implement #include_next if we can provide

strong enough rational for why we need it.

 

 

    * Limited expression SFINAE.  Some areas in the code are straightforward
      to fix, and others are not.

 

I'm violently against working around MSVC template bugs. Every time I've seen

it done it makes the implementation worse.

 

Another point is that libc++ doesn't have an <atomic> implementation for MSVC

so adding MSVC support would required adding one.

 

 

* C-runtime
    * The Windows kernel has its own implementation of the C-runtime.  I don't
      know all the details on it.  I suspect (but don't know) that it is
      derived from Dinkumware, but I know that it is not the same C-runtime
      as used in user mode.

 

I'm very curious to know the changes required to support the kernel C runtime.

Hopefully it's similar enough to other supported C libraries that very few changes

are needed.

 

I hope my response has been helpful, and it hasn't scared you away from using libc++.

These are my initial reactions and are in no way absolute. Please let me know if I can

clarify anything or if I got anything wrong.

 

/Eric

 

 


_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [libc++] RFC: Add Windows kernel support and partial MSVC 2015 support

Richard Pennington via cfe-dev

>> There are often ways to safely use STL containers w/o exceptions (Especially when custom allocators are provided).

> Yes, but those ways generally involve going to terminate() on OOM. That's not OK in a driver.

There is another way, but it isn't conforming, it's easy to get wrong in the library, and it's easy to get wrong for users.  It's basically the alternative that my company has used with STL Port.

You take an allocator (maybe even the default allocator), and you return nullptr on failure.  You also add a bool member (possibly static) that indicates whether the allocator has failed.  Then you check that bool anytime you use an STL container in a way that could fail.

 

This approach sort-of works with std::vector, especially if you develop reserve() paranoia.  It sort-of works with std::string if you ignore all the temporaries that std::string operations tend to generate.  It doesn't work very well at all for the node based containers, because they tend to dereference the node immediately after allocating it.

 

It's not really an approach I want to endorse.  My company has some home-rolled, STL inspired containers that use error codes, and sometimes even require "T" to have specific constructors that take those error codes by reference.  That approach is still error-prone, but less error-prone than using an STL container in a way that it wasn't intended.

 

> Before responding in depth I need to ask this question: Why not use Clang/C2 or simply Clang+LLVM?

> I'm assuming they don't support targeting the kernel?

I haven't even attempted to use Clang targeting the Windows kernel.  I did attempt to point Clang's static analyzer at a kernel component once, and I got some decent results... plus a lot of compiler errors that I hadn't worked through.  I've done nothing with Clang/C2 so far.

There are some strange MSVC flags required to build for the kernel.  I didn't see any web documentation for making stdcall the default calling convention, or the dreaded /Zc:wchar_t-, which turns wchar_t into an unsigned short.

Even if I did use Clang though, that would only address one of the four sub-features you identified, the port to MSVC.

 

> However I have large concerns regarding the changes required for [Kernel C library] and [Kernel subset] as I suspect they won't satisfy the above requirement.

Understood.  This was feedback I was expecting.  Subsets and niche platforms are a maintenance burden.

 

> For example What if the Windows Kernel C library is incomplete, or significantly different from existing implementations [...]

Depends on what it is.  If it is something niche, I will likely want to just not support that particular use.  For example, floating point use "works" in x86 kernel mode, with a pile of caveats and extra code.  If supporting that becomes annoying from my side, I'll likely just drop float.h and cfloat support.  Basically, if the kernel C library doesn't have support for it, I probably shouldn't be doing it...

 

... unless it's using the STL in the kernel.  Then, I'm totally in the right :)

 

> My main concern with (4) is the limited feature set that has been proposed. Specifically

> how it limits the libc++ internals to the same feature set and the changes that would be

> needed to support and maintain it.

>

> First Libc++ cannot reasonably limit itself to the proposed language and library feature set since

> it must be free to use "restricted" features within the implementation of non-restricted ones whenever

> it is beneficial to the implementation. The burden of maintaining a working restricted feature set could

> not fall on upstream maintainers.

I'm not going to make any promises... but would a public build bot address your concerns?  What if the restricted feature set got standardized as the C++2x definition of a freestanding implementation?

 

I can definitely see maintaining a restricted include set to be burdensome  I will have to think on how to address that in a reasonable way.  Your clang-tidy suggestion may be useful here.

 

I don't think maintaining a restricted use of features will be particularly hard though.  For <array>, <tuple>, <utility>, and <algorithm>, do you foresee changes where an existing function doesn't throw, but starts to throw?  Or doesn't currently allocate, but starts to allocate?  If we were talking about the containers, or iostreams, or regex, I would definitely agree that the implementation flexibility is needed.  I'm not sure I agree when it's a smaller list of low level headers though.

 

>

> Second, the changes required to either guard restricted features using #ifdef or remove restricted features

> by re-structuring the headers would be vast and would require constant maintenance. Frankly I don't

> see how libc++ or its existing users could benefit from upstreaming these changes (For example adding

> #ifdef guards for every usage of `operator new`).

 

I hope to use this project as a way to provide implementation experience to revise the definition of a freestanding implementation.  I think that C++11 got it wrong, and that the current definition isn't useful.  I also believe that other kernel, embedded, and bare-metal users would want the same feature set that I'm looking for.  This feature set would have been useful to me on other projects that weren't targeting the Windows Kernel.

 

So for the cost, I agree that it will require a substantial amount of work.  Let's say that it's somewhere between _LIBCPP_HAS_NO_THREADS and _LIBCPP_NO_EXCEPTIONS amount of work.  There is still benefit for other users though.

 

> I think it would be much better to get as much of libc++ compiling as possible, even if it depends on restricted

> features, and then finding another mechanism to restrict the features end-users are allowed to use (Such as clang-tidy).

> This would eliminate the need to restructure headers or spam out internal #ifdef guards.

One thing I want to avoid is turning straightforward mistakes into errors that are even more cryptic than what C++ users have grown accustomed to.  I don't really want the failure condition of inplace_merge to be a linker error complaining about a missing operator new.  clang-tidy might help with that.  #ifdefs are a straightforward way to address that concern, though it requires a substantial amount of effort.  Would you be opposed to widespread static_asserts?

 

> I don't think partitioning <algorithm> into smaller headers would be beneficial

> to existing libc++ users (If that's what you're suggesting).

The three obvious options are 1) do nothing, 2) #ifdefs around the existing declarations and definitions, and 3) #ifdefs around #includes that pull in the supported feature set.  I could also put static_asserts in select areas that I know will cause trouble.

 

>> * I'll also need to figure out how to not drag along unwanted header dependencies.

 

> I don't see how libc++ could upstream changes since it requires every header

> it currently includes (modulo bugs).

The straightforward (and labor intensive) way is with #ifdef _LIBCPP_HAS_NO_ALLOC

 

 

> My only idea for handling this would be to write a script to rename main() to some

> unique function name and creating a separate test-driver that calls the re-named main.

I had similar thoughts.  Alternatively, I could just run execution tests overnight.  I can call main just fine, I just can't call 5000 different functions each called main.

 

> I could envision a fix which replaces `<assert.h>` when compiling the tests, but that

> would be a hack. Maybe the Windows Kernel C library provides a mechanism for replacing

> the assert handler?

I think the best solution, long term, is to use an "expect" macro instead of an assert macro, similar to google test.  I'm not sure that I want to take that on though.  "assert" is used in more than 4000 tests.  I'll have to look at this more later.

 

> Using hard-coded kernel paths is not a change I see being upstream-able. However

> we might be able to convince MSVC to implement #include_next if we can provide

> strong enough rational for why we need it.

These wouldn't be hard coded paths.  The code would look something like...

#if __LIBCPP_HAS_INCLUDE_NEXT

#include_next <limits.h>

#else

#include __LIBCPP_HACK_INCLUDE_NEXT(limits.h)

#endif

__config would have the definition of __LIBCPP_HACK_INCLUDE_NEXT, and

config_site.in would have the relative path needed to glue everything together ("../../km/crt" in my case).

 

> I'm violently against working around MSVC template bugs. Every time I've seen

> it done it makes the implementation worse.

That's fair.  I will say that not _all_ of the workarounds make worse code,

but quite a few do.  This may end up being what forces me to MSVC 2017.

 

One example of a workaround that I don't think made things worse...

struct _LIBCPP_TYPE_VIS __check_tuple_constructor_fail {

     template <class ...>

-    static constexpr bool __enable_default() { return false; }

+    static constexpr bool __enable_default = false;

> Another point is that libc++ doesn't have an <atomic> implementation for MSVC

> so adding MSVC support would required adding one.

Yep, I've got one implemented (and untested).  This is the biggest reason why

I excluded ARM support explicitly.

 

> I hope my response has been helpful, and it hasn't scared you away from using libc++.

> These are my initial reactions and are in no way absolute. Please let me know if I can

> clarify anything or if I got anything wrong.

The response has been helpful and useful.  No complaints here on the tone or style of your reply.

 

Thanks,

Ben Craig

 

From: Billy O'Neal (VC LIBS) [mailto:[hidden email]]
Sent: Monday, March 27, 2017 6:42 PM
To: Eric Fiselier <[hidden email]>; Ben Craig <[hidden email]>
Cc: via cfe-dev <[hidden email]>; [hidden email]; Stephan T. Lavavej <[hidden email]>; [hidden email]
Subject: RE: [libc++] RFC: Add Windows kernel support and partial MSVC 2015 support

 

> There are often ways to safely use STL containers w/o exceptions (Especially when custom allocators are provided).

 

Yes, but those ways generally involve going to terminate() on OOM. That's not OK in a driver.

 

From: [hidden email]
Sent: Monday, March 27, 2017 4:37 PM
To: [hidden email]
Cc: [hidden email]; [hidden email]; [hidden email]; [hidden email]; [hidden email]
Subject: Re: [libc++] RFC: Add Windows kernel support and partial MSVC 2015 support

 

 

 

 

 

On Sat, Mar 25, 2017 at 9:18 AM, Ben Craig <[hidden email]> wrote:

Abstract:
I would like to add Windows x86 and x64 kernel support to libc++.  My initial
compiler would be MSVC 2015 Update 3, though MSVC 2017 shouldn't be difficult to
add after the fact.

I would like to know if this is a port that the libc++ maintainers are willing
to accept.

 

Before responding in depth I need to ask this question: Why not use Clang/C2 or simply Clang+LLVM?

I'm assuming they don't support targeting the kernel?

 

There seem to be three separate changes going on here, and each of the changes

will benefits and challenges. For that reason I think it's best to consider them

separately. 

 

(1) Porting to MSVC

(2) Supporting the Windows Kernel C library.

(3) Implementation a Freestanding configuration of libc++

(4) Implementing the Kernel-space compatible configuration described in the original email.

 

In general I'm happy to accept specific or niche libc++ changes as long as they

aren't detrimental to the overall libc++ quality and implementation complexity.

Changes to better support (1) or (3) would be beneficial to the larger libc++ audience and I would b

happy to upstream them. However I have large concerns regarding the changes required for (2) and (4)

as I suspect they won't satisfy the above requirement.

 

For example What if the Windows Kernel C library is incomplete, or significantly different from

existing implementations, and supporting it requires re-implementing the missing parts within libc++?

These portions would be virtually untestable outside of the Windows Kernel environment and would

quickly become unmaintained. Having such code in Libc++ could quickly become a detriment.

 

My main concern with (4) is the limited feature set that has been proposed. Specifically

how it limits the libc++ internals to the same feature set and the changes that would be

needed to support and maintain it.

 

First Libc++ cannot reasonably limit itself to the proposed language and library feature set since

it must be free to use "restricted" features within the implementation of non-restricted ones whenever

it is beneficial to the implementation. The burden of maintaining a working restricted feature set could

not fall on upstream maintainers.

 

Second, the changes required to either guard restricted features using #ifdef or remove restricted features

by re-structuring the headers would be vast and would require constant maintenance. Frankly I don't

see how libc++ or its existing users could benefit from upstreaming these changes (For example adding

#ifdef guards for every usage of `operator new`).

 

I think it would be much better to get as much of libc++ compiling as possible, even if it depends on restricted

features, and then finding another mechanism to restrict the features end-users are allowed to use (Such as clang-tidy).

This would eliminate the need to restructure headers or spam out internal #ifdef guards.

 

 

This means that string, vector, and the non-array containers won't be ported.
Any class or function that requires throwing exceptions to meet their standards
required behavior will be omitted.  That rules out a lot of classes that
allocate memory.

 

There are often ways to safely use STL containers w/o exceptions (Especially when

custom allocators are provided).

 

 

Avoiding allocations allows us to sidestep one other large issue.  In the
kernel, not all memory is equal.  There are several memory pools to choose from,
but the two most common memory pools are the pageable pool and the non-pageable
pool.  There is no clear correct answer for which pool a global operator new
should use, so we simply won't require an allocating new to be present for our
implementation.  Placement new shall remain though.

 

Containers don't use new/delete directly but instead go through the specified allocator,

allowing containers to change the allocation behavior on a per-object basis. Not 

supporting containers because of global operator new's behavior seems misguided.

 

 

My employer has significant experience using C++ in the kernel.  We have been
using a modified version of STLPort for quite some time and learned a lot about
C++ library usage in the kernel, often the hard way.  The big, obvious lesson
is that a lot of the STL is either difficult, or impossible to work with when
exceptions are off and std::terminate is undesired.  There's nothing wrong with
sorting in the kernel though. 


Challenges:
* Header partitioning.

 

Libc++ prefers larger monolithic headers over many well-partitioned headers. The idea is that hitting the filesystem

multiple times is slower than processing the single include file. Any proposed changes should keep this in mind.

 

    * I don't have an exact list of what functions and classes I will be
      keeping.  I do know that some of the headers I want to bring along have
      parts that I won't be keeping.  For instance, many of the algorithms
      allocate a temporary buffer in order to meet performance requirements.

 

I don't think partitioning <algorithm> into smaller headers would be beneficial

to existing libc++ users (If that's what you're suggesting). 

 

 

    * I'll also need to figure out how to not drag along unwanted header
      dependencies.

 

I don't see how libc++ could upstream changes since it requires every header

it currently includes (modulo bugs).

 

* Testing.
    * Installing code so that it can be run in the kernel takes several seconds
      for each binary. 

    * There is no facility in the Windows kernel for running a program starting
      at "main" in the context of the kernel.

 

I suspect this will be the largest hurdle to get the test-suite running. My only

idea for handling this would be to write a script to rename main() to some

unique function name and creating a separate test-driver that calls the re-named main.

 

This could also allow us to combine multiple tests into a single executable, avoiding

the cost of installing every test manually.

 

 

 

    * The 32-bit Windows kernel requires a default calling convention of
      stdcall, but warns if "main" is not cdecl.
    * A common failure mode for libc++ tests is to crash or assert.  That
      will "blue-screen" a machine.

 

I could envision a fix which replaces `<assert.h>` when compiling the tests, but that

would be a hack. Maybe the Windows Kernel C library provides a mechanism for replacing

the assert handler?

 

 .

* MSVC compiler feature set
    * No #include_next.  I'll need to use computed include paths, like an
      STL using caveman.

 

Using hard-coded kernel paths is not a change I see being upstream-able. However

we might be able to convince MSVC to implement #include_next if we can provide

strong enough rational for why we need it.

 

 

    * Limited expression SFINAE.  Some areas in the code are straightforward
      to fix, and others are not.

 

I'm violently against working around MSVC template bugs. Every time I've seen

it done it makes the implementation worse.

 

Another point is that libc++ doesn't have an <atomic> implementation for MSVC

so adding MSVC support would required adding one.

 

 

* C-runtime
    * The Windows kernel has its own implementation of the C-runtime.  I don't
      know all the details on it.  I suspect (but don't know) that it is
      derived from Dinkumware, but I know that it is not the same C-runtime
      as used in user mode.

 

I'm very curious to know the changes required to support the kernel C runtime.

Hopefully it's similar enough to other supported C libraries that very few changes

are needed.

 

I hope my response has been helpful, and it hasn't scared you away from using libc++.

These are my initial reactions and are in no way absolute. Please let me know if I can

clarify anything or if I got anything wrong.

 

/Eric

 

 


_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [libc++] RFC: Add Windows kernel support and partial MSVC 2015 support

Richard Pennington via cfe-dev
On Tue, Mar 28, 2017 at 11:28 AM, Ben Craig via cfe-dev <[hidden email]> wrote:

>> There are often ways to safely use STL containers w/o exceptions (Especially when custom allocators are provided).

> Yes, but those ways generally involve going to terminate() on OOM. That's not OK in a driver.

There is another way, but it isn't conforming, it's easy to get wrong in the library, and it's easy to get wrong for users.  It's basically the alternative that my company has used with STL Port.

You take an allocator (maybe even the default allocator), and you return nullptr on failure.  You also add a bool member (possibly static) that indicates whether the allocator has failed.  Then you check that bool anytime you use an STL container in a way that could fail.

 

This approach sort-of works with std::vector, especially if you develop reserve() paranoia.  It sort-of works with std::string if you ignore all the temporaries that std::string operations tend to generate.  It doesn't work very well at all for the node based containers, because they tend to dereference the node immediately after allocating it.

 

It's not really an approach I want to endorse.  My company has some home-rolled, STL inspired containers that use error codes, and sometimes even require "T" to have specific constructors that take those error codes by reference.  That approach is still error-prone, but less error-prone than using an STL container in a way that it wasn't intended.

 

> Before responding in depth I need to ask this question: Why not use Clang/C2 or simply Clang+LLVM?

> I'm assuming they don't support targeting the kernel?

I haven't even attempted to use Clang targeting the Windows kernel.  I did attempt to point Clang's static analyzer at a kernel component once, and I got some decent results... plus a lot of compiler errors that I hadn't worked through.  I've done nothing with Clang/C2 so far.

There are some strange MSVC flags required to build for the kernel.  I didn't see any web documentation for making stdcall the default calling convention,


clang-cl implements /Gz as far as I can tell.
 

or the dreaded /Zc:wchar_t-, which turns wchar_t into an unsigned short.


This isn't implemented, but probably isn't too hard to add, if you want to give it a shot.
 

Even if I did use Clang though, that would only address one of the four sub-features you identified, the port to MSVC.

 

> However I have large concerns regarding the changes required for [Kernel C library] and [Kernel subset] as I suspect they won't satisfy the above requirement.

Understood.  This was feedback I was expecting.  Subsets and niche platforms are a maintenance burden.

 

> For example What if the Windows Kernel C library is incomplete, or significantly different from existing implementations [...]

Depends on what it is.  If it is something niche, I will likely want to just not support that particular use.  For example, floating point use "works" in x86 kernel mode, with a pile of caveats and extra code.  If supporting that becomes annoying from my side, I'll likely just drop float.h and cfloat support.  Basically, if the kernel C library doesn't have support for it, I probably shouldn't be doing it...

 

... unless it's using the STL in the kernel.  Then, I'm totally in the right :)

 

> My main concern with (4) is the limited feature set that has been proposed. Specifically

> how it limits the libc++ internals to the same feature set and the changes that would be

> needed to support and maintain it.

>

> First Libc++ cannot reasonably limit itself to the proposed language and library feature set since

> it must be free to use "restricted" features within the implementation of non-restricted ones whenever

> it is beneficial to the implementation. The burden of maintaining a working restricted feature set could

> not fall on upstream maintainers.

I'm not going to make any promises... but would a public build bot address your concerns?  What if the restricted feature set got standardized as the C++2x definition of a freestanding implementation?

 

I can definitely see maintaining a restricted include set to be burdensome  I will have to think on how to address that in a reasonable way.  Your clang-tidy suggestion may be useful here.

 

I don't think maintaining a restricted use of features will be particularly hard though.  For <array>, <tuple>, <utility>, and <algorithm>, do you foresee changes where an existing function doesn't throw, but starts to throw?  Or doesn't currently allocate, but starts to allocate?  If we were talking about the containers, or iostreams, or regex, I would definitely agree that the implementation flexibility is needed.  I'm not sure I agree when it's a smaller list of low level headers though.

 

>

> Second, the changes required to either guard restricted features using #ifdef or remove restricted features

> by re-structuring the headers would be vast and would require constant maintenance. Frankly I don't

> see how libc++ or its existing users could benefit from upstreaming these changes (For example adding

> #ifdef guards for every usage of `operator new`).

 

I hope to use this project as a way to provide implementation experience to revise the definition of a freestanding implementation.  I think that C++11 got it wrong, and that the current definition isn't useful.  I also believe that other kernel, embedded, and bare-metal users would want the same feature set that I'm looking for.  This feature set would have been useful to me on other projects that weren't targeting the Windows Kernel.

 

So for the cost, I agree that it will require a substantial amount of work.  Let's say that it's somewhere between _LIBCPP_HAS_NO_THREADS and _LIBCPP_NO_EXCEPTIONS amount of work.  There is still benefit for other users though.

 

> I think it would be much better to get as much of libc++ compiling as possible, even if it depends on restricted

> features, and then finding another mechanism to restrict the features end-users are allowed to use (Such as clang-tidy).

> This would eliminate the need to restructure headers or spam out internal #ifdef guards.

One thing I want to avoid is turning straightforward mistakes into errors that are even more cryptic than what C++ users have grown accustomed to.  I don't really want the failure condition of inplace_merge to be a linker error complaining about a missing operator new.  clang-tidy might help with that.  #ifdefs are a straightforward way to address that concern, though it requires a substantial amount of effort.  Would you be opposed to widespread static_asserts?

 

> I don't think partitioning <algorithm> into smaller headers would be beneficial

> to existing libc++ users (If that's what you're suggesting).

The three obvious options are 1) do nothing, 2) #ifdefs around the existing declarations and definitions, and 3) #ifdefs around #includes that pull in the supported feature set.  I could also put static_asserts in select areas that I know will cause trouble.

 

>> * I'll also need to figure out how to not drag along unwanted header dependencies.

 

> I don't see how libc++ could upstream changes since it requires every header

> it currently includes (modulo bugs).

The straightforward (and labor intensive) way is with #ifdef _LIBCPP_HAS_NO_ALLOC

 

 

> My only idea for handling this would be to write a script to rename main() to some

> unique function name and creating a separate test-driver that calls the re-named main.

I had similar thoughts.  Alternatively, I could just run execution tests overnight.  I can call main just fine, I just can't call 5000 different functions each called main.

 

> I could envision a fix which replaces `<assert.h>` when compiling the tests, but that

> would be a hack. Maybe the Windows Kernel C library provides a mechanism for replacing

> the assert handler?

I think the best solution, long term, is to use an "expect" macro instead of an assert macro, similar to google test.  I'm not sure that I want to take that on though.  "assert" is used in more than 4000 tests.  I'll have to look at this more later.

 

> Using hard-coded kernel paths is not a change I see being upstream-able. However

> we might be able to convince MSVC to implement #include_next if we can provide

> strong enough rational for why we need it.

These wouldn't be hard coded paths.  The code would look something like...

#if __LIBCPP_HAS_INCLUDE_NEXT

#include_next <limits.h>

#else

#include __LIBCPP_HACK_INCLUDE_NEXT(limits.h)

#endif

__config would have the definition of __LIBCPP_HACK_INCLUDE_NEXT, and

config_site.in would have the relative path needed to glue everything together ("../../km/crt" in my case).

 

> I'm violently against working around MSVC template bugs. Every time I've seen

> it done it makes the implementation worse.

That's fair.  I will say that not _all_ of the workarounds make worse code,

but quite a few do.  This may end up being what forces me to MSVC 2017.

 

One example of a workaround that I don't think made things worse...

struct _LIBCPP_TYPE_VIS __check_tuple_constructor_fail {

     template <class ...>

-    static constexpr bool __enable_default() { return false; }

+    static constexpr bool __enable_default = false;

> Another point is that libc++ doesn't have an <atomic> implementation for MSVC

> so adding MSVC support would required adding one.

Yep, I've got one implemented (and untested).  This is the biggest reason why

I excluded ARM support explicitly.

 

> I hope my response has been helpful, and it hasn't scared you away from using libc++.

> These are my initial reactions and are in no way absolute. Please let me know if I can

> clarify anything or if I got anything wrong.

The response has been helpful and useful.  No complaints here on the tone or style of your reply.

 

Thanks,

Ben Craig

 

From: Billy O'Neal (VC LIBS) [mailto:[hidden email]]
Sent: Monday, March 27, 2017 6:42 PM
To: Eric Fiselier <[hidden email]>; Ben Craig <[hidden email]>
Cc: via cfe-dev <[hidden email]>; [hidden email]; Stephan T. Lavavej <[hidden email]>; [hidden email]
Subject: RE: [libc++] RFC: Add Windows kernel support and partial MSVC 2015 support

 

> There are often ways to safely use STL containers w/o exceptions (Especially when custom allocators are provided).

 

Yes, but those ways generally involve going to terminate() on OOM. That's not OK in a driver.

 

From: [hidden email]
Sent: Monday, March 27, 2017 4:37 PM
To: [hidden email]
Cc: [hidden email]; [hidden email]; [hidden email]; [hidden email]; [hidden email]
Subject: Re: [libc++] RFC: Add Windows kernel support and partial MSVC 2015 support

 

 

 

 

 

On Sat, Mar 25, 2017 at 9:18 AM, Ben Craig <[hidden email]> wrote:

Abstract:
I would like to add Windows x86 and x64 kernel support to libc++.  My initial
compiler would be MSVC 2015 Update 3, though MSVC 2017 shouldn't be difficult to
add after the fact.

I would like to know if this is a port that the libc++ maintainers are willing
to accept.

 

Before responding in depth I need to ask this question: Why not use Clang/C2 or simply Clang+LLVM?

I'm assuming they don't support targeting the kernel?

 

There seem to be three separate changes going on here, and each of the changes

will benefits and challenges. For that reason I think it's best to consider them

separately. 

 

(1) Porting to MSVC

(2) Supporting the Windows Kernel C library.

(3) Implementation a Freestanding configuration of libc++

(4) Implementing the Kernel-space compatible configuration described in the original email.

 

In general I'm happy to accept specific or niche libc++ changes as long as they

aren't detrimental to the overall libc++ quality and implementation complexity.

Changes to better support (1) or (3) would be beneficial to the larger libc++ audience and I would b

happy to upstream them. However I have large concerns regarding the changes required for (2) and (4)

as I suspect they won't satisfy the above requirement.

 

For example What if the Windows Kernel C library is incomplete, or significantly different from

existing implementations, and supporting it requires re-implementing the missing parts within libc++?

These portions would be virtually untestable outside of the Windows Kernel environment and would

quickly become unmaintained. Having such code in Libc++ could quickly become a detriment.

 

My main concern with (4) is the limited feature set that has been proposed. Specifically

how it limits the libc++ internals to the same feature set and the changes that would be

needed to support and maintain it.

 

First Libc++ cannot reasonably limit itself to the proposed language and library feature set since

it must be free to use "restricted" features within the implementation of non-restricted ones whenever

it is beneficial to the implementation. The burden of maintaining a working restricted feature set could

not fall on upstream maintainers.

 

Second, the changes required to either guard restricted features using #ifdef or remove restricted features

by re-structuring the headers would be vast and would require constant maintenance. Frankly I don't

see how libc++ or its existing users could benefit from upstreaming these changes (For example adding

#ifdef guards for every usage of `operator new`).

 

I think it would be much better to get as much of libc++ compiling as possible, even if it depends on restricted

features, and then finding another mechanism to restrict the features end-users are allowed to use (Such as clang-tidy).

This would eliminate the need to restructure headers or spam out internal #ifdef guards.

 

 

This means that string, vector, and the non-array containers won't be ported.
Any class or function that requires throwing exceptions to meet their standards
required behavior will be omitted.  That rules out a lot of classes that
allocate memory.

 

There are often ways to safely use STL containers w/o exceptions (Especially when

custom allocators are provided).

 

 

Avoiding allocations allows us to sidestep one other large issue.  In the
kernel, not all memory is equal.  There are several memory pools to choose from,
but the two most common memory pools are the pageable pool and the non-pageable
pool.  There is no clear correct answer for which pool a global operator new
should use, so we simply won't require an allocating new to be present for our
implementation.  Placement new shall remain though.

 

Containers don't use new/delete directly but instead go through the specified allocator,

allowing containers to change the allocation behavior on a per-object basis. Not 

supporting containers because of global operator new's behavior seems misguided.

 

 

My employer has significant experience using C++ in the kernel.  We have been
using a modified version of STLPort for quite some time and learned a lot about
C++ library usage in the kernel, often the hard way.  The big, obvious lesson
is that a lot of the STL is either difficult, or impossible to work with when
exceptions are off and std::terminate is undesired.  There's nothing wrong with
sorting in the kernel though. 


Challenges:
* Header partitioning.

 

Libc++ prefers larger monolithic headers over many well-partitioned headers. The idea is that hitting the filesystem

multiple times is slower than processing the single include file. Any proposed changes should keep this in mind.

 

    * I don't have an exact list of what functions and classes I will be
      keeping.  I do know that some of the headers I want to bring along have
      parts that I won't be keeping.  For instance, many of the algorithms
      allocate a temporary buffer in order to meet performance requirements.

 

I don't think partitioning <algorithm> into smaller headers would be beneficial

to existing libc++ users (If that's what you're suggesting). 

 

 

    * I'll also need to figure out how to not drag along unwanted header
      dependencies.

 

I don't see how libc++ could upstream changes since it requires every header

it currently includes (modulo bugs).

 

* Testing.
    * Installing code so that it can be run in the kernel takes several seconds
      for each binary. 

    * There is no facility in the Windows kernel for running a program starting
      at "main" in the context of the kernel.

 

I suspect this will be the largest hurdle to get the test-suite running. My only

idea for handling this would be to write a script to rename main() to some

unique function name and creating a separate test-driver that calls the re-named main.

 

This could also allow us to combine multiple tests into a single executable, avoiding

the cost of installing every test manually.

 

 

 

    * The 32-bit Windows kernel requires a default calling convention of
      stdcall, but warns if "main" is not cdecl.
    * A common failure mode for libc++ tests is to crash or assert.  That
      will "blue-screen" a machine.

 

I could envision a fix which replaces `<assert.h>` when compiling the tests, but that

would be a hack. Maybe the Windows Kernel C library provides a mechanism for replacing

the assert handler?

 

 .

* MSVC compiler feature set
    * No #include_next.  I'll need to use computed include paths, like an
      STL using caveman.

 

Using hard-coded kernel paths is not a change I see being upstream-able. However

we might be able to convince MSVC to implement #include_next if we can provide

strong enough rational for why we need it.

 

 

    * Limited expression SFINAE.  Some areas in the code are straightforward
      to fix, and others are not.

 

I'm violently against working around MSVC template bugs. Every time I've seen

it done it makes the implementation worse.

 

Another point is that libc++ doesn't have an <atomic> implementation for MSVC

so adding MSVC support would required adding one.

 

 

* C-runtime
    * The Windows kernel has its own implementation of the C-runtime.  I don't
      know all the details on it.  I suspect (but don't know) that it is
      derived from Dinkumware, but I know that it is not the same C-runtime
      as used in user mode.

 

I'm very curious to know the changes required to support the kernel C runtime.

Hopefully it's similar enough to other supported C libraries that very few changes

are needed.

 

I hope my response has been helpful, and it hasn't scared you away from using libc++.

These are my initial reactions and are in no way absolute. Please let me know if I can

clarify anything or if I got anything wrong.

 

/Eric

 

 


_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev



_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [libc++] RFC: Add Windows kernel support and partial MSVC 2015 support

Richard Pennington via cfe-dev
In reply to this post by Richard Pennington via cfe-dev

Even if I did use Clang though, that would only address one of the four sub-features you identified, the port to MSVC.


Right, it would only address the MSVC port but that's at least one less work-item to reach the goal.

 

> However I have large concerns regarding the changes required for [Kernel C library] and [Kernel subset] as I suspect they won't satisfy the above requirement.

Understood.  This was feedback I was expecting.  Subsets and niche platforms are a maintenance burden.


Libc++ has support for other niche C libraries, but that support has been mostly limited to #ifdef'ing out parts of the <cfoo> headers because the underlying C library
doesn't provide it. If supporting the Kernel C library requires only similar changes then I think those would be reasonable to accept. 

The complexity for C library ports usually comes from locales and input/output functions. If those are no concern then I'm hopeful it will be easy to support
the Kernel C library.


>

> First Libc++ cannot reasonably limit itself to the proposed language and library feature set since

> it must be free to use "restricted" features within the implementation of non-restricted ones whenever

> it is beneficial to the implementation. The burden of maintaining a working restricted feature set could

> not fall on upstream maintainers.

I'm not going to make any promises... but would a public build bot address your concerns?  What if the restricted feature set got standardized as the C++2x definition of a freestanding implementation?


A public buildbot would be a requirement, and I would be happy to host it.

I would be much more comfortable maintaining a standardized definition of a "freestanding implementation", and I would support your push to fix the C++17 definition of freestanding.
 

 

I can definitely see maintaining a restricted include set to be burdensome  I will have to think on how to address that in a reasonable way.  Your clang-tidy suggestion may be useful here.

 

I don't think maintaining a restricted use of features will be particularly hard though.  For <array>, <tuple>, <utility>, and <algorithm>, do you foresee changes where an existing function doesn't throw, but starts to throw?  Or doesn't currently allocate, but starts to allocate?  If we were talking about the containers, or iostreams, or regex, I would definitely agree that the implementation flexibility is needed.  I'm not sure I agree when it's a smaller list of low level headers though.


I agree with your examples. Existing functions are unlikely to suddenly require exceptions or allocations. However it is conceivable that <tuple> or <utility> could require some internals, which themselves don't allocate or throw, that were previously only used by restricted features, or that live in restricted headers. For example libc++ often uses unique_ptr as a scope guard, but the class is primarily used to deal with dynamically memory and likely wouldn't be a part of freestanding mode. Forbidding its use in freestanding features could quickly become problematic. (Note: <algorithm> already widely depends on unique_ptr for this purpose).

I posit that the implementation requires the general freedom to use classes not supported in freestanding mode. We'll likely be able to strike a balance between what the implementation needs and what freestanding users expect, but there needs to be leeway; Leeway that restricting entire headers in freestanding mode does not provide.

 

> I think it would be much better to get as much of libc++ compiling as possible, even if it depends on restricted

> features, and then finding another mechanism to restrict the features end-users are allowed to use (Such as clang-tidy).

> This would eliminate the need to restructure headers or spam out internal #ifdef guards.

 

One thing I want to avoid is turning straightforward mistakes into errors that are even more cryptic than what C++ users have grown accustomed to.  I don't really want the failure condition of inplace_merge to be a linker error complaining about a missing operator new.  clang-tidy might help with that.  #ifdefs are a straightforward way to address that concern, though it requires a substantial amount of effort.  Would you be opposed to widespread static_asserts?


What kind of static asserts? Can you provide an example? 
 

>> * I'll also need to figure out how to not drag along unwanted header dependencies.

 

> I don't see how libc++ could upstream changes since it requires every header

> it currently includes (modulo bugs).

The straightforward (and labor intensive) way is with #ifdef _LIBCPP_HAS_NO_ALLOC

 


_LIBCPP_HAS_NO_ALLOC is exactly the invasive change I wanted to avoid. First <new> is currently a freestanding header,
and I think it should stay that way. Second I want to avoid the maintenance cost of such a restrictive
configuration. This macro would have to guard the the vast majority of the library, unlike
_LIBCPP_HAS_EXCEPTIONS and _LIBCPP_NO_RTTI which have a much smaller and more manageable scope.

If the end goal is simply to diagnose when end users use features that dynamically allocate I suspect there
are less invasive ways to do it. First there are two important things to note:

(1) Libc++ has very few naked new/delete calls. Almost all dynamic allocations go through std::allocator.
(2) std::allocator is typically used in dependent contexts (templates). 

Instead of #ifdef'ing out std::allocator and everything that uses it, we could make instantiating std::allocator::allocate
produce a diagnostic. This would make the changes needed within libc++ significantly smaller while still
producing reasonable diagnostics when users attempt to use those types.



> Using hard-coded kernel paths is not a change I see being upstream-able. However

> we might be able to convince MSVC to implement #include_next if we can provide

> strong enough rational for why we need it.

These wouldn't be hard coded paths.  The code would look something like...

#if __LIBCPP_HAS_INCLUDE_NEXT

#include_next <limits.h>

#else

#include __LIBCPP_HACK_INCLUDE_NEXT(limits.h)

#endif

__config would have the definition of __LIBCPP_HACK_INCLUDE_NEXT, and

config_site.in would have the relative path needed to glue everything together ("../../km/crt" in my case).


That sounds like a fairly reasonable solution. Although aren't macro expansions inside #include directives UB?
 

 

> I'm violently against working around MSVC template bugs. Every time I've seen

> it done it makes the implementation worse.

That's fair.  I will say that not _all_ of the workarounds make worse code,

but quite a few do.  This may end up being what forces me to MSVC 2017.

 

One example of a workaround that I don't think made things worse...

struct _LIBCPP_TYPE_VIS __check_tuple_constructor_fail {

     template <class ...>

-    static constexpr bool __enable_default() { return false; }

+    static constexpr bool __enable_default = false;


Your example uses C++14 variable templates, which I don't think you intended because MSVC 2015 doesn't support
them either.

Ironically that's a perfect example of a change that would make things drasticly worse.
First the suggested fix cannot be applied because variable templates are C++14 features.
Second we couldn't use a non-template in this case because it's important that the result is
type dependent in the non-fail cases to prevent eager SFINAE evaluation.

The <tuple> SFINAE is very complicated and it's full of subtle but important details.
It can barely manage it with a fully conforming C++11 compiler.

 

> Another point is that libc++ doesn't have an <atomic> implementation for MSVC

> so adding MSVC support would required adding one.

Yep, I've got one implemented (and untested).  This is the biggest reason why

I excluded ARM support explicitly.


Nice!
 


_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [libc++] RFC: Add Windows kernel support and partial MSVC 2015 support

Richard Pennington via cfe-dev

I can definitely see maintaining a restricted include set to be burdensome  I will have to think on how to address that in a reasonable way.  Your clang-tidy suggestion may be useful here.

I don't think maintaining a restricted use of features will be particularly hard though.  For <array>, <tuple>, <utility>, and <algorithm>, do you foresee changes where an existing function doesn't throw, but starts to throw?  Or doesn't currently allocate, but starts to allocate?  If we were talking about the containers, or iostreams, or regex, I would definitely agree that the implementation flexibility is needed.  I'm not sure I agree when it's a smaller list of low level headers though.

 

I agree with your examples. Existing functions are unlikely to suddenly require exceptions or allocations. However it is conceivable that <tuple> or <utility> could require some internals, which themselves don't allocate or throw, that were previously only used by restricted features, or that live in restricted headers. For example libc++ often uses unique_ptr as a scope guard, but the class is primarily used to deal with dynamically memory and likely wouldn't be a part of freestanding mode. Forbidding its use in freestanding features could quickly become problematic. (Note: <algorithm> already widely depends on unique_ptr for this purpose).

 

I posit that the implementation requires the general freedom to use classes not supported in freestanding mode. We'll likely be able to strike a balance between what the implementation needs and what freestanding users expect, but there needs to be leeway; Leeway that restricting entire headers in freestanding mode does not provide.

 

Unique_ptr is a strange beast for exactly the reasons you describe.  I’m tempted to include it in a freestanding definition, but it would definitely feel odd pulling it along, but not taking make_unique, and not allowing the default deleter.  I think I also agree that restricting by header is too blunt of an instrument to validate the freestanding restrictions.

 

> I think it would be much better to get as much of libc++ compiling as possible, even if it depends on restricted

> features, and then finding another mechanism to restrict the features end-users are allowed to use (Such as clang-tidy).

> This would eliminate the need to restructure headers or spam out internal #ifdef guards.

 

One thing I want to avoid is turning straightforward mistakes into errors that are even more cryptic than what C++ users have grown accustomed to.  I don't really want the failure condition of inplace_merge to be a linker error complaining about a missing operator new.  clang-tidy might help with that.  #ifdefs are a straightforward way to address that concern, though it requires a substantial amount of effort.  Would you be opposed to widespread static_asserts?

 

What kind of static asserts? Can you provide an example? 

What I had in mind here was for the first line of inplace_merge (and other allocating algorithms) to be something along the lines of static_assert(!_LIBCPP_HAS_NO_ALLOC).  I think you had a much better suggestion though…

 

>> * I'll also need to figure out how to not drag along unwanted header dependencies.

 

> I don't see how libc++ could upstream changes since it requires every header

> it currently includes (modulo bugs).

The straightforward (and labor intensive) way is with #ifdef _LIBCPP_HAS_NO_ALLOC

 

 

_LIBCPP_HAS_NO_ALLOC is exactly the invasive change I wanted to avoid. First <new> is currently a freestanding header,

and I think it should stay that way. Second I want to avoid the maintenance cost of such a restrictive

configuration. This macro would have to guard the the vast majority of the library, unlike

_LIBCPP_HAS_EXCEPTIONS and _LIBCPP_NO_RTTI which have a much smaller and more manageable scope.

 

If the end goal is simply to diagnose when end users use features that dynamically allocate I suspect there

are less invasive ways to do it. First there are two important things to note:

 

(1) Libc++ has very few naked new/delete calls. Almost all dynamic allocations go through std::allocator.

(2) std::allocator is typically used in dependent contexts (templates). 

 

Instead of #ifdef'ing out std::allocator and everything that uses it, we could make instantiating std::allocator::allocate

produce a diagnostic. This would make the changes needed within libc++ significantly smaller while still

producing reasonable diagnostics when users attempt to use those types.

 

Putting a static assert (or some other kind of diagnostic) in std::allocator::allocate seems like an excellent idea to me.  I’d also want to hunt down the other places where raw new is used, but I think this approach has a lot of promise.  We can document what we officially support, but say that other things might work.  If someone instantiates code that hits std::allocator, they will get a static assert.  If they write their own code that calls new or delete, they will get a linker error (in the Windows Kernel at least).  If they write their own allocator that uses ExAllocatePoolWithTag, or malloc, or whatnot, then that’s fine, they will just need to be super careful that they don’t shoot themselves in the foot with std::list gotchas.

 

> Using hard-coded kernel paths is not a change I see being upstream-able. However

> we might be able to convince MSVC to implement #include_next if we can provide

> strong enough rational for why we need it.

These wouldn't be hard coded paths.  The code would look something like...

#if __LIBCPP_HAS_INCLUDE_NEXT

#include_next <limits.h>

#else

#include __LIBCPP_HACK_INCLUDE_NEXT(limits.h)

#endif

__config would have the definition of __LIBCPP_HACK_INCLUDE_NEXT, and

config_site.in would have the relative path needed to glue everything together ("../../km/crt" in my case).

 

That sounds like a fairly reasonable solution. Although aren't macro expansions inside #include directives UB?

 

The GCC docs claim it is implementation-defined behavior ( https://gcc.gnu.org/onlinedocs/cpp/Computed-Includes.html ).  I haven’t done the research in the spec to know for sure.  STLPort does it though, and the computed include hackery there seems to work fine for lots of versions of GCC, Clang, MSVC, and I think even Intel’s compiler.

 

 

> I'm violently against working around MSVC template bugs. Every time I've seen

> it done it makes the implementation worse.

That's fair.  I will say that not _all_ of the workarounds make worse code,

but quite a few do.  This may end up being what forces me to MSVC 2017.

 

One example of a workaround that I don't think made things worse...

struct _LIBCPP_TYPE_VIS __check_tuple_constructor_fail {

     template <class ...>

-    static constexpr bool __enable_default() { return false; }

+    static constexpr bool __enable_default = false;

 

Your example uses C++14 variable templates, which I don't think you intended because MSVC 2015 doesn't support

them either.

 

Ironically that's a perfect example of a change that would make things drasticly worse.

First the suggested fix cannot be applied because variable templates are C++14 features.

Second we couldn't use a non-template in this case because it's important that the result is

type dependent in the non-fail cases to prevent eager SFINAE evaluation.

 

The <tuple> SFINAE is very complicated and it's full of subtle but important details.

It can barely manage it with a fully conforming C++11 compiler.

 

I almost didn’t include an example at all, because this was going to be the inevitable outcome, especially since I picked something involving std::tuple.

 


_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [libc++] RFC: Add Windows kernel support and partial MSVC 2015 support

Richard Pennington via cfe-dev

> MSVC 2015 doesn't support them either.

 

MSVC added variable templates in 2015 Update 2.

 

From: [hidden email]
Sent: Wednesday, March 29, 2017 3:38 PM
To: [hidden email]
Cc: [hidden email]; [hidden email]; [hidden email]; [hidden email]; [hidden email]
Subject: RE: [libc++] RFC: Add Windows kernel support and partial MSVC 2015 support

 

I can definitely see maintaining a restricted include set to be burdensome  I will have to think on how to address that in a reasonable way.  Your clang-tidy suggestion may be useful here.

I don't think maintaining a restricted use of features will be particularly hard though.  For <array>, <tuple>, <utility>, and <algorithm>, do you foresee changes where an existing function doesn't throw, but starts to throw?  Or doesn't currently allocate, but starts to allocate?  If we were talking about the containers, or iostreams, or regex, I would definitely agree that the implementation flexibility is needed.  I'm not sure I agree when it's a smaller list of low level headers though.

 

I agree with your examples. Existing functions are unlikely to suddenly require exceptions or allocations. However it is conceivable that <tuple> or <utility> could require some internals, which themselves don't allocate or throw, that were previously only used by restricted features, or that live in restricted headers. For example libc++ often uses unique_ptr as a scope guard, but the class is primarily used to deal with dynamically memory and likely wouldn't be a part of freestanding mode. Forbidding its use in freestanding features could quickly become problematic. (Note: <algorithm> already widely depends on unique_ptr for this purpose).

 

I posit that the implementation requires the general freedom to use classes not supported in freestanding mode. We'll likely be able to strike a balance between what the implementation needs and what freestanding users expect, but there needs to be leeway; Leeway that restricting entire headers in freestanding mode does not provide.

 

Unique_ptr is a strange beast for exactly the reasons you describe.  I’m tempted to include it in a freestanding definition, but it would definitely feel odd pulling it along, but not taking make_unique, and not allowing the default deleter.  I think I also agree that restricting by header is too blunt of an instrument to validate the freestanding restrictions.

 

> I think it would be much better to get as much of libc++ compiling as possible, even if it depends on restricted

> features, and then finding another mechanism to restrict the features end-users are allowed to use (Such as clang-tidy).

> This would eliminate the need to restructure headers or spam out internal #ifdef guards.

 

One thing I want to avoid is turning straightforward mistakes into errors that are even more cryptic than what C++ users have grown accustomed to.  I don't really want the failure condition of inplace_merge to be a linker error complaining about a missing operator new.  clang-tidy might help with that.  #ifdefs are a straightforward way to address that concern, though it requires a substantial amount of effort.  Would you be opposed to widespread static_asserts?

 

What kind of static asserts? Can you provide an example? 

What I had in mind here was for the first line of inplace_merge (and other allocating algorithms) to be something along the lines of static_assert(!_LIBCPP_HAS_NO_ALLOC).  I think you had a much better suggestion though…

 

>> * I'll also need to figure out how to not drag along unwanted header dependencies.

 

> I don't see how libc++ could upstream changes since it requires every header

> it currently includes (modulo bugs).

The straightforward (and labor intensive) way is with #ifdef _LIBCPP_HAS_NO_ALLOC

 

 

_LIBCPP_HAS_NO_ALLOC is exactly the invasive change I wanted to avoid. First <new> is currently a freestanding header,

and I think it should stay that way. Second I want to avoid the maintenance cost of such a restrictive

configuration. This macro would have to guard the the vast majority of the library, unlike

_LIBCPP_HAS_EXCEPTIONS and _LIBCPP_NO_RTTI which have a much smaller and more manageable scope.

 

If the end goal is simply to diagnose when end users use features that dynamically allocate I suspect there

are less invasive ways to do it. First there are two important things to note:

 

(1) Libc++ has very few naked new/delete calls. Almost all dynamic allocations go through std::allocator.

(2) std::allocator is typically used in dependent contexts (templates). 

 

Instead of #ifdef'ing out std::allocator and everything that uses it, we could make instantiating std::allocator::allocate

produce a diagnostic. This would make the changes needed within libc++ significantly smaller while still

producing reasonable diagnostics when users attempt to use those types.

 

Putting a static assert (or some other kind of diagnostic) in std::allocator::allocate seems like an excellent idea to me.  I’d also want to hunt down the other places where raw new is used, but I think this approach has a lot of promise.  We can document what we officially support, but say that other things might work.  If someone instantiates code that hits std::allocator, they will get a static assert.  If they write their own code that calls new or delete, they will get a linker error (in the Windows Kernel at least).  If they write their own allocator that uses ExAllocatePoolWithTag, or malloc, or whatnot, then that’s fine, they will just need to be super careful that they don’t shoot themselves in the foot with std::list gotchas.

 

> Using hard-coded kernel paths is not a change I see being upstream-able. However

> we might be able to convince MSVC to implement #include_next if we can provide

> strong enough rational for why we need it.

These wouldn't be hard coded paths.  The code would look something like...

#if __LIBCPP_HAS_INCLUDE_NEXT

#include_next <limits.h>

#else

#include __LIBCPP_HACK_INCLUDE_NEXT(limits.h)

#endif

__config would have the definition of __LIBCPP_HACK_INCLUDE_NEXT, and

config_site.in would have the relative path needed to glue everything together ("../../km/crt" in my case).

 

That sounds like a fairly reasonable solution. Although aren't macro expansions inside #include directives UB?

 

The GCC docs claim it is implementation-defined behavior ( https://gcc.gnu.org/onlinedocs/cpp/Computed-Includes.html ).  I haven’t done the research in the spec to know for sure.  STLPort does it though, and the computed include hackery there seems to work fine for lots of versions of GCC, Clang, MSVC, and I think even Intel’s compiler.

 

 

> I'm violently against working around MSVC template bugs. Every time I've seen

> it done it makes the implementation worse.

That's fair.  I will say that not _all_ of the workarounds make worse code,

but quite a few do.  This may end up being what forces me to MSVC 2017.

 

One example of a workaround that I don't think made things worse...

struct _LIBCPP_TYPE_VIS __check_tuple_constructor_fail {

     template <class ...>

-    static constexpr bool __enable_default() { return false; }

+    static constexpr bool __enable_default = false;

 

Your example uses C++14 variable templates, which I don't think you intended because MSVC 2015 doesn't support

them either.

 

Ironically that's a perfect example of a change that would make things drasticly worse.

First the suggested fix cannot be applied because variable templates are C++14 features.

Second we couldn't use a non-template in this case because it's important that the result is

type dependent in the non-fail cases to prevent eager SFINAE evaluation.

 

The <tuple> SFINAE is very complicated and it's full of subtle but important details.

It can barely manage it with a fully conforming C++11 compiler.

 

I almost didn’t include an example at all, because this was going to be the inevitable outcome, especially since I picked something involving std::tuple.

 

 


_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [libc++] RFC: Add Windows kernel support and partial MSVC 2015 support

Richard Pennington via cfe-dev


On Wed, Mar 29, 2017 at 4:44 PM, Billy O'Neal (VC LIBS) <[hidden email]> wrote:

> MSVC 2015 doesn't support them either.

 

MSVC added variable templates in 2015 Update 2.


Sorry my mistake. I'm still not quite sure what MSVC 2015 issue was being demonstrated though.
 

 

From: [hidden email]
Sent: Wednesday, March 29, 2017 3:38 PM
To: [hidden email]
Cc: [hidden email]; [hidden email]; [hidden email]; [hidden email]; [hidden email]
Subject: RE: [libc++] RFC: Add Windows kernel support and partial MSVC 2015 support

 

I can definitely see maintaining a restricted include set to be burdensome  I will have to think on how to address that in a reasonable way.  Your clang-tidy suggestion may be useful here.

I don't think maintaining a restricted use of features will be particularly hard though.  For <array>, <tuple>, <utility>, and <algorithm>, do you foresee changes where an existing function doesn't throw, but starts to throw?  Or doesn't currently allocate, but starts to allocate?  If we were talking about the containers, or iostreams, or regex, I would definitely agree that the implementation flexibility is needed.  I'm not sure I agree when it's a smaller list of low level headers though.

 

I agree with your examples. Existing functions are unlikely to suddenly require exceptions or allocations. However it is conceivable that <tuple> or <utility> could require some internals, which themselves don't allocate or throw, that were previously only used by restricted features, or that live in restricted headers. For example libc++ often uses unique_ptr as a scope guard, but the class is primarily used to deal with dynamically memory and likely wouldn't be a part of freestanding mode. Forbidding its use in freestanding features could quickly become problematic. (Note: <algorithm> already widely depends on unique_ptr for this purpose).

 

I posit that the implementation requires the general freedom to use classes not supported in freestanding mode. We'll likely be able to strike a balance between what the implementation needs and what freestanding users expect, but there needs to be leeway; Leeway that restricting entire headers in freestanding mode does not provide.

 

Unique_ptr is a strange beast for exactly the reasons you describe.  I’m tempted to include it in a freestanding definition, but it would definitely feel odd pulling it along, but not taking make_unique, and not allowing the default deleter.  I think I also agree that restricting by header is too blunt of an instrument to validate the freestanding restrictions.

 

> I think it would be much better to get as much of libc++ compiling as possible, even if it depends on restricted

> features, and then finding another mechanism to restrict the features end-users are allowed to use (Such as clang-tidy).

> This would eliminate the need to restructure headers or spam out internal #ifdef guards.

 

One thing I want to avoid is turning straightforward mistakes into errors that are even more cryptic than what C++ users have grown accustomed to.  I don't really want the failure condition of inplace_merge to be a linker error complaining about a missing operator new.  clang-tidy might help with that.  #ifdefs are a straightforward way to address that concern, though it requires a substantial amount of effort.  Would you be opposed to widespread static_asserts?

 

What kind of static asserts? Can you provide an example? 

What I had in mind here was for the first line of inplace_merge (and other allocating algorithms) to be something along the lines of static_assert(!_LIBCPP_HAS_NO_ALLOC).  I think you had a much better suggestion though…

 

>> * I'll also need to figure out how to not drag along unwanted header dependencies.

 

> I don't see how libc++ could upstream changes since it requires every header

> it currently includes (modulo bugs).

The straightforward (and labor intensive) way is with #ifdef _LIBCPP_HAS_NO_ALLOC

 

 

_LIBCPP_HAS_NO_ALLOC is exactly the invasive change I wanted to avoid. First <new> is currently a freestanding header,

and I think it should stay that way. Second I want to avoid the maintenance cost of such a restrictive

configuration. This macro would have to guard the the vast majority of the library, unlike

_LIBCPP_HAS_EXCEPTIONS and _LIBCPP_NO_RTTI which have a much smaller and more manageable scope.

 

If the end goal is simply to diagnose when end users use features that dynamically allocate I suspect there

are less invasive ways to do it. First there are two important things to note:

 

(1) Libc++ has very few naked new/delete calls. Almost all dynamic allocations go through std::allocator.

(2) std::allocator is typically used in dependent contexts (templates). 

 

Instead of #ifdef'ing out std::allocator and everything that uses it, we could make instantiating std::allocator::allocate

produce a diagnostic. This would make the changes needed within libc++ significantly smaller while still

producing reasonable diagnostics when users attempt to use those types.

 

Putting a static assert (or some other kind of diagnostic) in std::allocator::allocate seems like an excellent idea to me.  I’d also want to hunt down the other places where raw new is used, but I think this approach has a lot of promise.  We can document what we officially support, but say that other things might work.  If someone instantiates code that hits std::allocator, they will get a static assert.  If they write their own code that calls new or delete, they will get a linker error (in the Windows Kernel at least).  If they write their own allocator that uses ExAllocatePoolWithTag, or malloc, or whatnot, then that’s fine, they will just need to be super careful that they don’t shoot themselves in the foot with std::list gotchas.

 

> Using hard-coded kernel paths is not a change I see being upstream-able. However

> we might be able to convince MSVC to implement #include_next if we can provide

> strong enough rational for why we need it.

These wouldn't be hard coded paths.  The code would look something like...

#if __LIBCPP_HAS_INCLUDE_NEXT

#include_next <limits.h>

#else

#include __LIBCPP_HACK_INCLUDE_NEXT(limits.h)

#endif

__config would have the definition of __LIBCPP_HACK_INCLUDE_NEXT, and

config_site.in would have the relative path needed to glue everything together ("../../km/crt" in my case).

 

That sounds like a fairly reasonable solution. Although aren't macro expansions inside #include directives UB?

 

The GCC docs claim it is implementation-defined behavior ( https://gcc.gnu.org/onlinedocs/cpp/Computed-Includes.html ).  I haven’t done the research in the spec to know for sure.  STLPort does it though, and the computed include hackery there seems to work fine for lots of versions of GCC, Clang, MSVC, and I think even Intel’s compiler.

 

 

> I'm violently against working around MSVC template bugs. Every time I've seen

> it done it makes the implementation worse.

That's fair.  I will say that not _all_ of the workarounds make worse code,

but quite a few do.  This may end up being what forces me to MSVC 2017.

 

One example of a workaround that I don't think made things worse...

struct _LIBCPP_TYPE_VIS __check_tuple_constructor_fail {

     template <class ...>

-    static constexpr bool __enable_default() { return false; }

+    static constexpr bool __enable_default = false;

 

Your example uses C++14 variable templates, which I don't think you intended because MSVC 2015 doesn't support

them either.

 

Ironically that's a perfect example of a change that would make things drasticly worse.

First the suggested fix cannot be applied because variable templates are C++14 features.

Second we couldn't use a non-template in this case because it's important that the result is

type dependent in the non-fail cases to prevent eager SFINAE evaluation.

 

The <tuple> SFINAE is very complicated and it's full of subtle but important details.

It can barely manage it with a fully conforming C++11 compiler.

 

I almost didn’t include an example at all, because this was going to be the inevitable outcome, especially since I picked something involving std::tuple.

 

 



_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [libc++] RFC: Add Windows kernel support and partial MSVC 2015 support

Richard Pennington via cfe-dev
In reply to this post by Richard Pennington via cfe-dev

What kind of static asserts? Can you provide an example? 

What I had in mind here was for the first line of inplace_merge (and other allocating algorithms) to be something along the lines of static_assert(!_LIBCPP_HAS_NO_ALLOC).  I think you had a much better suggestion though…


The algorithms all use std::get_temporary_buffer to allocate memory, and IIRC all of the algorithms should still work if std::get_temporary_buffer returns no additional memory. In this particular case I think it would be better to patch get_temporary_buffer and leave the algorithms as-is.

All other allocation cases that need static_assert should be handled by the std::allocation suggestion.
 

>> * I'll also need to figure out how to not drag along unwanted header dependencies.

 

> I don't see how libc++ could upstream changes since it requires every header

> it currently includes (modulo bugs).

The straightforward (and labor intensive) way is with #ifdef _LIBCPP_HAS_NO_ALLOC

 

 

_LIBCPP_HAS_NO_ALLOC is exactly the invasive change I wanted to avoid. First <new> is currently a freestanding header,

and I think it should stay that way. Second I want to avoid the maintenance cost of such a restrictive

configuration. This macro would have to guard the the vast majority of the library, unlike

_LIBCPP_HAS_EXCEPTIONS and _LIBCPP_NO_RTTI which have a much smaller and more manageable scope.

 

If the end goal is simply to diagnose when end users use features that dynamically allocate I suspect there

are less invasive ways to do it. First there are two important things to note:

 

(1) Libc++ has very few naked new/delete calls. Almost all dynamic allocations go through std::allocator.

(2) std::allocator is typically used in dependent contexts (templates). 

 

Instead of #ifdef'ing out std::allocator and everything that uses it, we could make instantiating std::allocator::allocate

produce a diagnostic. This would make the changes needed within libc++ significantly smaller while still

producing reasonable diagnostics when users attempt to use those types.

 

Putting a static assert (or some other kind of diagnostic) in std::allocator::allocate seems like an excellent idea to me.  I’d also want to hunt down the other places where raw new is used, but I think this approach has a lot of promise. 


I think most usages of raw new could be made to use std::allocator instead, so they wouldn't have to be handled separately. 
 

We can document what we officially support, but say that other things might work.  If someone instantiates code that hits std::allocator, they will get a static assert.  If they write their own code that calls new or delete, they will get a linker error (in the Windows Kernel at least). 


If me manage to make libc++ always use std::allocator instead of raw new then it might be feasible to mark the new/delete definitions as unavailable, since they would only have one caller internally. This would allow usages to be diagnosed at compile-time instead of link-time. It would also ensure that replacement definitions of new/delete are not found at link time.
 


_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Loading...