How to package library TSes

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

How to package library TSes

Marshall Clow
There's a lot of changes coming to the standard library. The standards committee is looking at (and in several cases, approved) proposals for networking, ranges, concurrency, parallelism, transactional memory, file-system, and concepts. Each of these will require changes to the standard library.

Some (most) of them are "Technical Specifications", which means that they are not officially part of the standard. The goal for then TS-es is to give implementors and users experience with the features, so that the standards committee can get feedback about when (and if) they should be adopted into the standard.

The other part of this is that the TS-es *change*. People give feedback, and the committee updates the proposals, and the code changes. There's a reason that most of them live in the namespace "std::experimental".

Some of these proposals can be implemented as templates that live in a header file, and require no new functionality from the dynamic library libc++.dylib. Most of them, however, have a separately compiled part, and that is the purpose of this email.

How should these parts be delivered? 

The choices, I see, are:
* As part of the libc++ dynamic library.
* As a set of statically linkable object libraries.
* As a set of dynamic libraries.
* Something else

The first option, it seems to me, is a non-starter. There are some systems (FreeSBD, Mac OS X) that ship libc++ as the system runtime library, and replacing that library (on the fly) is a easy way to render your system inoperative. 

On the other hand, having the vendors ship updated libraries a few times/year (or less) is a great way to slow down the entire process.

So, I'm asking - what do people want?  (and why)

-- Marshall


_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: How to package library TSes

David Chisnall-4
Hi Marshall,

For some background: We ship libc++ in both the base system and in ports.  Some things use the version in ports, but we expect it to be a drop-in replacement (i.e. if something links libc++ and works with the one in base, then it must also work with the one in ports).  Mostly the one in ports is used by things that require C++14 on systems that ship a libc++ that’s C++11-only.

If we’re going to ship them in the FreeBSD base system at all, then we need to be able to guarantee ABI (and API) compatibility over the lifetime of a FreeBSD release (5 years).  This means that we’d want to make sure, at the very least, they they’re in versioned namespaces and that people can opt in to newer ones if they want, but still have the older ones available.

As to which library it goes in, having one library is attractive even if it means that it’s going to grow a load of compat code over time.  For FreeBSD 12, we’d happily turn off the compat code with FreeBSD 11 (and provide a compat version with it enabled) and some people building embedded systems may wish to turn it off in their builds anyway.  Having them as separate libraries doesn’t really address this problem.  Having them as static libraries is likely to cause issues when two or more shared libraries link against them.

David

> On 21 Jul 2015, at 19:38, Marshall Clow <[hidden email]> wrote:
>
> There's a lot of changes coming to the standard library. The standards committee is looking at (and in several cases, approved) proposals for networking, ranges, concurrency, parallelism, transactional memory, file-system, and concepts. Each of these will require changes to the standard library.
>
> Some (most) of them are "Technical Specifications", which means that they are not officially part of the standard. The goal for then TS-es is to give implementors and users experience with the features, so that the standards committee can get feedback about when (and if) they should be adopted into the standard.
>
> The other part of this is that the TS-es *change*. People give feedback, and the committee updates the proposals, and the code changes. There's a reason that most of them live in the namespace "std::experimental".
>
> Some of these proposals can be implemented as templates that live in a header file, and require no new functionality from the dynamic library libc++.dylib. Most of them, however, have a separately compiled part, and that is the purpose of this email.
>
> How should these parts be delivered?
>
> The choices, I see, are:
> * As part of the libc++ dynamic library.
> * As a set of statically linkable object libraries.
> * As a set of dynamic libraries.
> * Something else
>
> The first option, it seems to me, is a non-starter. There are some systems (FreeSBD, Mac OS X) that ship libc++ as the system runtime library, and replacing that library (on the fly) is a easy way to render your system inoperative.
>
> On the other hand, having the vendors ship updated libraries a few times/year (or less) is a great way to slow down the entire process.
>
> So, I'm asking - what do people want?  (and why)
>
> -- Marshall
>
> _______________________________________________
> cfe-dev mailing list
> [hidden email]
> http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev


_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: How to package library TSes

Marshall Clow
On Wed, Jul 22, 2015 at 1:55 AM, David Chisnall <[hidden email]> wrote:
Hi Marshall,

For some background: We ship libc++ in both the base system and in ports.  Some things use the version in ports, but we expect it to be a drop-in replacement (i.e. if something links libc++ and works with the one in base, then it must also work with the one in ports).  Mostly the one in ports is used by things that require C++14 on systems that ship a libc++ that’s C++11-only.

OK - thanks for the background.
 
If we’re going to ship them in the FreeBSD base system at all, then we need to be able to guarantee ABI (and API) compatibility over the lifetime of a FreeBSD release (5 years).  This means that we’d want to make sure, at the very least, they they’re in versioned namespaces and that people can opt in to newer ones if they want, but still have the older ones available.

Yeah, I can't do that for the TSes.

They're "experimental". That means that they can (and will) change - and the changes are not under my control. 

I (and the others who work on libc++) try very hard to maintain ABI compatibility for the things in the standard; and I think we do a pretty good job. But it's a fair amount of work, and I don't want to take on the burden of doing that for stuff that the committee has declared as "not final".

That's one of the reasons that I started this email thread - to figure out a good way to hand this.

-- Marshall


_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: How to package library TSes

Dan Albert

For cases like the base library that do require that kind of absolute compatibility maybe the best solution is to just not include any of the experimental features. If I understand David correctly the one in ports could include the experimental features for the users that want it.

On Jul 22, 2015 8:40 PM, "Marshall Clow" <[hidden email]> wrote:
On Wed, Jul 22, 2015 at 1:55 AM, David Chisnall <[hidden email]> wrote:
Hi Marshall,

For some background: We ship libc++ in both the base system and in ports.  Some things use the version in ports, but we expect it to be a drop-in replacement (i.e. if something links libc++ and works with the one in base, then it must also work with the one in ports).  Mostly the one in ports is used by things that require C++14 on systems that ship a libc++ that’s C++11-only.

OK - thanks for the background.
 
If we’re going to ship them in the FreeBSD base system at all, then we need to be able to guarantee ABI (and API) compatibility over the lifetime of a FreeBSD release (5 years).  This means that we’d want to make sure, at the very least, they they’re in versioned namespaces and that people can opt in to newer ones if they want, but still have the older ones available.

Yeah, I can't do that for the TSes.

They're "experimental". That means that they can (and will) change - and the changes are not under my control. 

I (and the others who work on libc++) try very hard to maintain ABI compatibility for the things in the standard; and I think we do a pretty good job. But it's a fair amount of work, and I don't want to take on the burden of doing that for stuff that the committee has declared as "not final".

That's one of the reasons that I started this email thread - to figure out a good way to hand this.

-- Marshall


_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev


_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: How to package library TSes

Marshall Clow
On Wed, Jul 22, 2015 at 8:54 PM, Dan Albert <[hidden email]> wrote:

For cases like the base library that do require that kind of absolute compatibility maybe the best solution is to just not include any of the experimental features. If I understand David correctly the one in ports could include the experimental features for the users that want it.

I think that that could work for FreeBSD.

-- Marshall

_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: How to package library TSes

Marshall Clow
On Wed, Jul 22, 2015 at 9:43 PM, Marshall Clow <[hidden email]> wrote:
On Wed, Jul 22, 2015 at 8:54 PM, Dan Albert <[hidden email]> wrote:

For cases like the base library that do require that kind of absolute compatibility maybe the best solution is to just not include any of the experimental features. If I understand David correctly the one in ports could include the experimental features for the users that want it.

I think that that could work for FreeBSD.


Followup - some of the TSes are almost completely implemented in headers, and do not require any changes to the dylib.

-- Marshall



_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: How to package library TSes

David Chisnall-4
On 23 Jul 2015, at 05:45, Marshall Clow <[hidden email]> wrote:
>
> On Wed, Jul 22, 2015 at 9:43 PM, Marshall Clow <[hidden email]> wrote:
> On Wed, Jul 22, 2015 at 8:54 PM, Dan Albert <[hidden email]> wrote:
> For cases like the base library that do require that kind of absolute compatibility maybe the best solution is to just not include any of the experimental features. If I understand David correctly the one in ports could include the experimental features for the users that want it.
>
> I think that that could work for FreeBSD.
>
>
> Followup - some of the TSes are almost completely implemented in headers, and do not require any changes to the dylib.

With regard to ABI compatibility, there are two issues:

1) Library A uses v1, library B uses v2, program links to library A and library B.

2) Library A uses v1, library B uses v2, library A exposes v1 types at library API boundaries.

I think the second case is not worth considering - if you use experimental features in your public interface, then you expect stuff to break.  The first case would be nice to support though.  As long as any ABI-breaking change has a new namespace version, then we can provide a library that contains the old and new versions.

This is particularly important if the ABI changes are the result of API changes - it’s trivial for us to recompile everything (we rebuild all packages roughly once a day anyway), but it’s much harder to make every third-party package update their API usage (even getting all of the tr1:: users to use the C++11 versions for simple things like unordered_map and unordered_set in libc++ required patching a thousand or so ports, and I’d like to not have to do that again).

If we aim not to ship the experimental stuff in the base system, and have a libc++-experimental.so provided by a port that contains the TS stuff, then that means that we don’t have to *guarantee* ABI stability, but we’re still going to have pain if we aren’t able to support multiple versions of the APIs.

David


_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: How to package library TSes

Ted Kremenek
In reply to this post by Marshall Clow

On Jul 22, 2015, at 8:38 PM, Marshall Clow <[hidden email]> wrote:

On Wed, Jul 22, 2015 at 1:55 AM, David Chisnall <[hidden email]> wrote:
Hi Marshall,

For some background: We ship libc++ in both the base system and in ports.  Some things use the version in ports, but we expect it to be a drop-in replacement (i.e. if something links libc++ and works with the one in base, then it must also work with the one in ports).  Mostly the one in ports is used by things that require C++14 on systems that ship a libc++ that’s C++11-only.

OK - thanks for the background.
 
If we’re going to ship them in the FreeBSD base system at all, then we need to be able to guarantee ABI (and API) compatibility over the lifetime of a FreeBSD release (5 years).  This means that we’d want to make sure, at the very least, they they’re in versioned namespaces and that people can opt in to newer ones if they want, but still have the older ones available.

Yeah, I can't do that for the TSes.

They're "experimental". That means that they can (and will) change - and the changes are not under my control. 

I (and the others who work on libc++) try very hard to maintain ABI compatibility for the things in the standard; and I think we do a pretty good job. But it's a fair amount of work, and I don't want to take on the burden of doing that for stuff that the committee has declared as "not final".

That's one of the reasons that I started this email thread - to figure out a good way to hand this.

-- Marshall

_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev

I just noticed this thread, so I apologize for being a little late.

On OS X, we can do something similar to how we are handling the Swift runtime:

- Build the TSes in a separate dylib, and distribute that dylib (with its headers) with the compiler.

- When building an app that uses a TS, embed the dylib for the TS inside the app bundle itself.

- The embedded dylib would need to be ABI compatible with the core libc++ dylib on the host system, but that’s as far as it would need to go.  As the TSes evolve, the dylib would change and the app would need to get rebuilt anyway when it picks up a new compiler.

The major downside of this approach is that it creates an expectation of backwards deployment to earlier OSes, even if the TS makes it into the core libc++ dylib on the host system at a later stage.

_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: How to package library TSes

David Chisnall-4

> On 28 Jul 2015, at 00:51, Ted Kremenek <[hidden email]> wrote:
>
> On OS X, we can do something similar to how we are handling the Swift runtime:
>
> - Build the TSes in a separate dylib, and distribute that dylib (with its headers) with the compiler.
>
> - When building an app that uses a TS, embed the dylib for the TS inside the app bundle itself.
>
> - The embedded dylib would need to be ABI compatible with the core libc++ dylib on the host system, but that’s as far as it would need to go.  As the TSes evolve, the dylib would change and the app would need to get rebuilt anyway when it picks up a new compiler.

I’m assuming that this means that you don’t support using the TSs inside third party binary-only frameworks?  Or does the two-level linking namespace thing manage this for things that don’t expose TS types in headers?

> The major downside of this approach is that it creates an expectation of backwards deployment to earlier OSes, even if the TS makes it into the core libc++ dylib on the host system at a later stage.

Wouldn’t weak linking address or filter libraries address this?

David



_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: How to package library TSes

Ted Kremenek

On Jul 28, 2015, at 12:39 AM, David Chisnall <[hidden email]> wrote:



On 28 Jul 2015, at 00:51, Ted Kremenek <[hidden email]> wrote:

On OS X, we can do something similar to how we are handling the Swift runtime:

- Build the TSes in a separate dylib, and distribute that dylib (with its headers) with the compiler.

- When building an app that uses a TS, embed the dylib for the TS inside the app bundle itself.

- The embedded dylib would need to be ABI compatible with the core libc++ dylib on the host system, but that’s as far as it would need to go.  As the TSes evolve, the dylib would change and the app would need to get rebuilt anyway when it picks up a new compiler.

I’m assuming that this means that you don’t support using the TSs inside third party binary-only frameworks? Or does the two-level linking namespace thing manage this for things that don’t expose TS types in headers?

Sorry, let me elaborate.

I think the easiest way to think about this is imagine the TS is packaged in its own 3rd party framework.  That framework could be used by any other framework in a users application, or even other 3rd party frameworks.  On Darwin, that TS framework could be embedded inside the app, and used by both the application and 3rd party framework.

Such a framework would suffer from the same problems as any C++ library.  The ABI/API might not be stable, and thus likely only could support a model where the 3rd party frameworks and application that use the TS framework are all built with the same compiler (assuming the TS library ships with the compiler).


The major downside of this approach is that it creates an expectation of backwards deployment to earlier OSes, even if the TS makes it into the core libc++ dylib on the host system at a later stage.

Wouldn’t weak linking address or filter libraries address this?

I’m not certain what you mean by using weak linking.  Here’s how I see it:

(1) The TS framework ships inside the app, not with the OS.  Let’s say this is OS version 1.

(2) The APIs move to the core libc++ library at a later OS release.  Let’s say this is OS  version 2.

How is the application build if it wants to deploy to running on the earlier OS release (i.e., OS version 1)?  There is no copy of the library on the OS, so all it could do is continue to embed it if it wanted to deploy to the earlier OS release.  On the Mac, many mainstream C++ apps deploy back several OS releases (for many years), and thus bundling the library with the compiler means that library would need to ship with the compiler long after the core functionality of that library migrates to the main libc++ library.

_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev