[RFC] Open sourcing and contributing TAPI back to the LLVM community

classic Classic list List threaded Threaded
26 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Re: [llvm-dev] [RFC] Open sourcing and contributing TAPI back to the LLVM community

Vassil Vassilev via cfe-dev
Was there any progress in the upstreaming effort? I'd be interested in having lld be able to link against tbd files, and I think it'd be cool if libtool -static could write tbd files (similar to thin archives on linux) since that should make archiving much faster.

Juergen, maybe uploading your initial patch to phabricator instead of attaching might get more traction?

On Wed, Apr 25, 2018 at 8:53 AM Juergen Ributzka via cfe-dev <[hidden email]> wrote:
The code on opensource.apple.com is the minimal code needed for libtapi to read TBD files with ld64. The code I pushed to GitHub on the other side includes the full TAPI tool source code. If you check the code you will see that I already use the LLVM license for everything, because the goal has always been to contribute this back to the community.

I attached an initial patch for libtapi and nm support to my original email. I also have some updated patches for that too, but somehow got derailed into "lets rewrite libobject" discussions.

On Tue, Apr 10, 2018 at 4:15 PM, John Ericson via llvm-dev <[hidden email]> wrote:

That sounds great to me, thanks Jake. I'm not Jurgen either, of course, but I'm happy to assist you if he is unavailable. I'm not also not qualified to audit the license, but do note Apple formally also released some code at https://opensource.apple.com/tarballs/tapi/. If there's anything else I can do to help, let me know.

Cheers,

John


On 04/10/2018 06:13 PM, Jake Ehrlich wrote:
Ideally Jurgen would cut up the code on github, put up an initial diff for a minimal viable tool, and then we would review it and then continue to copy code from the github repo into llvm and review it. I'm also willing to do that if Jurgen doesn't want to at this point though. I'd like the OK from Jurgen on that and I'd also like the OK from someone that the license stuff is all good to go (I'm not sure who should check licence stuff).

Best,
Jake

On Tue, Apr 10, 2018 at 2:39 PM John Ericson [hidden email] wrote:

Seems like there are a few of us interested in this then. I new around here and don't really know how decisions are made, so what's next? Just open a diff with the entire library??

John

On 04/10/2018 05:33 PM, Jake Ehrlich wrote:
Benifits of TBD:
1) It's human readable and diffs on TBDs correspond to changes in the ABI. Diffs can be automatically added to review processes to ensure that changes to the ABI are reviewed. The TBDs also document your precise ABI.
2) The size is smaller which means they can be shipped in an SDK instead of binaries to reduce the size of an SDK
3) Stubs are producible from TBDs (or should be) which means stubs for linking can be produced even if we don't directly support them in LLD. This lets you ship the smaller TBD files in place of larger binaries and still link things without direct linker support (assuming you already ship a toolchain with your SDK or expect your users to have this tool)

Since stubs are producible from TBDs I don't really see a downside. I think we need both, I was going to propose a yaml based representation for ELF for the above reasons anyhow.

On Tue, Apr 10, 2018 at 1:14 PM Andrew Kelley via llvm-dev <[hidden email]> wrote:
On Mon, Apr 9, 2018 at 10:11 PM, John Ericson via llvm-dev <[hidden email]> wrote:
> Regardless of any of that, given that TBD files _are_ an integral part of the apple platform, supporting them is certainly a necessity in order to have a working apple linker. So, if making LLD work for Apple/MachO is the justification for adding TBD support to LLVM, that seems self-evidently a reasonable thing to do. On the other hand, it looks like the LLD mach-o code is unmaintained and nobody seems to be much interested in it. And having code for reading TBD files in LLVM seems not terribly interesting, unless it is as part of a project to make the LLD MachO linker actually functional and supported.

Yes. I hope this can be reason enough. Hobbyists could push for LLD support for Mach-O besides Apple, and if LLD is to displace other linkers this is a necessary component as you say. Better to upstream now before the code diverges than more work later? Conversely if nothing happens, I doubt libtapi would be a greater drag on the codebase than the MachO LLD code, so whatever cost/benefit analysis exists for keeping that around could also apply to this.


Speaking for the Zig project here, our goal is to support cross-compilation for any target, on any target, without requiring installation of any target-specific SDK. So, for example, these use cases:
 * on linux, compile & link a binary targeting macos
 * on windows, compile & link a binary targeting macos

This works today, although it depends on a patch to LLD to fix the MACH-O linker that is not high enough quality to upstream.

So we have a vested interest in improving the MACH-O linker, and in fact a Zig community member has fixed at least one bug in MACH-O LLD: reviews.llvm.org/D35387

I don't fully understand how TBD or TAPI works, but I hope that it results in improvements to the MACH-O linker.

 
_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev



_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev


_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev

_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: [llvm-dev] [RFC] Open sourcing and contributing TAPI back to the LLVM community

Vassil Vassilev via cfe-dev
A member of my team [hidden email] is going to drop a proposal for the ELF part of this soon (like sometime next week) and will be working on the implementation. I'll be one of the reviewers for anything that comes out of that so we can be sure it will get reviewed as well. The goal is to have a single tool/library that would include Juergen's original code and the new ELF code.

On Thu, Sep 20, 2018 at 10:45 AM Nico Weber <[hidden email]> wrote:
Was there any progress in the upstreaming effort? I'd be interested in having lld be able to link against tbd files, and I think it'd be cool if libtool -static could write tbd files (similar to thin archives on linux) since that should make archiving much faster.

Juergen, maybe uploading your initial patch to phabricator instead of attaching might get more traction?

On Wed, Apr 25, 2018 at 8:53 AM Juergen Ributzka via cfe-dev <[hidden email]> wrote:
The code on opensource.apple.com is the minimal code needed for libtapi to read TBD files with ld64. The code I pushed to GitHub on the other side includes the full TAPI tool source code. If you check the code you will see that I already use the LLVM license for everything, because the goal has always been to contribute this back to the community.

I attached an initial patch for libtapi and nm support to my original email. I also have some updated patches for that too, but somehow got derailed into "lets rewrite libobject" discussions.

On Tue, Apr 10, 2018 at 4:15 PM, John Ericson via llvm-dev <[hidden email]> wrote:

That sounds great to me, thanks Jake. I'm not Jurgen either, of course, but I'm happy to assist you if he is unavailable. I'm not also not qualified to audit the license, but do note Apple formally also released some code at https://opensource.apple.com/tarballs/tapi/. If there's anything else I can do to help, let me know.

Cheers,

John


On 04/10/2018 06:13 PM, Jake Ehrlich wrote:
Ideally Jurgen would cut up the code on github, put up an initial diff for a minimal viable tool, and then we would review it and then continue to copy code from the github repo into llvm and review it. I'm also willing to do that if Jurgen doesn't want to at this point though. I'd like the OK from Jurgen on that and I'd also like the OK from someone that the license stuff is all good to go (I'm not sure who should check licence stuff).

Best,
Jake

On Tue, Apr 10, 2018 at 2:39 PM John Ericson [hidden email] wrote:

Seems like there are a few of us interested in this then. I new around here and don't really know how decisions are made, so what's next? Just open a diff with the entire library??

John

On 04/10/2018 05:33 PM, Jake Ehrlich wrote:
Benifits of TBD:
1) It's human readable and diffs on TBDs correspond to changes in the ABI. Diffs can be automatically added to review processes to ensure that changes to the ABI are reviewed. The TBDs also document your precise ABI.
2) The size is smaller which means they can be shipped in an SDK instead of binaries to reduce the size of an SDK
3) Stubs are producible from TBDs (or should be) which means stubs for linking can be produced even if we don't directly support them in LLD. This lets you ship the smaller TBD files in place of larger binaries and still link things without direct linker support (assuming you already ship a toolchain with your SDK or expect your users to have this tool)

Since stubs are producible from TBDs I don't really see a downside. I think we need both, I was going to propose a yaml based representation for ELF for the above reasons anyhow.

On Tue, Apr 10, 2018 at 1:14 PM Andrew Kelley via llvm-dev <[hidden email]> wrote:
On Mon, Apr 9, 2018 at 10:11 PM, John Ericson via llvm-dev <[hidden email]> wrote:
> Regardless of any of that, given that TBD files _are_ an integral part of the apple platform, supporting them is certainly a necessity in order to have a working apple linker. So, if making LLD work for Apple/MachO is the justification for adding TBD support to LLVM, that seems self-evidently a reasonable thing to do. On the other hand, it looks like the LLD mach-o code is unmaintained and nobody seems to be much interested in it. And having code for reading TBD files in LLVM seems not terribly interesting, unless it is as part of a project to make the LLD MachO linker actually functional and supported.

Yes. I hope this can be reason enough. Hobbyists could push for LLD support for Mach-O besides Apple, and if LLD is to displace other linkers this is a necessary component as you say. Better to upstream now before the code diverges than more work later? Conversely if nothing happens, I doubt libtapi would be a greater drag on the codebase than the MachO LLD code, so whatever cost/benefit analysis exists for keeping that around could also apply to this.


Speaking for the Zig project here, our goal is to support cross-compilation for any target, on any target, without requiring installation of any target-specific SDK. So, for example, these use cases:
 * on linux, compile & link a binary targeting macos
 * on windows, compile & link a binary targeting macos

This works today, although it depends on a patch to LLD to fix the MACH-O linker that is not high enough quality to upstream.

So we have a vested interest in improving the MACH-O linker, and in fact a Zig community member has fixed at least one bug in MACH-O LLD: reviews.llvm.org/D35387

I don't fully understand how TBD or TAPI works, but I hope that it results in improvements to the MACH-O linker.

 
_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev



_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev


_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev

_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: [llvm-dev] [RFC] Open sourcing and contributing TAPI back to the LLVM community

Vassil Vassilev via cfe-dev
Great to hear, thanks!

On Thu, Sep 20, 2018 at 1:56 PM Jake Ehrlich <[hidden email]> wrote:
A member of my team [hidden email] is going to drop a proposal for the ELF part of this soon (like sometime next week) and will be working on the implementation. I'll be one of the reviewers for anything that comes out of that so we can be sure it will get reviewed as well. The goal is to have a single tool/library that would include Juergen's original code and the new ELF code.

On Thu, Sep 20, 2018 at 10:45 AM Nico Weber <[hidden email]> wrote:
Was there any progress in the upstreaming effort? I'd be interested in having lld be able to link against tbd files, and I think it'd be cool if libtool -static could write tbd files (similar to thin archives on linux) since that should make archiving much faster.

Juergen, maybe uploading your initial patch to phabricator instead of attaching might get more traction?

On Wed, Apr 25, 2018 at 8:53 AM Juergen Ributzka via cfe-dev <[hidden email]> wrote:
The code on opensource.apple.com is the minimal code needed for libtapi to read TBD files with ld64. The code I pushed to GitHub on the other side includes the full TAPI tool source code. If you check the code you will see that I already use the LLVM license for everything, because the goal has always been to contribute this back to the community.

I attached an initial patch for libtapi and nm support to my original email. I also have some updated patches for that too, but somehow got derailed into "lets rewrite libobject" discussions.

On Tue, Apr 10, 2018 at 4:15 PM, John Ericson via llvm-dev <[hidden email]> wrote:

That sounds great to me, thanks Jake. I'm not Jurgen either, of course, but I'm happy to assist you if he is unavailable. I'm not also not qualified to audit the license, but do note Apple formally also released some code at https://opensource.apple.com/tarballs/tapi/. If there's anything else I can do to help, let me know.

Cheers,

John


On 04/10/2018 06:13 PM, Jake Ehrlich wrote:
Ideally Jurgen would cut up the code on github, put up an initial diff for a minimal viable tool, and then we would review it and then continue to copy code from the github repo into llvm and review it. I'm also willing to do that if Jurgen doesn't want to at this point though. I'd like the OK from Jurgen on that and I'd also like the OK from someone that the license stuff is all good to go (I'm not sure who should check licence stuff).

Best,
Jake

On Tue, Apr 10, 2018 at 2:39 PM John Ericson [hidden email] wrote:

Seems like there are a few of us interested in this then. I new around here and don't really know how decisions are made, so what's next? Just open a diff with the entire library??

John

On 04/10/2018 05:33 PM, Jake Ehrlich wrote:
Benifits of TBD:
1) It's human readable and diffs on TBDs correspond to changes in the ABI. Diffs can be automatically added to review processes to ensure that changes to the ABI are reviewed. The TBDs also document your precise ABI.
2) The size is smaller which means they can be shipped in an SDK instead of binaries to reduce the size of an SDK
3) Stubs are producible from TBDs (or should be) which means stubs for linking can be produced even if we don't directly support them in LLD. This lets you ship the smaller TBD files in place of larger binaries and still link things without direct linker support (assuming you already ship a toolchain with your SDK or expect your users to have this tool)

Since stubs are producible from TBDs I don't really see a downside. I think we need both, I was going to propose a yaml based representation for ELF for the above reasons anyhow.

On Tue, Apr 10, 2018 at 1:14 PM Andrew Kelley via llvm-dev <[hidden email]> wrote:
On Mon, Apr 9, 2018 at 10:11 PM, John Ericson via llvm-dev <[hidden email]> wrote:
> Regardless of any of that, given that TBD files _are_ an integral part of the apple platform, supporting them is certainly a necessity in order to have a working apple linker. So, if making LLD work for Apple/MachO is the justification for adding TBD support to LLVM, that seems self-evidently a reasonable thing to do. On the other hand, it looks like the LLD mach-o code is unmaintained and nobody seems to be much interested in it. And having code for reading TBD files in LLVM seems not terribly interesting, unless it is as part of a project to make the LLD MachO linker actually functional and supported.

Yes. I hope this can be reason enough. Hobbyists could push for LLD support for Mach-O besides Apple, and if LLD is to displace other linkers this is a necessary component as you say. Better to upstream now before the code diverges than more work later? Conversely if nothing happens, I doubt libtapi would be a greater drag on the codebase than the MachO LLD code, so whatever cost/benefit analysis exists for keeping that around could also apply to this.


Speaking for the Zig project here, our goal is to support cross-compilation for any target, on any target, without requiring installation of any target-specific SDK. So, for example, these use cases:
 * on linux, compile & link a binary targeting macos
 * on windows, compile & link a binary targeting macos

This works today, although it depends on a patch to LLD to fix the MACH-O linker that is not high enough quality to upstream.

So we have a vested interest in improving the MACH-O linker, and in fact a Zig community member has fixed at least one bug in MACH-O LLD: reviews.llvm.org/D35387

I don't fully understand how TBD or TAPI works, but I hope that it results in improvements to the MACH-O linker.

 
_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev



_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev


_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev

_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: [llvm-dev] [RFC] Open sourcing and contributing TAPI back to the LLVM community

Vassil Vassilev via cfe-dev
This is great to hear. Please add Steven Wu and me as reviewers. Unfortunately I won't be available for the next weeks, because I am on my honeymoon, but I would like a chance to review the code once I am back.

Thanks

Cheers,
Juergen

On Thu, Sep 20, 2018 at 11:50 AM Nico Weber <[hidden email]> wrote:
Great to hear, thanks!

On Thu, Sep 20, 2018 at 1:56 PM Jake Ehrlich <[hidden email]> wrote:
A member of my team [hidden email] is going to drop a proposal for the ELF part of this soon (like sometime next week) and will be working on the implementation. I'll be one of the reviewers for anything that comes out of that so we can be sure it will get reviewed as well. The goal is to have a single tool/library that would include Juergen's original code and the new ELF code.

On Thu, Sep 20, 2018 at 10:45 AM Nico Weber <[hidden email]> wrote:
Was there any progress in the upstreaming effort? I'd be interested in having lld be able to link against tbd files, and I think it'd be cool if libtool -static could write tbd files (similar to thin archives on linux) since that should make archiving much faster.

Juergen, maybe uploading your initial patch to phabricator instead of attaching might get more traction?

On Wed, Apr 25, 2018 at 8:53 AM Juergen Ributzka via cfe-dev <[hidden email]> wrote:
The code on opensource.apple.com is the minimal code needed for libtapi to read TBD files with ld64. The code I pushed to GitHub on the other side includes the full TAPI tool source code. If you check the code you will see that I already use the LLVM license for everything, because the goal has always been to contribute this back to the community.

I attached an initial patch for libtapi and nm support to my original email. I also have some updated patches for that too, but somehow got derailed into "lets rewrite libobject" discussions.

On Tue, Apr 10, 2018 at 4:15 PM, John Ericson via llvm-dev <[hidden email]> wrote:

That sounds great to me, thanks Jake. I'm not Jurgen either, of course, but I'm happy to assist you if he is unavailable. I'm not also not qualified to audit the license, but do note Apple formally also released some code at https://opensource.apple.com/tarballs/tapi/. If there's anything else I can do to help, let me know.

Cheers,

John


On 04/10/2018 06:13 PM, Jake Ehrlich wrote:
Ideally Jurgen would cut up the code on github, put up an initial diff for a minimal viable tool, and then we would review it and then continue to copy code from the github repo into llvm and review it. I'm also willing to do that if Jurgen doesn't want to at this point though. I'd like the OK from Jurgen on that and I'd also like the OK from someone that the license stuff is all good to go (I'm not sure who should check licence stuff).

Best,
Jake

On Tue, Apr 10, 2018 at 2:39 PM John Ericson [hidden email] wrote:

Seems like there are a few of us interested in this then. I new around here and don't really know how decisions are made, so what's next? Just open a diff with the entire library??

John

On 04/10/2018 05:33 PM, Jake Ehrlich wrote:
Benifits of TBD:
1) It's human readable and diffs on TBDs correspond to changes in the ABI. Diffs can be automatically added to review processes to ensure that changes to the ABI are reviewed. The TBDs also document your precise ABI.
2) The size is smaller which means they can be shipped in an SDK instead of binaries to reduce the size of an SDK
3) Stubs are producible from TBDs (or should be) which means stubs for linking can be produced even if we don't directly support them in LLD. This lets you ship the smaller TBD files in place of larger binaries and still link things without direct linker support (assuming you already ship a toolchain with your SDK or expect your users to have this tool)

Since stubs are producible from TBDs I don't really see a downside. I think we need both, I was going to propose a yaml based representation for ELF for the above reasons anyhow.

On Tue, Apr 10, 2018 at 1:14 PM Andrew Kelley via llvm-dev <[hidden email]> wrote:
On Mon, Apr 9, 2018 at 10:11 PM, John Ericson via llvm-dev <[hidden email]> wrote:
> Regardless of any of that, given that TBD files _are_ an integral part of the apple platform, supporting them is certainly a necessity in order to have a working apple linker. So, if making LLD work for Apple/MachO is the justification for adding TBD support to LLVM, that seems self-evidently a reasonable thing to do. On the other hand, it looks like the LLD mach-o code is unmaintained and nobody seems to be much interested in it. And having code for reading TBD files in LLVM seems not terribly interesting, unless it is as part of a project to make the LLD MachO linker actually functional and supported.

Yes. I hope this can be reason enough. Hobbyists could push for LLD support for Mach-O besides Apple, and if LLD is to displace other linkers this is a necessary component as you say. Better to upstream now before the code diverges than more work later? Conversely if nothing happens, I doubt libtapi would be a greater drag on the codebase than the MachO LLD code, so whatever cost/benefit analysis exists for keeping that around could also apply to this.


Speaking for the Zig project here, our goal is to support cross-compilation for any target, on any target, without requiring installation of any target-specific SDK. So, for example, these use cases:
 * on linux, compile & link a binary targeting macos
 * on windows, compile & link a binary targeting macos

This works today, although it depends on a patch to LLD to fix the MACH-O linker that is not high enough quality to upstream.

So we have a vested interest in improving the MACH-O linker, and in fact a Zig community member has fixed at least one bug in MACH-O LLD: reviews.llvm.org/D35387

I don't fully understand how TBD or TAPI works, but I hope that it results in improvements to the MACH-O linker.

 
_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev



_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev


_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev

_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: [llvm-dev] [RFC] Open sourcing and contributing TAPI back to the LLVM community

Vassil Vassilev via cfe-dev
Awesome, having reviewers puts us ahead of the curve! I'm quite excited for this to make it in.

On Thu, Sep 20, 2018 at 1:59 PM Juergen Ributzka <[hidden email]> wrote:
This is great to hear. Please add Steven Wu and me as reviewers. Unfortunately I won't be available for the next weeks, because I am on my honeymoon, but I would like a chance to review the code once I am back.

Thanks

Cheers,
Juergen

On Thu, Sep 20, 2018 at 11:50 AM Nico Weber <[hidden email]> wrote:
Great to hear, thanks!

On Thu, Sep 20, 2018 at 1:56 PM Jake Ehrlich <[hidden email]> wrote:
A member of my team [hidden email] is going to drop a proposal for the ELF part of this soon (like sometime next week) and will be working on the implementation. I'll be one of the reviewers for anything that comes out of that so we can be sure it will get reviewed as well. The goal is to have a single tool/library that would include Juergen's original code and the new ELF code.

On Thu, Sep 20, 2018 at 10:45 AM Nico Weber <[hidden email]> wrote:
Was there any progress in the upstreaming effort? I'd be interested in having lld be able to link against tbd files, and I think it'd be cool if libtool -static could write tbd files (similar to thin archives on linux) since that should make archiving much faster.

Juergen, maybe uploading your initial patch to phabricator instead of attaching might get more traction?

On Wed, Apr 25, 2018 at 8:53 AM Juergen Ributzka via cfe-dev <[hidden email]> wrote:
The code on opensource.apple.com is the minimal code needed for libtapi to read TBD files with ld64. The code I pushed to GitHub on the other side includes the full TAPI tool source code. If you check the code you will see that I already use the LLVM license for everything, because the goal has always been to contribute this back to the community.

I attached an initial patch for libtapi and nm support to my original email. I also have some updated patches for that too, but somehow got derailed into "lets rewrite libobject" discussions.

On Tue, Apr 10, 2018 at 4:15 PM, John Ericson via llvm-dev <[hidden email]> wrote:

That sounds great to me, thanks Jake. I'm not Jurgen either, of course, but I'm happy to assist you if he is unavailable. I'm not also not qualified to audit the license, but do note Apple formally also released some code at https://opensource.apple.com/tarballs/tapi/. If there's anything else I can do to help, let me know.

Cheers,

John


On 04/10/2018 06:13 PM, Jake Ehrlich wrote:
Ideally Jurgen would cut up the code on github, put up an initial diff for a minimal viable tool, and then we would review it and then continue to copy code from the github repo into llvm and review it. I'm also willing to do that if Jurgen doesn't want to at this point though. I'd like the OK from Jurgen on that and I'd also like the OK from someone that the license stuff is all good to go (I'm not sure who should check licence stuff).

Best,
Jake

On Tue, Apr 10, 2018 at 2:39 PM John Ericson [hidden email] wrote:

Seems like there are a few of us interested in this then. I new around here and don't really know how decisions are made, so what's next? Just open a diff with the entire library??

John

On 04/10/2018 05:33 PM, Jake Ehrlich wrote:
Benifits of TBD:
1) It's human readable and diffs on TBDs correspond to changes in the ABI. Diffs can be automatically added to review processes to ensure that changes to the ABI are reviewed. The TBDs also document your precise ABI.
2) The size is smaller which means they can be shipped in an SDK instead of binaries to reduce the size of an SDK
3) Stubs are producible from TBDs (or should be) which means stubs for linking can be produced even if we don't directly support them in LLD. This lets you ship the smaller TBD files in place of larger binaries and still link things without direct linker support (assuming you already ship a toolchain with your SDK or expect your users to have this tool)

Since stubs are producible from TBDs I don't really see a downside. I think we need both, I was going to propose a yaml based representation for ELF for the above reasons anyhow.

On Tue, Apr 10, 2018 at 1:14 PM Andrew Kelley via llvm-dev <[hidden email]> wrote:
On Mon, Apr 9, 2018 at 10:11 PM, John Ericson via llvm-dev <[hidden email]> wrote:
> Regardless of any of that, given that TBD files _are_ an integral part of the apple platform, supporting them is certainly a necessity in order to have a working apple linker. So, if making LLD work for Apple/MachO is the justification for adding TBD support to LLVM, that seems self-evidently a reasonable thing to do. On the other hand, it looks like the LLD mach-o code is unmaintained and nobody seems to be much interested in it. And having code for reading TBD files in LLVM seems not terribly interesting, unless it is as part of a project to make the LLD MachO linker actually functional and supported.

Yes. I hope this can be reason enough. Hobbyists could push for LLD support for Mach-O besides Apple, and if LLD is to displace other linkers this is a necessary component as you say. Better to upstream now before the code diverges than more work later? Conversely if nothing happens, I doubt libtapi would be a greater drag on the codebase than the MachO LLD code, so whatever cost/benefit analysis exists for keeping that around could also apply to this.


Speaking for the Zig project here, our goal is to support cross-compilation for any target, on any target, without requiring installation of any target-specific SDK. So, for example, these use cases:
 * on linux, compile & link a binary targeting macos
 * on windows, compile & link a binary targeting macos

This works today, although it depends on a patch to LLD to fix the MACH-O linker that is not high enough quality to upstream.

So we have a vested interest in improving the MACH-O linker, and in fact a Zig community member has fixed at least one bug in MACH-O LLD: reviews.llvm.org/D35387

I don't fully understand how TBD or TAPI works, but I hope that it results in improvements to the MACH-O linker.

 
_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev



_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev


_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev

_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: [llvm-dev] [RFC] Open sourcing and contributing TAPI back to the LLVM community

Vassil Vassilev via cfe-dev
Add me too please, I've got an area I'd like to investigate along with it and keeping track via reviews is awesome :)

(and I'm happy to at least do small llvm style reviews on it and then let Juergen or Steven do the final review)

On Thu, Sep 20, 2018 at 2:22 PM Jake Ehrlich via cfe-dev <[hidden email]> wrote:
Awesome, having reviewers puts us ahead of the curve! I'm quite excited for this to make it in.


On Thu, Sep 20, 2018 at 1:59 PM Juergen Ributzka <[hidden email]> wrote:
This is great to hear. Please add Steven Wu and me as reviewers. Unfortunately I won't be available for the next weeks, because I am on my honeymoon, but I would like a chance to review the code once I am back.

Thanks

Cheers,
Juergen

On Thu, Sep 20, 2018 at 11:50 AM Nico Weber <[hidden email]> wrote:
Great to hear, thanks!

On Thu, Sep 20, 2018 at 1:56 PM Jake Ehrlich <[hidden email]> wrote:
A member of my team [hidden email] is going to drop a proposal for the ELF part of this soon (like sometime next week) and will be working on the implementation. I'll be one of the reviewers for anything that comes out of that so we can be sure it will get reviewed as well. The goal is to have a single tool/library that would include Juergen's original code and the new ELF code.

On Thu, Sep 20, 2018 at 10:45 AM Nico Weber <[hidden email]> wrote:
Was there any progress in the upstreaming effort? I'd be interested in having lld be able to link against tbd files, and I think it'd be cool if libtool -static could write tbd files (similar to thin archives on linux) since that should make archiving much faster.

Juergen, maybe uploading your initial patch to phabricator instead of attaching might get more traction?

On Wed, Apr 25, 2018 at 8:53 AM Juergen Ributzka via cfe-dev <[hidden email]> wrote:
The code on opensource.apple.com is the minimal code needed for libtapi to read TBD files with ld64. The code I pushed to GitHub on the other side includes the full TAPI tool source code. If you check the code you will see that I already use the LLVM license for everything, because the goal has always been to contribute this back to the community.

I attached an initial patch for libtapi and nm support to my original email. I also have some updated patches for that too, but somehow got derailed into "lets rewrite libobject" discussions.

On Tue, Apr 10, 2018 at 4:15 PM, John Ericson via llvm-dev <[hidden email]> wrote:

That sounds great to me, thanks Jake. I'm not Jurgen either, of course, but I'm happy to assist you if he is unavailable. I'm not also not qualified to audit the license, but do note Apple formally also released some code at https://opensource.apple.com/tarballs/tapi/. If there's anything else I can do to help, let me know.

Cheers,

John


On 04/10/2018 06:13 PM, Jake Ehrlich wrote:
Ideally Jurgen would cut up the code on github, put up an initial diff for a minimal viable tool, and then we would review it and then continue to copy code from the github repo into llvm and review it. I'm also willing to do that if Jurgen doesn't want to at this point though. I'd like the OK from Jurgen on that and I'd also like the OK from someone that the license stuff is all good to go (I'm not sure who should check licence stuff).

Best,
Jake

On Tue, Apr 10, 2018 at 2:39 PM John Ericson [hidden email] wrote:

Seems like there are a few of us interested in this then. I new around here and don't really know how decisions are made, so what's next? Just open a diff with the entire library??

John

On 04/10/2018 05:33 PM, Jake Ehrlich wrote:
Benifits of TBD:
1) It's human readable and diffs on TBDs correspond to changes in the ABI. Diffs can be automatically added to review processes to ensure that changes to the ABI are reviewed. The TBDs also document your precise ABI.
2) The size is smaller which means they can be shipped in an SDK instead of binaries to reduce the size of an SDK
3) Stubs are producible from TBDs (or should be) which means stubs for linking can be produced even if we don't directly support them in LLD. This lets you ship the smaller TBD files in place of larger binaries and still link things without direct linker support (assuming you already ship a toolchain with your SDK or expect your users to have this tool)

Since stubs are producible from TBDs I don't really see a downside. I think we need both, I was going to propose a yaml based representation for ELF for the above reasons anyhow.

On Tue, Apr 10, 2018 at 1:14 PM Andrew Kelley via llvm-dev <[hidden email]> wrote:
On Mon, Apr 9, 2018 at 10:11 PM, John Ericson via llvm-dev <[hidden email]> wrote:
> Regardless of any of that, given that TBD files _are_ an integral part of the apple platform, supporting them is certainly a necessity in order to have a working apple linker. So, if making LLD work for Apple/MachO is the justification for adding TBD support to LLVM, that seems self-evidently a reasonable thing to do. On the other hand, it looks like the LLD mach-o code is unmaintained and nobody seems to be much interested in it. And having code for reading TBD files in LLVM seems not terribly interesting, unless it is as part of a project to make the LLD MachO linker actually functional and supported.

Yes. I hope this can be reason enough. Hobbyists could push for LLD support for Mach-O besides Apple, and if LLD is to displace other linkers this is a necessary component as you say. Better to upstream now before the code diverges than more work later? Conversely if nothing happens, I doubt libtapi would be a greater drag on the codebase than the MachO LLD code, so whatever cost/benefit analysis exists for keeping that around could also apply to this.


Speaking for the Zig project here, our goal is to support cross-compilation for any target, on any target, without requiring installation of any target-specific SDK. So, for example, these use cases:
 * on linux, compile & link a binary targeting macos
 * on windows, compile & link a binary targeting macos

This works today, although it depends on a patch to LLD to fix the MACH-O linker that is not high enough quality to upstream.

So we have a vested interest in improving the MACH-O linker, and in fact a Zig community member has fixed at least one bug in MACH-O LLD: reviews.llvm.org/D35387

I don't fully understand how TBD or TAPI works, but I hope that it results in improvements to the MACH-O linker.

 
_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev



_______________________________________________
LLVM Developers mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev


_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev

_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
12