First shot at Bug 4127 - The clang driver should support cross compilation

classic Classic list List threaded Threaded
26 messages Options
12
Reply | Threaded
Open this post in threaded view
|

First shot at Bug 4127 - The clang driver should support cross compilation

Sebastian Pop
Hi,

The attached patch uses the JSON parser to read the target
from a config file.  I would like to get some opinions on this
patch before we go further to fix
http://llvm.org/bugs/show_bug.cgi?id=4127

In a first implementation, I would like to have at least the following
information registered in the config file:

[
    {
        "target": "arm-none-linux-gnueabi",
        "c-include-dirs": "/gcc/path:/libc/path",
        "cxx-include-root": "/some/other/path"
    }
]

In a second step, I would like to add the full explicit path to the
assembler, to the linker, the -L paths, and the -I paths.

I would like the default config file to be created by the configure scripts,
and all the logic that is currently executed at each clang compilation
to be moved to configure time.

One other point to be discussed is the place where to install these
target config files: in the current patch I am looking in targets, a subdir
of the install dir.  Is there a better place to put these config files?

Thanks,
Sebastian
--
Qualcomm Innovation Center, Inc is a member of Code Aurora Forum

_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev

0001-read-the-target-from-a-config-file.patch (6K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: First shot at Bug 4127 - The clang driver should support cross compilation

Joerg Sonnenberger
On Mon, Jan 09, 2012 at 03:00:30PM -0600, Sebastian Pop wrote:
> I would like the default config file to be created by the configure scripts,
> and all the logic that is currently executed at each clang compilation
> to be moved to configure time.

This is the part I object to. Please do not make cross-compilation more
painful than it has to be. It works well for some of us without any new
movable parts, just by specifying the right target and --sysroot.

Joerg
_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: First shot at Bug 4127 - The clang driver should support cross compilation

Rafael Espíndola
In reply to this post by Sebastian Pop
On 09/01/12 04:00 PM, Sebastian Pop wrote:

> Hi,
>
> The attached patch uses the JSON parser to read the target
> from a config file.  I would like to get some opinions on this
> patch before we go further to fix
> http://llvm.org/bugs/show_bug.cgi?id=4127
>
> In a first implementation, I would like to have at least the following
> information registered in the config file:
>
> [
>     {
> "target": "arm-none-linux-gnueabi",
> "c-include-dirs": "/gcc/path:/libc/path",
> "cxx-include-root": "/some/other/path"
>     }
> ]

Looks like a good place to start. What is your transition plan, add a
--target-config option and if the option is not used fallback to the
current implementation?

I noticed uses of llvm:err() in the patch. I assume those are just for
debugging while the patch is being developed, right?

> In a second step, I would like to add the full explicit path to the
> assembler, to the linker, the -L paths, and the -I paths.

cool.

> I would like the default config file to be created by the configure scripts,
> and all the logic that is currently executed at each clang compilation
> to be moved to configure time.

I would suggest moving the current logic to a helper tool
(clang-gen-config?). That way the same clang binary can be installed in
two different linux systems and the uses just has to create a config
file at install time.

> One other point to be discussed is the place where to install these
> target config files: in the current patch I am looking in targets, a subdir
> of the install dir.  Is there a better place to put these config files?
>
> Thanks,
> Sebastian
> --
> Qualcomm Innovation Center, Inc is a member of Code Aurora Forum
>
>


Thanks,
Rafael
_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: First shot at Bug 4127 - The clang driver should support cross compilation

Rafael Espíndola
In reply to this post by Joerg Sonnenberger
On 09/01/12 04:46 PM, Joerg Sonnenberger wrote:
> On Mon, Jan 09, 2012 at 03:00:30PM -0600, Sebastian Pop wrote:
>> I would like the default config file to be created by the configure scripts,
>> and all the logic that is currently executed at each clang compilation
>> to be moved to configure time.
>
> This is the part I object to. Please do not make cross-compilation more
> painful than it has to be. It works well for some of us without any new
> movable parts, just by specifying the right target and --sysroot.

So this would cut the number of options you have to use in halve :-)

> Joerg

Cheers,
Rafael
_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: First shot at Bug 4127 - The clang driver should support cross compilation

Chandler Carruth
In reply to this post by Joerg Sonnenberger
On Mon, Jan 9, 2012 at 1:46 PM, Joerg Sonnenberger <[hidden email]> wrote:
On Mon, Jan 09, 2012 at 03:00:30PM -0600, Sebastian Pop wrote:
> I would like the default config file to be created by the configure scripts,
> and all the logic that is currently executed at each clang compilation
> to be moved to configure time.

This is the part I object to. Please do not make cross-compilation more
painful than it has to be. It works well for some of us without any new
movable parts, just by specifying the right target and --sysroot.

I completely agree. I'm really opposed to removing baked-in support for the most common platforms. This should be about extending our ability to support less common platforms more easily.

Also, I'm not really excited about starting to attack this problem from this direction. I think the first steps need to be refactoring and rationalizing the existing code in the driver to have a better design and structure, before we start extending it to support referencing an external data file to collect information. Otherwise I fear we will embed the wrong structure or information in the data file, and use these data files as crutches to avoid properly designing the internals.

_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: First shot at Bug 4127 - The clang driver should support cross compilation

James Molloy-2
In reply to this post by Sebastian Pop
Hi Sebastian,

Were you at the driver BoF at the conference? We discussed the cross compilation story there (although your solution does look similar to that discussed, so maybe that's not a coincidence...)

I'm actually already working on a patch to do the same. I think the problem is more involved than just setting up target and include dirs - I think the target database should have OS specific information too (which flags are required per OS etc, so a distro maintainer can change whatever he may need at package time without patching the source).

As I say, I'm working on a patch that I think is a superset of yours and would conflict massively. I've been planning it for some time and think I have a viable end goal and route to get there. Would it be possible for you to hold off your patch for a week or so until mine is ready and can be used as a base?

Cheers,

James

-----Original Message-----
From: Sebastian Pop [mailto:[hidden email]]
Sent: 09 January 2012 21:01
To: [hidden email] Developers; [hidden email]
Cc: Eric Christopher; James Molloy
Subject: First shot at Bug 4127 - The clang driver should support cross compilation

Hi,

The attached patch uses the JSON parser to read the target
from a config file.  I would like to get some opinions on this
patch before we go further to fix
http://llvm.org/bugs/show_bug.cgi?id=4127

In a first implementation, I would like to have at least the following
information registered in the config file:

[
    {
        "target": "arm-none-linux-gnueabi",
        "c-include-dirs": "/gcc/path:/libc/path",
        "cxx-include-root": "/some/other/path"
    }
]

In a second step, I would like to add the full explicit path to the
assembler, to the linker, the -L paths, and the -I paths.

I would like the default config file to be created by the configure scripts,
and all the logic that is currently executed at each clang compilation
to be moved to configure time.

One other point to be discussed is the place where to install these
target config files: in the current patch I am looking in targets, a subdir
of the install dir.  Is there a better place to put these config files?

Thanks,
Sebastian
--
Qualcomm Innovation Center, Inc is a member of Code Aurora Forum

-- IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium.  Thank you.


_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: First shot at Bug 4127 - The clang driver should support cross compilation

Konstantin Tokarev
In reply to this post by Sebastian Pop


10.01.2012, 01:00, "Sebastian Pop" <[hidden email]>:

> Hi,
>
> The attached patch uses the JSON parser to read the target
> from a config file.  I would like to get some opinions on this
> patch before we go further to fix
> http://llvm.org/bugs/show_bug.cgi?id=4127
>
> In a first implementation, I would like to have at least the following
> information registered in the config file:
>
> [
>     {
>         "target": "arm-none-linux-gnueabi",
>         "c-include-dirs": "/gcc/path:/libc/path",
>         "cxx-include-root": "/some/other/path"
>     }
> ]
>

I propose to embed Lua and use Lua table instead of JSON. Config could look like

toolchain {
    target = "arm-none-linux-gnueabi",
    c_include_dirs = { "/gcc/path", "/libc/path" },
    cxx_include_root = "/some/other/path",
    as = "/path/to/as",
    ld = "/path/to/ld"
}

toolchain {
-- next toolchain definition
..
}

It would also bring scripting capabilities.

--
Regards,
Konstantin
_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: First shot at Bug 4127 - The clang driver should support cross compilation

James Molloy-2
Hi,

The idea of embedding a scripting language has been veto'd on multiple occasions because of

(a) the dependency on a scripting language / putting a scripting interpreter in the tree
(b) speed issues
(c) turing completeness makes it impossible to guarantee a decent speed of execution.

The problem has been discussed and a solution (JSON target database) is winging its way to the list shortly. It's just taken me a while to work out just how much of Tools.cpp and ToolChains.cpp I can rip out (5000 lines of "kinda-sorta-common-code"), when, and how to do it incrementally.

Cheers,

James

-----Original Message-----
From: Konstantin Tokarev [mailto:[hidden email]]
Sent: 10 January 2012 09:02
To: Sebastian Pop
Cc: [hidden email] Developers; [hidden email]; James Molloy
Subject: Re: [cfe-dev] First shot at Bug 4127 - The clang driver should support cross compilation



10.01.2012, 01:00, "Sebastian Pop" <[hidden email]>:

> Hi,
>
> The attached patch uses the JSON parser to read the target
> from a config file.  I would like to get some opinions on this
> patch before we go further to fix
> http://llvm.org/bugs/show_bug.cgi?id=4127
>
> In a first implementation, I would like to have at least the following
> information registered in the config file:
>
> [
>     {
>         "target": "arm-none-linux-gnueabi",
>         "c-include-dirs": "/gcc/path:/libc/path",
>         "cxx-include-root": "/some/other/path"
>     }
> ]
>

I propose to embed Lua and use Lua table instead of JSON. Config could look like

toolchain {
    target = "arm-none-linux-gnueabi",
    c_include_dirs = { "/gcc/path", "/libc/path" },
    cxx_include_root = "/some/other/path",
    as = "/path/to/as",
    ld = "/path/to/ld"
}

toolchain {
-- next toolchain definition
..
}

It would also bring scripting capabilities.

--
Regards,
Konstantin


-- IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium.  Thank you.


_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: First shot at Bug 4127 - The clang driver should support cross compilation

Konstantin Tokarev


10.01.2012, 13:09, "James Molloy" <[hidden email]>:
> Hi,
>
> The idea of embedding a scripting language has been veto'd on multiple occasions because of
>
> (a) the dependency on a scripting language / putting a scripting interpreter in the tree
> (b) speed issues
> (c) turing completeness makes it impossible to guarantee a decent speed of execution.

While you're using Lua as data storage format it won't cause any performance problems. If some user needs to use scripting to accomplish the task, it's not your problem if performance is optimal (however, Lua interpreter itself is damn fast).

--
Regards,
Konstantin
_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: First shot at Bug 4127 - The clang driver should support cross compilation

James Molloy-2
> While you're using Lua as data storage format it won't cause any performance problems.

In that case what is the point in using a scripting language?

> If some user needs to use scripting to accomplish the task, it's not your problem if performance is optimal (however, Lua interpreter itself is damn fast).

Actually it is. If we provide a mechanism for a distributor to undo all the hard work everyone's put in to making Clang fast by merely writing a bad script, we have to live with the reputation of being slow. It doesn't matter whose fault it is as far as the user is concerned.

It's a no go as far as I'm concerned, and I'm sure when the Apple guys (Eric) wake up they'll weigh in :)

Cheers,

James

-----Original Message-----
From: Konstantin Tokarev [mailto:[hidden email]]
Sent: 10 January 2012 09:15
To: James Molloy
Cc: Sebastian Pop; [hidden email] Developers; [hidden email]
Subject: Re: [cfe-dev] First shot at Bug 4127 - The clang driver should support cross compilation



10.01.2012, 13:09, "James Molloy" <[hidden email]>:
> Hi,
>
> The idea of embedding a scripting language has been veto'd on multiple occasions because of
>
> (a) the dependency on a scripting language / putting a scripting interpreter in the tree
> (b) speed issues
> (c) turing completeness makes it impossible to guarantee a decent speed of execution.

While you're using Lua as data storage format it won't cause any performance problems. If some user needs to use scripting to accomplish the task, it's not your problem if performance is optimal (however, Lua interpreter itself is damn fast).

--
Regards,
Konstantin


-- IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium.  Thank you.


_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: First shot at Bug 4127 - The clang driver should support cross compilation

Chandler Carruth
In reply to this post by James Molloy-2
On Tue, Jan 10, 2012 at 12:25 AM, James Molloy <[hidden email]> wrote:
As I say, I'm working on a patch that I think is a superset of yours and would conflict massively. I've been planning it for some time and think I have a viable end goal and route to get there.

It would be really nice to have smaller patches rather than larger patches, and earlier discussion of them.

Again, I remain very concerned about doing lots of work around configuration files to configure a *broken* driver design. I think we'll just end up with broken config file designs as well, and we'll simultaneously make it that much harder to refactor and change the driver in the future.

I am still pushing to see refactoring and design work on the *existing* use cases the driver supports before extending the use cases. I don't know how to support cross compilation for more and more diverse platforms prior to getting cross compilation for very basic platforms, or even non-cross compilation into a better state.

Consider that system header search logic for Darwin, MinGW, and Cygwin is still largely implemented in the Frontend rather than the Driver. This is something I'm actively working on for reference...

_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: First shot at Bug 4127 - The clang driver should support cross compilation

James Molloy-2
> It would be really nice to have smaller patches rather than larger patches, and earlier discussion of them.
>
> Again, I remain very concerned about doing lots of work around configuration files to configure a *broken* driver design. I think we'll just end up > with broken config file designs as well, and we'll simultaneously make it that much harder to refactor and change the driver in the future.
>
> I am still pushing to see refactoring and design work on the *existing* use cases the driver supports before extending the use cases. I don't know how to support cross compilation for more and more diverse platforms prior to getting cross compilation for very basic platforms, or even non-cross compilation into a better state.

> Consider that system header search logic for Darwin, MinGW, and Cygwin is still largely implemented in the Frontend rather than the Driver. This is something I'm actively working on for reference...

Indeed, and this is why it has taken so long since the conference to get something solid (apart from other work). Not only do we need to work out an end goal but make it achievable in small reviewable steps.

I've studied the internals of the driver and thought of several different ways of factoring it, including a composable "pass" framework where arguments get successively modified by composable passes. That had some promise, as the baked in behaviour could be completely controlled at runtime easily, but was a no go because, and this is the major part:

I can't find a way of validating that a gigantic refactor makes no functional change in the driver. The regression tests aren't sufficient, and I'm likely to break Darwin or some other target with a huge refactor - Tools.cpp for example contains 5200 lines, some of which are common and others not.

Add to that the fact that after many iterations I still come back to the current driver design as "not broken". There's nothing wrong with the concept of Tools and ToolChains - in fact as an abstraction they suit reality well.

The main thing I see being the problem is the use of subclassing to parameterise the Tool classes. Because they weren't designed for parameterisation to start with, people have also copypasta'd huge chunks of code around. There are at least 5 different functions that can driver "ld" or "as", for example, each subtly different because one or two have had bugfixes, some have trashed behaviour they don't support, etc etc.

So here's my general "vision":

 * A subclass of Tool will relate solely to the command it is driving/producing, not OS/Arch specific configuration thereof. For example, "binutils::As", "binutils::Ld", "gcc::Compile", "gcc::Link", "gcc::Assemble", "visualstudio::Link".
   * These tools will have a parameter "std::vector<std::string> ExtraArgs", which is a list of extra arguments to give to the tool. This will be created elsewhere.
   * I have yet to work out where Darwin will fit here - ideally I'd like to have Darwin do all its funky logic and stick it all in ExtraArgs then go independent from there, but I don't know the best solution.

 * A target should be able to select any tool for any JobAction. This makes hard-baked ToolChains superfluous. You shouldn't have to subclass ToolChain for your target, because it will be dynamically generated by...

 * The "target database". I think this should be able to parameterise the Tools in any way required - all OS-specific stuff (With the exception of Darwin - that probably requires too much imperative code) should be in the DB.
  * This can take two forms - hard-baked and JSON. The hard-baked version I see being a tablegen file similar (as possible) to the JSON representation, which is compiled into Clang for speed.
  * This way, we keep the speed and extensibility and channel them both through the same interface, so that anything you can do hard-coded you can also change at runtime.


So here's my migration plan:

  1. The target database is where all the current imperative configuration should be factored out to. Create an initial draft schema, a ToolChain/HostInfo that uses it. At this point I suggest only using JSON as this will be easier to change should the schema change than a tablegen backend. The tablegen backend can be added later and the JSON data ported over for speed.
  2. Create the first of the "properly independent" Tools - binutils::Ld and binutils::As, and use the target database to parameterise them.
    * Probably first patch checkin point? Use a new driver debug flag to enable the new behaviour -ccc-dynamic-driver.
  3. Port more ToolChains to the target database. For linux, we'd need to keep the distro detection logic outside the targetdb, but then we shouldn't need clever header detection methods as we can bake the expected header locations for a given distro into the target database.
  4. Sort out what we're doing with Darwin. Is it having its own set of Tools and living in its own domain, or is it linked to the independent tools?
  5. The Big Switchover, at which point we can remove ideally around 4000 lines in Tools.cpp and 90% of ToolChains.cpp (probably also HostInfo.cpp) and end up with a driver which is centrally configurable both at compile and runtime.


OK, so there's a full braindump. I was going to throw this up for discussion in true LLVM style - "with a patch" - in a week or so but my hand has been pushed ;)

Note that this doesn't address the parsing logic disparity between Driver and Frontend - that's not my aim. I'm hoping to "fix the driver for cross-compilation", not fix the entire driver. I'm hoping someone else might chip in there!

Let the heckling commence! ;)

Cheers,

James



From: Chandler Carruth [mailto:[hidden email]]
Sent: 10 January 2012 09:24
To: James Molloy
Cc: Sebastian Pop; [hidden email] Developers; [hidden email]
Subject: Re: [cfe-dev] First shot at Bug 4127 - The clang driver should support cross compilation

On Tue, Jan 10, 2012 at 12:25 AM, James Molloy <[hidden email]> wrote:
As I say, I'm working on a patch that I think is a superset of yours and would conflict massively. I've been planning it for some time and think I have a viable end goal and route to get there.

It would be really nice to have smaller patches rather than larger patches, and earlier discussion of them.

Again, I remain very concerned about doing lots of work around configuration files to configure a *broken* driver design. I think we'll just end up with broken config file designs as well, and we'll simultaneously make it that much harder to refactor and change the driver in the future.

I am still pushing to see refactoring and design work on the *existing* use cases the driver supports before extending the use cases. I don't know how to support cross compilation for more and more diverse platforms prior to getting cross compilation for very basic platforms, or even non-cross compilation into a better state.

Consider that system header search logic for Darwin, MinGW, and Cygwin is still largely implemented in the Frontend rather than the Driver. This is something I'm actively working on for reference...

-- IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium.  Thank you.

_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: First shot at Bug 4127 - The clang driver should support cross compilation

David Chisnall
On 10 Jan 2012, at 10:00, James Molloy wrote:

>  * This can take two forms - hard-baked and JSON. The hard-baked version I see being a tablegen file similar (as possible) to the JSON representation, which is compiled into Clang for speed.

Is there any reason why you can't create a tool that parses the JSON and emits a C++ class so that it can be exactly the same for the cases that don't require any custom executable logic?  This would make prototyping new platforms a lot easier, since it would mean that you'd just poke the JSON until it worked and then add it to the list of JSON files to be transformed into C++ in the default build when it was ready (and people shipping clang for their obscure platform could just add it as part of their default build, even if it were of no interest to anyone else).

David

-- Sent from my brain
_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: First shot at Bug 4127 - The clang driver should support cross compilation

James Molloy-2
Certainly a good idea, and probably better than having two different formats.

I like it (and if that's the only issue you have with my über plan, I'm a happy man!)

Cheers,

James

-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of David Chisnall
Sent: 10 January 2012 10:42
To: James Molloy
Cc: Sebastian Pop; [hidden email] Developers; [hidden email]
Subject: Re: [cfe-dev] First shot at Bug 4127 - The clang driver should support cross compilation

On 10 Jan 2012, at 10:00, James Molloy wrote:

>  * This can take two forms - hard-baked and JSON. The hard-baked version I see being a tablegen file similar (as possible) to the JSON representation, which is compiled into Clang for speed.

Is there any reason why you can't create a tool that parses the JSON and emits a C++ class so that it can be exactly the same for the cases that don't require any custom executable logic?  This would make prototyping new platforms a lot easier, since it would mean that you'd just poke the JSON until it worked and then add it to the list of JSON files to be transformed into C++ in the default build when it was ready (and people shipping clang for their obscure platform could just add it as part of their default build, even if it were of no interest to anyone else).

David

-- Sent from my brain
_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev


-- IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium.  Thank you.


_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: First shot at Bug 4127 - The clang driver should support cross compilation

Ruben Van Boxem
In reply to this post by James Molloy-2

Op 10 jan. 2012 11:50 schreef "James Molloy" <[hidden email]> het volgende:
>
> > It would be really nice to have smaller patches rather than larger patches, and earlier discussion of them.
> >
> > Again, I remain very concerned about doing lots of work around configuration files to configure a *broken* driver design. I think we'll just end up > with broken config file designs as well, and we'll simultaneously make it that much harder to refactor and change the driver in the future.
> >
> > I am still pushing to see refactoring and design work on the *existing* use cases the driver supports before extending the use cases. I don't know how to support cross compilation for more and more diverse platforms prior to getting cross compilation for very basic platforms, or even non-cross compilation into a better state.
>
> > Consider that system header search logic for Darwin, MinGW, and Cygwin is still largely implemented in the Frontend rather than the Driver. This is something I'm actively working on for reference...
>
> Indeed, and this is why it has taken so long since the conference to get something solid (apart from other work). Not only do we need to work out an end goal but make it achievable in small reviewable steps.
>
> I've studied the internals of the driver and thought of several different ways of factoring it, including a composable "pass" framework where arguments get successively modified by composable passes. That had some promise, as the baked in behaviour could be completely controlled at runtime easily, but was a no go because, and this is the major part:
>
> I can't find a way of validating that a gigantic refactor makes no functional change in the driver. The regression tests aren't sufficient, and I'm likely to break Darwin or some other target with a huge refactor - Tools.cpp for example contains 5200 lines, some of which are common and others not.
>
> Add to that the fact that after many iterations I still come back to the current driver design as "not broken". There's nothing wrong with the concept of Tools and ToolChains - in fact as an abstraction they suit reality well.
>
> The main thing I see being the problem is the use of subclassing to parameterise the Tool classes. Because they weren't designed for parameterisation to start with, people have also copypasta'd huge chunks of code around. There are at least 5 different functions that can driver "ld" or "as", for example, each subtly different because one or two have had bugfixes, some have trashed behaviour they don't support, etc etc.
>
> So here's my general "vision":
>
>  * A subclass of Tool will relate solely to the command it is driving/producing, not OS/Arch specific configuration thereof. For example, "binutils::As", "binutils::Ld", "gcc::Compile", "gcc::Link", "gcc::Assemble", "visualstudio::Link".
>   * These tools will have a parameter "std::vector<std::string> ExtraArgs", which is a list of extra arguments to give to the tool. This will be created elsewhere.
>   * I have yet to work out where Darwin will fit here - ideally I'd like to have Darwin do all its funky logic and stick it all in ExtraArgs then go independent from there, but I don't know the best solution.
>
>  * A target should be able to select any tool for any JobAction. This makes hard-baked ToolChains superfluous. You shouldn't have to subclass ToolChain for your target, because it will be dynamically generated by...
>
>  * The "target database". I think this should be able to parameterise the Tools in any way required - all OS-specific stuff (With the exception of Darwin - that probably requires too much imperative code) should be in the DB.
>  * This can take two forms - hard-baked and JSON. The hard-baked version I see being a tablegen file similar (as possible) to the JSON representation, which is compiled into Clang for speed.
>  * This way, we keep the speed and extensibility and channel them both through the same interface, so that anything you can do hard-coded you can also change at runtime.
>
>
> So here's my migration plan:
>
>  1. The target database is where all the current imperative configuration should be factored out to. Create an initial draft schema, a ToolChain/HostInfo that uses it. At this point I suggest only using JSON as this will be easier to change should the schema change than a tablegen backend. The tablegen backend can be added later and the JSON data ported over for speed.
>  2. Create the first of the "properly independent" Tools - binutils::Ld and binutils::As, and use the target database to parameterise them.

This will lead to clang using ld to link directly, removing dependency on gcc to link, which is a good thing. This might require some form of versioning for newer features being added to newer ld versions.

>    * Probably first patch checkin point? Use a new driver debug flag to enable the new behaviour -ccc-dynamic-driver.
>  3. Port more ToolChains to the target database. For linux, we'd need to keep the distro detection logic outside the targetdb, but then we shouldn't need clever header detection methods as we can bake the expected header locations for a given distro into the target database.

This would keep every Linux distro and version in the clang codebase. Is it not preferred to have at least header search dirs moved out of the code, to some configure flag or conf file? True Linux cross-compilation may not want to use native system headers, but some specific to a target.

>  4. Sort out what we're doing with Darwin. Is it having its own set of Tools and living in its own domain, or is it linked to the independent tools?
>  5. The Big Switchover, at which point we can remove ideally around 4000 lines in Tools.cpp and 90% of ToolChains.cpp (probably also HostInfo.cpp) and end up with a driver which is centrally configurable both at compile and runtime.

Don't forget initheadersearch,  that's where most of the cruft is located.

These are just my two cents...

Ruben

>
>
> OK, so there's a full braindump. I was going to throw this up for discussion in true LLVM style - "with a patch" - in a week or so but my hand has been pushed ;)
>
> Note that this doesn't address the parsing logic disparity between Driver and Frontend - that's not my aim. I'm hoping to "fix the driver for cross-compilation", not fix the entire driver. I'm hoping someone else might chip in there!
>
> Let the heckling commence! ;)
>
> Cheers,
>
> James
>
>
>
> From: Chandler Carruth [mailto:[hidden email]]
> Sent: 10 January 2012 09:24
> To: James Molloy
> Cc: Sebastian Pop; [hidden email] Developers; [hidden email]
> Subject: Re: [cfe-dev] First shot at Bug 4127 - The clang driver should support cross compilation
>
> On Tue, Jan 10, 2012 at 12:25 AM, James Molloy <[hidden email]> wrote:
> As I say, I'm working on a patch that I think is a superset of yours and would conflict massively. I've been planning it for some time and think I have a viable end goal and route to get there.
>
> It would be really nice to have smaller patches rather than larger patches, and earlier discussion of them.
>
> Again, I remain very concerned about doing lots of work around configuration files to configure a *broken* driver design. I think we'll just end up with broken config file designs as well, and we'll simultaneously make it that much harder to refactor and change the driver in the future.
>
> I am still pushing to see refactoring and design work on the *existing* use cases the driver supports before extending the use cases. I don't know how to support cross compilation for more and more diverse platforms prior to getting cross compilation for very basic platforms, or even non-cross compilation into a better state.
>
> Consider that system header search logic for Darwin, MinGW, and Cygwin is still largely implemented in the Frontend rather than the Driver. This is something I'm actively working on for reference...
>
> -- IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium.  Thank you.
>
> _______________________________________________
> cfe-dev mailing list
> [hidden email]
> http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev


_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: First shot at Bug 4127 - The clang driver should support cross compilation

Konstantin Tokarev
In reply to this post by David Chisnall


10.01.2012, 14:41, "David Chisnall" <[hidden email]>:
> On 10 Jan 2012, at 10:00, James Molloy wrote:
>
>>   * This can take two forms - hard-baked and JSON. The hard-baked version I see being a tablegen file similar (as possible) to the JSON representation, which is compiled into Clang for speed.
>
> Is there any reason why you can't create a tool that parses the JSON and emits a C++ class so that it can be exactly the same for the cases that don't require any custom executable logic?  This would make prototyping new platforms a lot easier, since it would mean that you'd just poke the JSON until it worked and then add it to the list of JSON files to be transformed into C++ in the default build when it was ready (and people shipping clang for their obscure platform could just add it as part of their default build, even if it were of no interest to anyone else).

What about custom cross-toolchain setups? They would require recompilation?

--
Regards,
Konstantin
_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: First shot at Bug 4127 - The clang driver should support cross compilation

James Molloy-2
Custom setups have two options in this case.

They must add their setup to the JSON target DB. If they want to, then can then recompile and clang would parse that and create C++ classes to do the work. If not, they could distribute that JSON file and Clang would read it at runtime and do the same job, just slightly slower.

-----Original Message-----
From: Konstantin Tokarev [mailto:[hidden email]]
Sent: 10 January 2012 11:34
To: David Chisnall
Cc: James Molloy; Sebastian Pop; [hidden email] Developers; [hidden email]
Subject: Re: [cfe-dev] First shot at Bug 4127 - The clang driver should support cross compilation



10.01.2012, 14:41, "David Chisnall" <[hidden email]>:
> On 10 Jan 2012, at 10:00, James Molloy wrote:
>
>>   * This can take two forms - hard-baked and JSON. The hard-baked version I see being a tablegen file similar (as possible) to the JSON representation, which is compiled into Clang for speed.
>
> Is there any reason why you can't create a tool that parses the JSON and emits a C++ class so that it can be exactly the same for the cases that don't require any custom executable logic?  This would make prototyping new platforms a lot easier, since it would mean that you'd just poke the JSON until it worked and then add it to the list of JSON files to be transformed into C++ in the default build when it was ready (and people shipping clang for their obscure platform could just add it as part of their default build, even if it were of no interest to anyone else).

What about custom cross-toolchain setups? They would require recompilation?

--
Regards,
Konstantin


-- IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium.  Thank you.


_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: First shot at Bug 4127 - The clang driver should support cross compilation

James Molloy-2
In reply to this post by Ruben Van Boxem
> This will lead to clang using ld to link directly, removing dependency on gcc to link, which is a good thing. This might require some form of versioning for newer features being added to newer ld versions.

Yes, although you'd still maintain the option of using GCC instead. The idea is that now it'll be more easily interchangeable, and also Ld/As are simpler to port to the new concept as a starting point rather than the GCC compat class :)

> This would keep every Linux distro and version in the clang codebase. Is it not preferred to have at least header search dirs moved out of the code, to some configure flag or conf file? True Linux cross-compilation may not want to use native system headers, but some specific to a target.

It would, for the sole reason that distro/version detection is complex and requires imperative code. It isn't easy to specify in a config file how to detect if a box is a specific distro/version.

Not to mention separation of concerns - I see the target database as "I'm compiling for X. What do I do?" rather than "What am I compiling for?". I see these as two separate concerns which should be handled separately.

> Don't forget initheadersearch,  that's where most of the cruft is located.

Indeed :( Chandler appears to care more about this, so I'm happy to leave this to him...
These are just my two cents...
Ruben

From: Ruben Van Boxem [mailto:[hidden email]]
Sent: 10 January 2012 11:32
To: James Molloy
Cc: [hidden email] Developers; Sebastian Pop; Chandler Carruth; [hidden email]
Subject: Re: [cfe-dev] First shot at Bug 4127 - The clang driver should support cross compilation

Op 10 jan. 2012 11:50 schreef "James Molloy" <[hidden email]> het volgende:

>
> > It would be really nice to have smaller patches rather than larger patches, and earlier discussion of them.
> >
> > Again, I remain very concerned about doing lots of work around configuration files to configure a *broken* driver design. I think we'll just end up > with broken config file designs as well, and we'll simultaneously make it that much harder to refactor and change the driver in the future.
> >
> > I am still pushing to see refactoring and design work on the *existing* use cases the driver supports before extending the use cases. I don't know how to support cross compilation for more and more diverse platforms prior to getting cross compilation for very basic platforms, or even non-cross compilation into a better state.
>
> > Consider that system header search logic for Darwin, MinGW, and Cygwin is still largely implemented in the Frontend rather than the Driver. This is something I'm actively working on for reference...
>
> Indeed, and this is why it has taken so long since the conference to get something solid (apart from other work). Not only do we need to work out an end goal but make it achievable in small reviewable steps.
>
> I've studied the internals of the driver and thought of several different ways of factoring it, including a composable "pass" framework where arguments get successively modified by composable passes. That had some promise, as the baked in behaviour could be completely controlled at runtime easily, but was a no go because, and this is the major part:
>
> I can't find a way of validating that a gigantic refactor makes no functional change in the driver. The regression tests aren't sufficient, and I'm likely to break Darwin or some other target with a huge refactor - Tools.cpp for example contains 5200 lines, some of which are common and others not.
>
> Add to that the fact that after many iterations I still come back to the current driver design as "not broken". There's nothing wrong with the concept of Tools and ToolChains - in fact as an abstraction they suit reality well.
>
> The main thing I see being the problem is the use of subclassing to parameterise the Tool classes. Because they weren't designed for parameterisation to start with, people have also copypasta'd huge chunks of code around. There are at least 5 different functions that can driver "ld" or "as", for example, each subtly different because one or two have had bugfixes, some have trashed behaviour they don't support, etc etc.
>
> So here's my general "vision":
>
>  * A subclass of Tool will relate solely to the command it is driving/producing, not OS/Arch specific configuration thereof. For example, "binutils::As", "binutils::Ld", "gcc::Compile", "gcc::Link", "gcc::Assemble", "visualstudio::Link".
>   * These tools will have a parameter "std::vector<std::string> ExtraArgs", which is a list of extra arguments to give to the tool. This will be created elsewhere.
>   * I have yet to work out where Darwin will fit here - ideally I'd like to have Darwin do all its funky logic and stick it all in ExtraArgs then go independent from there, but I don't know the best solution.
>
>  * A target should be able to select any tool for any JobAction. This makes hard-baked ToolChains superfluous. You shouldn't have to subclass ToolChain for your target, because it will be dynamically generated by...
>
>  * The "target database". I think this should be able to parameterise the Tools in any way required - all OS-specific stuff (With the exception of Darwin - that probably requires too much imperative code) should be in the DB.
>  * This can take two forms - hard-baked and JSON. The hard-baked version I see being a tablegen file similar (as possible) to the JSON representation, which is compiled into Clang for speed.
>  * This way, we keep the speed and extensibility and channel them both through the same interface, so that anything you can do hard-coded you can also change at runtime.
>
>
> So here's my migration plan:
>
>  1. The target database is where all the current imperative configuration should be factored out to. Create an initial draft schema, a ToolChain/HostInfo that uses it. At this point I suggest only using JSON as this will be easier to change should the schema change than a tablegen backend. The tablegen backend can be added later and the JSON data ported over for speed.
>  2. Create the first of the "properly independent" Tools - binutils::Ld and binutils::As, and use the target database to parameterise them.
This will lead to clang using ld to link directly, removing dependency on gcc to link, which is a good thing. This might require some form of versioning for newer features being added to newer ld versions.
>    * Probably first patch checkin point? Use a new driver debug flag to enable the new behaviour -ccc-dynamic-driver.
>  3. Port more ToolChains to the target database. For linux, we'd need to keep the distro detection logic outside the targetdb, but then we shouldn't need clever header detection methods as we can bake the expected header locations for a given distro into the target database.
This would keep every Linux distro and version in the clang codebase. Is it not preferred to have at least header search dirs moved out of the code, to some configure flag or conf file? True Linux cross-compilation may not want to use native system headers, but some specific to a target.
>  4. Sort out what we're doing with Darwin. Is it having its own set of Tools and living in its own domain, or is it linked to the independent tools?
>  5. The Big Switchover, at which point we can remove ideally around 4000 lines in Tools.cpp and 90% of ToolChains.cpp (probably also HostInfo.cpp) and end up with a driver which is centrally configurable both at compile and runtime.
Don't forget initheadersearch,  that's where most of the cruft is located.
These are just my two cents...
Ruben

>
>
> OK, so there's a full braindump. I was going to throw this up for discussion in true LLVM style - "with a patch" - in a week or so but my hand has been pushed ;)
>
> Note that this doesn't address the parsing logic disparity between Driver and Frontend - that's not my aim. I'm hoping to "fix the driver for cross-compilation", not fix the entire driver. I'm hoping someone else might chip in there!
>
> Let the heckling commence! ;)
>
> Cheers,
>
> James
>
>
>
> From: Chandler Carruth [mailto:[hidden email]]
> Sent: 10 January 2012 09:24
> To: James Molloy
> Cc: Sebastian Pop; [hidden email] Developers; [hidden email]
> Subject: Re: [cfe-dev] First shot at Bug 4127 - The clang driver should support cross compilation
>
> On Tue, Jan 10, 2012 at 12:25 AM, James Molloy <[hidden email]> wrote:
> As I say, I'm working on a patch that I think is a superset of yours and would conflict massively. I've been planning it for some time and think I have a viable end goal and route to get there.
>
> It would be really nice to have smaller patches rather than larger patches, and earlier discussion of them.
>
> Again, I remain very concerned about doing lots of work around configuration files to configure a *broken* driver design. I think we'll just end up with broken config file designs as well, and we'll simultaneously make it that much harder to refactor and change the driver in the future.
>
> I am still pushing to see refactoring and design work on the *existing* use cases the driver supports before extending the use cases. I don't know how to support cross compilation for more and more diverse platforms prior to getting cross compilation for very basic platforms, or even non-cross compilation into a better state.
>
> Consider that system header search logic for Darwin, MinGW, and Cygwin is still largely implemented in the Frontend rather than the Driver. This is something I'm actively working on for reference...
>
> -- IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium.  Thank you.
>
> _______________________________________________
> cfe-dev mailing list
> [hidden email]
> http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev

-- IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium.  Thank you.

_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: First shot at Bug 4127 - The clang driver should support cross compilation

Joerg Sonnenberger
In reply to this post by Rafael Espíndola
On Mon, Jan 09, 2012 at 05:24:45PM -0500, Rafael Ávila de Espíndola wrote:

> On 09/01/12 04:46 PM, Joerg Sonnenberger wrote:
> > On Mon, Jan 09, 2012 at 03:00:30PM -0600, Sebastian Pop wrote:
> >> I would like the default config file to be created by the configure scripts,
> >> and all the logic that is currently executed at each clang compilation
> >> to be moved to configure time.
> >
> > This is the part I object to. Please do not make cross-compilation more
> > painful than it has to be. It works well for some of us without any new
> > movable parts, just by specifying the right target and --sysroot.
>
> So this would cut the number of options you have to use in halve :-)

No, it doesn't. It just means that I have to use one option and create
one file for each possible combination. That doesn't reduce the amount
of work...

Joerg
_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: First shot at Bug 4127 - The clang driver should support cross compilation

Sebastian Pop
In reply to this post by Chandler Carruth
On Mon, Jan 9, 2012 at 4:43 PM, Chandler Carruth <[hidden email]> wrote:

> On Mon, Jan 9, 2012 at 1:46 PM, Joerg Sonnenberger <[hidden email]>
> wrote:
>>
>> On Mon, Jan 09, 2012 at 03:00:30PM -0600, Sebastian Pop wrote:
>> > I would like the default config file to be created by the configure
>> > scripts,
>> > and all the logic that is currently executed at each clang compilation
>> > to be moved to configure time.
>>
>> This is the part I object to. Please do not make cross-compilation more
>> painful than it has to be. It works well for some of us without any new
>> movable parts, just by specifying the right target and --sysroot.
>
>
> I completely agree. I'm really opposed to removing baked-in support for the
> most common platforms. This should be about extending our ability to support
> less common platforms more easily.

Ok.  I don't have a strong opinion on having this detection at configure time,
though I would like clang to run this detection code only once, and not every
time clang is running.

> Also, I'm not really excited about starting to attack this problem from this
> direction. I think the first steps need to be refactoring and rationalizing
> the existing code in the driver to have a better design and structure,
> before we start extending it to support referencing an external data file to
> collect information. Otherwise I fear we will embed the wrong structure or
> information in the data file, and use these data files as crutches to avoid
> properly designing the internals.

I will of course let the clang maintainers lead the refactoring of the driver
and decide what makes sense to be saved in the target config files.

Sebastian
--
Qualcomm Innovation Center, Inc is a member of Code Aurora Forum
_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
12