Configuration files

classic Classic list List threaded Threaded
45 messages Options
123
Reply | Threaded
Open this post in threaded view
|

Re: Configuration files

James Y Knight via cfe-dev
2017-03-16 9:46 GMT+07:00 James Y Knight <[hidden email]>:
 
I'd really like to at least have a *design* for how this can eventually incorporate target-specific options before moving forward with adding a --config option, even if the initial commit won't implement the full design.

I don't believe hand-wavy "maybe we'll add syntax that looks kinda like a comment so older versions will ignore it" is good enough there.

I'd like to again keep in mind the use-case I mentioned a while ago. Approximately every linux distro configures GCC to set their default target cpu levels. E.g., Debian seems to set the following:
- On x86, the default CPU should be i686.
- But on x86-64, the default CPU isn't changed (aka it's left as "x86-64").
- For ppc32/ppc64, the default CPU should be power7 but tune for power8.
- For ppc64le, the default CPU should be power8.
- On ARM (hf), armv7-a should be the default cpu, vfpv3-d16 the default fpu, and it should default to thumb mode.
etc...

Note that those defaults are different on different releases of the distro.

The way you do this with GCC is via options passed to the configure script: --with-arch-32= --with-arch-64= --with-fpu= --with-mode= etc. which turn into values used in the target-specific OPTION_DEFAULT_SPECS macro. Since GCC only builds for one target at a time (or two, if you count 32/64 separately), and you're expected to need to build a new gcc any time you want to cross-compile, that's sufficient.

Clang is intrinsically a cross-compiler, so gcc's solution isn't good enough for clang (nor is clang's current behavior enough). So, what's the plan to actually solve this?

Initial versions of this proposal defined two kinds of config files:
- named, which should be explicitly specified by a user by option --config or be encoded into executable name as `armv7l-clang`.
- default, which is loaded always much like `.bashrc` or any similar file.
These two kinds of config file shared implementation but addressed different use cases, which made confusion during discussion. To facilitate review process the support of default config files was removed from the proposal. The issues you mention should mostly be solved by the default config files.

If the default config files were implemented in clang, driver would search binary directory for default configuration description. If compiler is named as `armv7l-clang`, driver first tries file `armv7l.cfg` then `clang.cfg`. If config file is found, options listed there are put into the set of driver arguments before any option specified in command line.

With this feature a distribution or SDK can supply set of config files as a part of clang package and tunes compiler appropriately.


1. There needs to a way to be able to configure the defaults for all supported architectures of the platform (either in a single config, or in multiple that clang knows how to select).


Each supported target can have separate config file.
 
2. Those platform defaults should, somehow, avoid interfering with the use of clang to cross-compile TO a different platform, and be easy to use FROM a different host platform. (find default config via sysroot, maybe?)


Default config in sysroot could be included by default clang config, however driver must know where the sysroot is. We could support a set of macros, for instance expand `$TARGET` in config file to target name as  `armv7l-clang`. This topic is not elaborated yet.
 
3. How do we recommend users select a target?
Do we need to look for the proper defaults when the user specifies "--target $TARGET" to clang?
Or maybe we favor of the "$TARGET-clang" symlink method?
Or maybe "--target" is a low-level feature not recommended for end-users, and we should steer people to using something like "-arch", to select a named architecture within the current platform configuration, like apple does with darwin-based platforms now?


To specify a target looks like a more flexible solution. "$TARGET-clang" symlink method was already implemented in early versions of https://reviews.llvm.org/D24933. It is possible also to extend treatment of `--target` so that it acted similar to `--config`.

Thanks,
--Serge


_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: Configuration files

James Y Knight via cfe-dev


On 03/16/2017 08:25 AM, Serge Pavlov wrote:
2017-03-16 9:46 GMT+07:00 James Y Knight <[hidden email]>:
 
I'd really like to at least have a *design* for how this can eventually incorporate target-specific options before moving forward with adding a --config option, even if the initial commit won't implement the full design.

I don't believe hand-wavy "maybe we'll add syntax that looks kinda like a comment so older versions will ignore it" is good enough there.

To be clear, there is a current design for this. As Serge mentions below, we handle PREFIX-clang to search for config files based on PREFIX. PREFIX here is usually a target, but can be other things as well for local customization. This was specifically intended to work for cross compiling. That having been said, we may need to extend this...


I'd like to again keep in mind the use-case I mentioned a while ago. Approximately every linux distro configures GCC to set their default target cpu levels. E.g., Debian seems to set the following:
- On x86, the default CPU should be i686.
- But on x86-64, the default CPU isn't changed (aka it's left as "x86-64").
- For ppc32/ppc64, the default CPU should be power7 but tune for power8.
- For ppc64le, the default CPU should be power8.
- On ARM (hf), armv7-a should be the default cpu, vfpv3-d16 the default fpu, and it should default to thumb mode.
etc...

Note that those defaults are different on different releases of the distro.

The way you do this with GCC is via options passed to the configure script: --with-arch-32= --with-arch-64= --with-fpu= --with-mode= etc. which turn into values used in the target-specific OPTION_DEFAULT_SPECS macro. Since GCC only builds for one target at a time (or two, if you count 32/64 separately), and you're expected to need to build a new gcc any time you want to cross-compile, that's sufficient.

Clang is intrinsically a cross-compiler, so gcc's solution isn't good enough for clang (nor is clang's current behavior enough). So, what's the plan to actually solve this?

Initial versions of this proposal defined two kinds of config files:
- named, which should be explicitly specified by a user by option --config or be encoded into executable name as `armv7l-clang`.
- default, which is loaded always much like `.bashrc` or any similar file.
These two kinds of config file shared implementation but addressed different use cases, which made confusion during discussion. To facilitate review process the support of default config files was removed from the proposal. The issues you mention should mostly be solved by the default config files.

If the default config files were implemented in clang, driver would search binary directory for default configuration description. If compiler is named as `armv7l-clang`, driver first tries file `armv7l.cfg` then `clang.cfg`. If config file is found, options listed there are put into the set of driver arguments before any option specified in command line.

With this feature a distribution or SDK can supply set of config files as a part of clang package and tunes compiler appropriately.


1. There needs to a way to be able to configure the defaults for all supported architectures of the platform (either in a single config, or in multiple that clang knows how to select).


Each supported target can have separate config file.
 
2. Those platform defaults should, somehow, avoid interfering with the use of clang to cross-compile TO a different platform, and be easy to use FROM a different host platform. (find default config via sysroot, maybe?)


Default config in sysroot could be included by default clang config, however driver must know where the sysroot is. We could support a set of macros, for instance expand `$TARGET` in config file to target name as  `armv7l-clang`. This topic is not elaborated yet.
 
3. How do we recommend users select a target?
Do we need to look for the proper defaults when the user specifies "--target $TARGET" to clang?
Or maybe we favor of the "$TARGET-clang" symlink method?
Or maybe "--target" is a low-level feature not recommended for end-users, and we should steer people to using something like "-arch", to select a named architecture within the current platform configuration, like apple does with darwin-based platforms now?


To specify a target looks like a more flexible solution. "$TARGET-clang" symlink method was already implemented in early versions of https://reviews.llvm.org/D24933. It is possible also to extend treatment of `--target` so that it acted similar to `--config`.

This seems like a good idea. Depending on how the "Linux distribution" use case actually works (which might also apply to cross-mounted file systems across different architectures?), we might also want to also do the search for the default target, even if none is explicitly provided, so that a distribution can ship a default set of files and the compiler will pick up the right one for the current system. James, does this correspond to what you had in mind?

Thanks again,
Hal


Thanks,
--Serge


-- 
Hal Finkel
Lead, Compiler Technology and Programming Languages
Leadership Computing Facility
Argonne National Laboratory

_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: Configuration files

James Y Knight via cfe-dev
In reply to this post by James Y Knight via cfe-dev


On Thu, Mar 16, 2017 at 9:25 AM, Serge Pavlov <[hidden email]> wrote:
2017-03-16 9:46 GMT+07:00 James Y Knight <[hidden email]>:
 
I'd really like to at least have a *design* for how this can eventually incorporate target-specific options before moving forward with adding a --config option, even if the initial commit won't implement the full design.

I don't believe hand-wavy "maybe we'll add syntax that looks kinda like a comment so older versions will ignore it" is good enough there.

I'd like to again keep in mind the use-case I mentioned a while ago. Approximately every linux distro configures GCC to set their default target cpu levels. E.g., Debian seems to set the following:
- On x86, the default CPU should be i686.
- But on x86-64, the default CPU isn't changed (aka it's left as "x86-64").
- For ppc32/ppc64, the default CPU should be power7 but tune for power8.
- For ppc64le, the default CPU should be power8.
- On ARM (hf), armv7-a should be the default cpu, vfpv3-d16 the default fpu, and it should default to thumb mode.
etc...

Note that those defaults are different on different releases of the distro.

The way you do this with GCC is via options passed to the configure script: --with-arch-32= --with-arch-64= --with-fpu= --with-mode= etc. which turn into values used in the target-specific OPTION_DEFAULT_SPECS macro. Since GCC only builds for one target at a time (or two, if you count 32/64 separately), and you're expected to need to build a new gcc any time you want to cross-compile, that's sufficient.

Clang is intrinsically a cross-compiler, so gcc's solution isn't good enough for clang (nor is clang's current behavior enough). So, what's the plan to actually solve this?

Initial versions of this proposal defined two kinds of config files:
- named, which should be explicitly specified by a user by option --config or be encoded into executable name as `armv7l-clang`.
- default, which is loaded always much like `.bashrc` or any similar file.
These two kinds of config file shared implementation but addressed different use cases, which made confusion during discussion. To facilitate review process the support of default config files was removed from the proposal. The issues you mention should mostly be solved by the default config files.

OK -- focusing on cross-compilation for a minute. Clang and LLD have the potential to make cross-compilation *MUCH* simpler than it's traditionally been, and I'd like to be careful to think about how to take the best advantage of the fact that we're getting really close to having a single toolchain which is able to easily build programs for any target architecture/environment.


Here's the scenario I'm using in my mind:

Let's say I've got a super-fast RISC-V machine running OpenBSD, and want to cross-build from that to all of the various supported Debian architectures. I want to use my existing OpenBSD-supplied clang+lld toolchain, because that toolchain can natively cross-compile to everything, if given appropriate command-line arguments. So, I don't need to make a special build of the compiler or linker. Woohoo!

I have copied over the root partition of the target Debian system into a directory on this machine, so I have headers and libraries to link against.

(I also need the crtbegin/end and libgcc or compiler-rt. For this thread, let's assume I copied those over as well, even though they're typically "part of" the compiler toolchain suite. A separate discussion should be had about what to do about making THAT part of the cross-compile story easier -- it's almost the only tricky part remaining!)

Now, given that scenario --

How do I specify that I want to cross-build to that debian system?

I need to know the proper configuration, and I need to know how I'm supposed to invoke clang to use the configuration.

At the moment, you might run something like:
 "clang --target x86_64-linux-gnu --sysroot=/path/to/debian-installation --other-super-important-cpu/abi-flags"

With your proposed patch, I would create a config file, "x86_64-debian.cfg", containing the above flags. I could put it in /etc/something or ~/.something, and run:
$ clang -config x86_64-debian

Alternatively, I could put it in ~/bin/x86_64-debian.cfg and
$ ln -s /usr/bin/clang ~/bin/x86_64-debian-clang
$ x86_64-debian-clang

If the default config files were implemented in clang, driver would search binary directory for default configuration description. If compiler is named as `armv7l-clang`, driver first tries file `armv7l.cfg` then `clang.cfg`. If config file is found, options listed there are put into the set of driver arguments before any option specified in command line.
 
With this feature a distribution or SDK can supply set of config files as a part of clang package and tunes compiler appropriately.
 
1. There needs to a way to be able to configure the defaults for all supported architectures of the platform (either in a single config, or in multiple that clang knows how to select).


Each supported target can have separate config file.

But, despite the target being x86_64, some software builds 32-bit x86 code, and will run "x86_64-debian-clang -m32" to do so. In that case, where does clang get the appropriate default x86 cpu from? (Note again that GCC compiles-in separate defaults for the 32 and 64-bit architecture variants to handle this).

If it's the case that each supported target has its own config file, clang will somehow need to know how to choose the "i386-debian.cfg" file when invoked as "x86_64-debian-clang -m32"....or something like that.

2. Those platform defaults should, somehow, avoid interfering with the use of clang to cross-compile TO a different platform, and be easy to use FROM a different host platform. (find default config via sysroot, maybe?)


Default config in sysroot could be included by default clang config, however driver must know where the sysroot is. We could support a set of macros, for instance expand `$TARGET` in config file to target name as  `armv7l-clang`. This topic is not elaborated yet.

So -- I should've separated out the two points in my comment above.

2.1 (TO different): What happens if you do something like "x86_64-debian-clang -target armv7a-apple-darwin". Is that an error? Or, if not, what does it do? Does it still use the x86_64-debian-clang config file? Or, does it completely ignore the config file?

This problem isn't so major for explicitly-specified configs ("Don't do that!"), but becomes a larger issue if there's a default config file.

This is why I was pondering if -target perhaps should not be the recommended way for end-users to select an architecture. The "-target" flag could be made to mean "ignore the platform configuration, just use this target with clang defaults". And, the "-arch" flag could mean: "within the current configuration, select this named architecture configuration".

2.2 (FROM different): Default config files could be searched for inside the specified sysroot. That is, "clang --sysroot=/wherever/" would find the proper configs for the platform automatically, if they've been distributed in the sysroot. The canonical way to cross-build with clang would be to use your existing compiler, and specify two additional arguments: --sysroot and -target (or -arch?).

3. How do we recommend users select a target?
Do we need to look for the proper defaults when the user specifies "--target $TARGET" to clang?
Or maybe we favor of the "$TARGET-clang" symlink method?
Or maybe "--target" is a low-level feature not recommended for end-users, and we should steer people to using something like "-arch", to select a named architecture within the current platform configuration, like apple does with darwin-based platforms now?


To specify a target looks like a more flexible solution. "$TARGET-clang" symlink method was already implemented in early versions of https://reviews.llvm.org/D24933. It is possible also to extend treatment of `--target` so that it acted similar to `--config`.

I note that Apple has hardcoded behavior in clang for their targets, that lets it do something sensible when specifying the sysroot:

That is, you can do:
$ clang -isysroot /path/to/iPhoneOS9.3.sdk -arch arm64
[Note, the "i" in the "isysroot" in that commandline there is correct, but a presumably-historical bogosity. It actually finds libraries in the specified "i"sysroot, too.]

Given the above, it:
1. Sets the target platform to ios9.3 (*-apple-ios9.3.0), based on matching the pathname specified by isysroot.
2. Sets the target to aarch64 (printed arm64-apple-watchos2.2.0), based on mapping the -arch name back to a triple.
3. Also sets the default CPU to "cyclone". (-target-cpu cyclone), based on hardcoded behavior for "-arch arm64".

That seems a pretty good example of what I'm talking about, except it's been implemented with driver hacks instead of a generic config file support.

_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: Configuration files

James Y Knight via cfe-dev
2017-03-20 12:48 GMT+07:00 James Y Knight <[hidden email]>:
On Thu, Mar 16, 2017 at 9:25 AM, Serge Pavlov <[hidden email]> wrote:
2017-03-16 9:46 GMT+07:00 James Y Knight <[hidden email]>:
 
I'd really like to at least have a *design* for how this can eventually incorporate target-specific options before moving forward with adding a --config option, even if the initial commit won't implement the full design.

I don't believe hand-wavy "maybe we'll add syntax that looks kinda like a comment so older versions will ignore it" is good enough there.

I'd like to again keep in mind the use-case I mentioned a while ago. Approximately every linux distro configures GCC to set their default target cpu levels. E.g., Debian seems to set the following:
- On x86, the default CPU should be i686.
- But on x86-64, the default CPU isn't changed (aka it's left as "x86-64").
- For ppc32/ppc64, the default CPU should be power7 but tune for power8.
- For ppc64le, the default CPU should be power8.
- On ARM (hf), armv7-a should be the default cpu, vfpv3-d16 the default fpu, and it should default to thumb mode.
etc...

Note that those defaults are different on different releases of the distro.

The way you do this with GCC is via options passed to the configure script: --with-arch-32= --with-arch-64= --with-fpu= --with-mode= etc. which turn into values used in the target-specific OPTION_DEFAULT_SPECS macro. Since GCC only builds for one target at a time (or two, if you count 32/64 separately), and you're expected to need to build a new gcc any time you want to cross-compile, that's sufficient.

Clang is intrinsically a cross-compiler, so gcc's solution isn't good enough for clang (nor is clang's current behavior enough). So, what's the plan to actually solve this?

Initial versions of this proposal defined two kinds of config files:
- named, which should be explicitly specified by a user by option --config or be encoded into executable name as `armv7l-clang`.
- default, which is loaded always much like `.bashrc` or any similar file.
These two kinds of config file shared implementation but addressed different use cases, which made confusion during discussion. To facilitate review process the support of default config files was removed from the proposal. The issues you mention should mostly be solved by the default config files.

OK -- focusing on cross-compilation for a minute. Clang and LLD have the potential to make cross-compilation *MUCH* simpler than it's traditionally been, and I'd like to be careful to think about how to take the best advantage of the fact that we're getting really close to having a single toolchain which is able to easily build programs for any target architecture/environment.

Here's the scenario I'm using in my mind:

Let's say I've got a super-fast RISC-V machine running OpenBSD, and want to cross-build from that to all of the various supported Debian architectures. I want to use my existing OpenBSD-supplied clang+lld toolchain, because that toolchain can natively cross-compile to everything, if given appropriate command-line arguments. So, I don't need to make a special build of the compiler or linker. Woohoo!

I have copied over the root partition of the target Debian system into a directory on this machine, so I have headers and libraries to link against.

(I also need the crtbegin/end and libgcc or compiler-rt. For this thread, let's assume I copied those over as well, even though they're typically "part of" the compiler toolchain suite. A separate discussion should be had about what to do about making THAT part of the cross-compile story easier -- it's almost the only tricky part remaining!)

Now, given that scenario --

How do I specify that I want to cross-build to that debian system?

I need to know the proper configuration, and I need to know how I'm supposed to invoke clang to use the configuration.

At the moment, you might run something like:
 "clang --target x86_64-linux-gnu --sysroot=/path/to/debian-installation --other-super-important-cpu/abi-flags"

With your proposed patch, I would create a config file, "x86_64-debian.cfg", containing the above flags. I could put it in /etc/something or ~/.something, and run:
$ clang -config x86_64-debian

Alternatively, I could put it in ~/bin/x86_64-debian.cfg and
$ ln -s /usr/bin/clang ~/bin/x86_64-debian-clang
$ x86_64-debian-clang


In general you are right I think.
 
If the default config files were implemented in clang, driver would search binary directory for default configuration description. If compiler is named as `armv7l-clang`, driver first tries file `armv7l.cfg` then `clang.cfg`. If config file is found, options listed there are put into the set of driver arguments before any option specified in command line.
 
With this feature a distribution or SDK can supply set of config files as a part of clang package and tunes compiler appropriately.
 
1. There needs to a way to be able to configure the defaults for all supported architectures of the platform (either in a single config, or in multiple that clang knows how to select).


Each supported target can have separate config file.

But, despite the target being x86_64, some software builds 32-bit x86 code, and will run "x86_64-debian-clang -m32" to do so. In that case, where does clang get the appropriate default x86 cpu from? (Note again that GCC compiles-in separate defaults for the 32 and 64-bit architecture variants to handle this).


This is a real problem. In fact -m32 convert the specified target `x86_64-debian-clang` into `i686-debian-clang`. We cannot rely on the specified target as effective target can be different. This is the root of evil.

Simple solution as config file cannot solve this problem. We could:
- extend syntax of config file by allowing some sort of conditional directives, or
- add additional logic (target specific) that reloads config file if effective target changes. For instance, invocation of `x86_64-debian-clang -m32` first loads `x86_64-debian.cfg` which contains -target=..., which sets defaults for 64-bit target, then command line is scanned to calculate effective target, the option `-m32` changes the effective target to `i686-debian-clang`. Driver then removes all options read from config file and puts defaults from `i686-debian.cfg`. Only then the command line options are processed by regular logic. It sounds complex, it fact it can be easily implemented.

More complex solution such as spec file does not suffer this problem, it can check presence of options like `m32` and conditionally choose appropriate defaults.
 
If it's the case that each supported target has its own config file, clang will somehow need to know how to choose the "i386-debian.cfg" file when invoked as "x86_64-debian-clang -m32"....or something like that.

2. Those platform defaults should, somehow, avoid interfering with the use of clang to cross-compile TO a different platform, and be easy to use FROM a different host platform. (find default config via sysroot, maybe?)


Default config in sysroot could be included by default clang config, however driver must know where the sysroot is. We could support a set of macros, for instance expand `$TARGET` in config file to target name as  `armv7l-clang`. This topic is not elaborated yet.

So -- I should've separated out the two points in my comment above.

2.1 (TO different): What happens if you do something like "x86_64-debian-clang -target armv7a-apple-darwin". Is that an error? Or, if not, what does it do? Does it still use the x86_64-debian-clang config file? Or, does it completely ignore the config file?


I think multiple options `-target` must produce error. We now can call clang in such way: `clang -target x86_64-linux -target armv7-apple-ios6.0.0 abc.c` and clang successfully executes. Is there any use case for such target override?
 
This problem isn't so major for explicitly-specified configs ("Don't do that!"), but becomes a larger issue if there's a default config file.

This is why I was pondering if -target perhaps should not be the recommended way for end-users to select an architecture. The "-target" flag could be made to mean "ignore the platform configuration, just use this target with clang defaults". And, the "-arch" flag could mean: "within the current configuration, select this named architecture configuration".


It would mean that clang must somehow calculate other things as ABI.
 
2.2 (FROM different): Default config files could be searched for inside the specified sysroot. That is, "clang --sysroot=/wherever/" would find the proper configs for the platform automatically, if they've been distributed in the sysroot. The canonical way to cross-build with clang would be to use your existing compiler, and specify two additional arguments: --sysroot and -target (or -arch?).

3. How do we recommend users select a target?
Do we need to look for the proper defaults when the user specifies "--target $TARGET" to clang?
Or maybe we favor of the "$TARGET-clang" symlink method?
Or maybe "--target" is a low-level feature not recommended for end-users, and we should steer people to using something like "-arch", to select a named architecture within the current platform configuration, like apple does with darwin-based platforms now?


To specify a target looks like a more flexible solution. "$TARGET-clang" symlink method was already implemented in early versions of https://reviews.llvm.org/D24933. It is possible also to extend treatment of `--target` so that it acted similar to `--config`.

I note that Apple has hardcoded behavior in clang for their targets, that lets it do something sensible when specifying the sysroot:

That is, you can do:
$ clang -isysroot /path/to/iPhoneOS9.3.sdk -arch arm64
[Note, the "i" in the "isysroot" in that commandline there is correct, but a presumably-historical bogosity. It actually finds libraries in the specified "i"sysroot, too.]

Given the above, it:
1. Sets the target platform to ios9.3 (*-apple-ios9.3.0), based on matching the pathname specified by isysroot.
2. Sets the target to aarch64 (printed arm64-apple-watchos2.2.0), based on mapping the -arch name back to a triple.
3. Also sets the default CPU to "cyclone". (-target-cpu cyclone), based on hardcoded behavior for "-arch arm64".

That seems a pretty good example of what I'm talking about, except it's been implemented with driver hacks instead of a generic config file support.

That would be one more way to specify config file, in addition to option `--config` and symlink `x86_64-debian-clang`. Does such way have advantage over these ways?
$ clang --config /path/to/iPhoneOS9.3.sdk.cfg
or
$ clang --config iPhoneOS9.3.sdk.cfg

Sysroot directories could contain config files that set defaults appropriate for this SDK. For instance, directory /path/to/iPhoneOS9.3.sdk contains file platform.cfg. Config file loaded by clang could contain command `@/path/to/iPhoneOS9.3.sdk/platform.cfg`. That could be used to separate per-SDK defaults from other defaults.

I don't object to such use case, we just need to find balance between solution complexity, user convenience and maintainability.

Thanks,
--Serge

_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: Configuration files

James Y Knight via cfe-dev


On 03/20/2017 07:19 AM, Serge Pavlov wrote:
2017-03-20 12:48 GMT+07:00 James Y Knight <[hidden email]>:
On Thu, Mar 16, 2017 at 9:25 AM, Serge Pavlov <[hidden email]> wrote:
2017-03-16 9:46 GMT+07:00 James Y Knight <[hidden email]>:
 
I'd really like to at least have a *design* for how this can eventually incorporate target-specific options before moving forward with adding a --config option, even if the initial commit won't implement the full design.

I don't believe hand-wavy "maybe we'll add syntax that looks kinda like a comment so older versions will ignore it" is good enough there.

I'd like to again keep in mind the use-case I mentioned a while ago. Approximately every linux distro configures GCC to set their default target cpu levels. E.g., Debian seems to set the following:
- On x86, the default CPU should be i686.
- But on x86-64, the default CPU isn't changed (aka it's left as "x86-64").
- For ppc32/ppc64, the default CPU should be power7 but tune for power8.
- For ppc64le, the default CPU should be power8.
- On ARM (hf), armv7-a should be the default cpu, vfpv3-d16 the default fpu, and it should default to thumb mode.
etc...

Note that those defaults are different on different releases of the distro.

The way you do this with GCC is via options passed to the configure script: --with-arch-32= --with-arch-64= --with-fpu= --with-mode= etc. which turn into values used in the target-specific OPTION_DEFAULT_SPECS macro. Since GCC only builds for one target at a time (or two, if you count 32/64 separately), and you're expected to need to build a new gcc any time you want to cross-compile, that's sufficient.

Clang is intrinsically a cross-compiler, so gcc's solution isn't good enough for clang (nor is clang's current behavior enough). So, what's the plan to actually solve this?

Initial versions of this proposal defined two kinds of config files:
- named, which should be explicitly specified by a user by option --config or be encoded into executable name as `armv7l-clang`.
- default, which is loaded always much like `.bashrc` or any similar file.
These two kinds of config file shared implementation but addressed different use cases, which made confusion during discussion. To facilitate review process the support of default config files was removed from the proposal. The issues you mention should mostly be solved by the default config files.

OK -- focusing on cross-compilation for a minute. Clang and LLD have the potential to make cross-compilation *MUCH* simpler than it's traditionally been, and I'd like to be careful to think about how to take the best advantage of the fact that we're getting really close to having a single toolchain which is able to easily build programs for any target architecture/environment.

Here's the scenario I'm using in my mind:

Let's say I've got a super-fast RISC-V machine running OpenBSD, and want to cross-build from that to all of the various supported Debian architectures. I want to use my existing OpenBSD-supplied clang+lld toolchain, because that toolchain can natively cross-compile to everything, if given appropriate command-line arguments. So, I don't need to make a special build of the compiler or linker. Woohoo!

I have copied over the root partition of the target Debian system into a directory on this machine, so I have headers and libraries to link against.

(I also need the crtbegin/end and libgcc or compiler-rt. For this thread, let's assume I copied those over as well, even though they're typically "part of" the compiler toolchain suite. A separate discussion should be had about what to do about making THAT part of the cross-compile story easier -- it's almost the only tricky part remaining!)

Now, given that scenario --

How do I specify that I want to cross-build to that debian system?

I need to know the proper configuration, and I need to know how I'm supposed to invoke clang to use the configuration.

At the moment, you might run something like:
 "clang --target x86_64-linux-gnu --sysroot=/path/to/debian-installation --other-super-important-cpu/abi-flags"

With your proposed patch, I would create a config file, "x86_64-debian.cfg", containing the above flags. I could put it in /etc/something or ~/.something, and run:
$ clang -config x86_64-debian

Alternatively, I could put it in ~/bin/x86_64-debian.cfg and
$ ln -s /usr/bin/clang ~/bin/x86_64-debian-clang
$ x86_64-debian-clang


In general you are right I think.
 
If the default config files were implemented in clang, driver would search binary directory for default configuration description. If compiler is named as `armv7l-clang`, driver first tries file `armv7l.cfg` then `clang.cfg`. If config file is found, options listed there are put into the set of driver arguments before any option specified in command line.
 
With this feature a distribution or SDK can supply set of config files as a part of clang package and tunes compiler appropriately.
 
1. There needs to a way to be able to configure the defaults for all supported architectures of the platform (either in a single config, or in multiple that clang knows how to select).


Each supported target can have separate config file.

But, despite the target being x86_64, some software builds 32-bit x86 code, and will run "x86_64-debian-clang -m32" to do so. In that case, where does clang get the appropriate default x86 cpu from? (Note again that GCC compiles-in separate defaults for the 32 and 64-bit architecture variants to handle this).


This is a real problem. In fact -m32 convert the specified target `x86_64-debian-clang` into `i686-debian-clang`. We cannot rely on the specified target as effective target can be different. This is the root of evil.

Simple solution as config file cannot solve this problem. We could:
- extend syntax of config file by allowing some sort of conditional directives, or
- add additional logic (target specific) that reloads config file if effective target changes. For instance, invocation of `x86_64-debian-clang -m32` first loads `x86_64-debian.cfg` which contains -target=..., which sets defaults for 64-bit target, then command line is scanned to calculate effective target, the option `-m32` changes the effective target to `i686-debian-clang`. Driver then removes all options read from config file and puts defaults from `i686-debian.cfg`. Only then the command line options are processed by regular logic. It sounds complex, it fact it can be easily implemented.

I agree. This is an unfortunate special case. I also agree that proposed solution (gather command line arguments including from config files, check for -m32/64, discard as necessary and reprocess) is a relatively clean solution. I'll note that our triple class has get32BitArchVariant/get64BitArchVariant helpers which make adjusting the triple relatively easy.


More complex solution such as spec file does not suffer this problem, it can check presence of options like `m32` and conditionally choose appropriate defaults.
 
If it's the case that each supported target has its own config file, clang will somehow need to know how to choose the "i386-debian.cfg" file when invoked as "x86_64-debian-clang -m32"....or something like that.

2. Those platform defaults should, somehow, avoid interfering with the use of clang to cross-compile TO a different platform, and be easy to use FROM a different host platform. (find default config via sysroot, maybe?)


Default config in sysroot could be included by default clang config, however driver must know where the sysroot is. We could support a set of macros, for instance expand `$TARGET` in config file to target name as  `armv7l-clang`. This topic is not elaborated yet.

So -- I should've separated out the two points in my comment above.

2.1 (TO different): What happens if you do something like "x86_64-debian-clang -target armv7a-apple-darwin". Is that an error? Or, if not, what does it do? Does it still use the x86_64-debian-clang config file? Or, does it completely ignore the config file?


I think multiple options `-target` must produce error. We now can call clang in such way: `clang -target x86_64-linux -target armv7-apple-ios6.0.0 abc.c` and clang successfully executes. Is there any use case for such target override?

I agree. I think we should handle this in a manner consistent with our current processing (we can already take a triple from the executable prefix and a target command-line argument).

 
This problem isn't so major for explicitly-specified configs ("Don't do that!"), but becomes a larger issue if there's a default config file.

This is why I was pondering if -target perhaps should not be the recommended way for end-users to select an architecture. The "-target" flag could be made to mean "ignore the platform configuration, just use this target with clang defaults". And, the "-arch" flag could mean: "within the current configuration, select this named architecture configuration".


It would mean that clang must somehow calculate other things as ABI.
 
2.2 (FROM different): Default config files could be searched for inside the specified sysroot. That is, "clang --sysroot=/wherever/" would find the proper configs for the platform automatically, if they've been distributed in the sysroot. The canonical way to cross-build with clang would be to use your existing compiler, and specify two additional arguments: --sysroot and -target (or -arch?).

3. How do we recommend users select a target?
Do we need to look for the proper defaults when the user specifies "--target $TARGET" to clang?
Or maybe we favor of the "$TARGET-clang" symlink method?
Or maybe "--target" is a low-level feature not recommended for end-users, and we should steer people to using something like "-arch", to select a named architecture within the current platform configuration, like apple does with darwin-based platforms now?


To specify a target looks like a more flexible solution. "$TARGET-clang" symlink method was already implemented in early versions of https://reviews.llvm.org/D24933. It is possible also to extend treatment of `--target` so that it acted similar to `--config`.

I note that Apple has hardcoded behavior in clang for their targets, that lets it do something sensible when specifying the sysroot:

That is, you can do:
$ clang -isysroot /path/to/iPhoneOS9.3.sdk -arch arm64
[Note, the "i" in the "isysroot" in that commandline there is correct, but a presumably-historical bogosity. It actually finds libraries in the specified "i"sysroot, too.]

Given the above, it:
1. Sets the target platform to ios9.3 (*-apple-ios9.3.0), based on matching the pathname specified by isysroot.
2. Sets the target to aarch64 (printed arm64-apple-watchos2.2.0), based on mapping the -arch name back to a triple.
3. Also sets the default CPU to "cyclone". (-target-cpu cyclone), based on hardcoded behavior for "-arch arm64".

That seems a pretty good example of what I'm talking about, except it's been implemented with driver hacks instead of a generic config file support.

James, what exactly are you proposing in this regard? Do you want us to factor out the driver hack so that it can also be used by the logic which looks for config files? Or you're trying to replace that hack altogether with some kind of config-file-based setup?

 -Hal


That would be one more way to specify config file, in addition to option `--config` and symlink `x86_64-debian-clang`. Does such way have advantage over these ways?
$ clang --config /path/to/iPhoneOS9.3.sdk.cfg
or
$ clang --config iPhoneOS9.3.sdk.cfg

Sysroot directories could contain config files that set defaults appropriate for this SDK. For instance, directory /path/to/iPhoneOS9.3.sdk contains file platform.cfg. Config file loaded by clang could contain command `@/path/to/iPhoneOS9.3.sdk/platform.cfg`. That could be used to separate per-SDK defaults from other defaults.

I don't object to such use case, we just need to find balance between solution complexity, user convenience and maintainability.

Thanks,
--Serge

-- 
Hal Finkel
Lead, Compiler Technology and Programming Languages
Leadership Computing Facility
Argonne National Laboratory

_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
123