Quantcast

Configuration files

classic Classic list List threaded Threaded
45 messages Options
123
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Configuration files

Ivan Garramona via cfe-dev
----- Original Message -----

> From: "Matthias Braun via cfe-dev" <[hidden email]>
> To: "Serge Pavlov" <[hidden email]>
> Cc: "Richard Smith" <[hidden email]>, "Clang Dev" <[hidden email]>
> Sent: Tuesday, September 27, 2016 1:45:09 PM
> Subject: Re: [cfe-dev] Configuration files
>
> Sorry for being late to the discussion,
>
> I think automatic user defaults are a bad idea: Right now when you
> invoke clang without any additional options you know that you are in
> a well known state. Build systems and projects rely on this having
> unknown compiler options active because the user put them into his
> configuration file is a recipe for disaster IMO! The example below
> "-std=c++14 -fcxx-exception" already illustrates a situation in
> which settings are used that can break the compilation of many
> projects!
>
> - I'm fine with a commandline option triggering reading a config file
> (@file)
> - Using a different executable name/symlinks to trigger loading of
> config files may be fine as well (build systems should not pick up
> myarch-clang by accident)

I agree with both points.

> - Automatically picking up options from user or system configuration
> files is too dangerous IMO!

There are definitely good user-experience reasons to want to do this - It might be nice to have some global configuration file that causes Clang to default to using -march=native or use some particular include path (setting CPATH and friends often works, but can also be flaky). The problem is indeed build system interaction, dealing with incompatible options, etc. For many things our last-option-wins scheme will do the right thing, but it is not clear, for example, that -nostdinc would have the expected effect if the config file adds include paths.

 -Hal

>
> - Matthias
>
> > On Sep 20, 2016, at 12:13 AM, Serge Pavlov via cfe-dev
> > <[hidden email]> wrote:
> >
> > Thanks to all for feedback!
> >
> > I tried to summarize the feedback in the form of user-visible
> > description (possibly a part of user documentation) and random set
> > of development specific notes.
> >
> > ------ User documentation ------
> >
> > Using clang as a part of toolchain, especially in the case of
> > cross-compilation, require setting up large amount of parameters -
> > location of headers and libraries, set of enabled warnings,
> > triplet names etc. Changing from debug to release build, or from
> > host processor to accelerator or any other change of build
> > configuration requires substantial changes of these parameters. To
> > help maintaining option sets for various build variants, sets of
> > options may be combined into configurations and stored in
> > *configuration files*. Selection of particular configuration file
> > activates all options it represents.
> >
> > Configuration file may be selected in several ways:
> > - Using command line option --config,
> > - By setting up environmental variable CLANGCFG,
> > - As default configuration.
> > Only one way may be used. If option ''--config" is specified,
> > CLANGCFG is not checked and the default configuration is not be
> > applied even if the requested configuration is not found.
> > Similarly, if variable CLANGCFG exists, default configuration is
> > never applied.
> >
> > Command line option --config expects argument either full path to
> > configuration file, or a name of configuration, for instance:
> > --config /home/user/cfgs/testing.cfg
> > --config debug
> > --config debug.cfg
> > If full path is specified, options are read from that file. If
> > configuration is specified by name with optional suffix ".cfg",
> > corresponding configuration file is searched in the directories in
> > the following order:
> > - ~/.llvm
> > - /etc/llvm
> >
> > If the option --config is absent, and environment variable CLANGCFG
> > is set, content of CLANGCFG is used as full path to configuration
> > file. If CLANGCFG is empty, config file is not used, no diagnostic
> > produced.
> >
> > If neither --config nor CLANGCFG are specified, default config file
> > is searched in the following order:
> > - ~/.llvm/clang.cfg
> > - /etc/llvm/clang.cfg
> > - clang.cfg in the directory where clang executable resides.
> >
> > Configuration file is the sequence of compiler options specified in
> > one or several lines. Lines started with '#' possibly prepended
> > with space characters are comments, they are ignored. Lines
> > started with '#" in the first column immediately followed by ':'
> > are reserved for directives. The file may reference other files
> > using usual syntax for response files: @included_file. Example of
> > configuration file:
> >
> > # Frontend options
> > -std=c++14 -fcxx-exceptions
> > # Directories
> > @include-dirs.cfg
> > @library-dirs
> >
> > Name of the active configuration file and options it provided can
> > be obtained by call of clang with options '-v' or '-###'.
> >
> > ------ End of user documentation ------
> >
> > Notes:
> >
> > 1. What should be the name of default config file, 'default.cfg'
> > or 'clang.cfg'? If some tool other than clang will use the same
> > mechanism, name 'clang.cfg' looks more appropriate.
> > 2. Should compiler emit a warning (or even an error) if the
> > specified configuration file was not found? Obviously absence of
> > default config file should be silently ignored, as well as empty
> > CLANGCFG variable. But what if configuration specified by
> > --config is not found? This looks like a severe error, as user
> > expectation are broken.
> > 3. Default config file may be searched for in  the directory where
> > clang executable resides. Should configuration specified by
> > --config be searched in this directory as well? I would say no,
> > because a user who need tuning configurations may prepare them in
> > home directory or use those provided by installation in system
> > directories. Ability to place default config into binary
> > directory is convenient for compiler developer or CI tools, they
> > may uses several variants of compiler simultaneously.
> > 4. Format of proposed config file is in fact gnu response file.
> > YAML format mentioned by Richard looks nice but it requires
> > development of proper parser. I would propose at first implement
> > the simple format. In future other format can be supported, we
> > can distinguish formats by putting a directive like
> > '#:format=yaml' or even automatically.
> > 5. Some comments may be reserved for future use. Format directive
> > mentioned above is an example. The sequence '#:' proposed as
> > marker is absolutely arbitrary, any other prefix may be used. Now
> > compiler may warn if it sees such comment. Probably we do not
> > need to bother about the future extensions now.
> > 6. Using response files in the form @file can imitate include
> > directive. If a user do not want to copy-and-past pieces of
> > config files, he may structure them using this feature.
> >
> >
> > Thanks,
> > --Serge
> >
> > _______________________________________________
> > cfe-dev mailing list
> > [hidden email]
> > http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>
> _______________________________________________
> cfe-dev mailing list
> [hidden email]
> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>

--
Hal Finkel
Lead, Compiler Technology and Programming Languages
Leadership Computing Facility
Argonne National Laboratory
_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Configuration files

Ivan Garramona via cfe-dev
In reply to this post by Ivan Garramona via cfe-dev
I really don't see what the purpose of this feature is, if it's only going to work effectively identically to an "@args-file" option.

If that's all you want, it doesn't even need to be in the clang driver, a 5 line shell script that looks at $0 to choose a config file to read can work just as well.

But I don't think that's really what people want -- I believe that any workable config-file scheme *must* interact with the various target specification command-line arguments, and allow for setting options dependent upon the actually-selected target.

On Tue, Sep 20, 2016 at 3:13 AM, Serge Pavlov via cfe-dev <[hidden email]> wrote:
Thanks to all for feedback!

I tried to summarize the feedback in the form of user-visible description (possibly a part of user documentation) and random set of development specific notes.

------ User documentation ------

Using clang as a part of toolchain, especially in the case of cross-compilation, require setting up large amount of parameters - location of headers and libraries, set of enabled warnings, triplet names etc. Changing from debug to release build, or from host processor to accelerator or any other change of build configuration requires substantial changes of these parameters. To help maintaining option sets for various build variants, sets of options may be combined into configurations and stored in *configuration files*. Selection of particular configuration file activates all options it represents.

Configuration file may be selected in several ways:
- Using command line option --config,
- By setting up environmental variable CLANGCFG,
- As default configuration.
Only one way may be used. If option ''--config" is specified, CLANGCFG is not checked and the default configuration is not be applied even if the requested configuration is not found. Similarly, if variable CLANGCFG exists, default configuration is never applied.

Command line option --config expects argument either full path to configuration file, or a name of configuration, for instance:
--config /home/user/cfgs/testing.cfg
--config debug
--config debug.cfg
If full path is specified, options are read from that file. If configuration is specified by name with optional suffix ".cfg", corresponding configuration file is searched in the directories in the following order:
- ~/.llvm
- /etc/llvm

If the option --config is absent, and environment variable CLANGCFG is set, content of CLANGCFG is used as full path to configuration file. If CLANGCFG is empty, config file is not used, no diagnostic produced.

If neither --config nor CLANGCFG are specified, default config file is searched in the following order:
- ~/.llvm/clang.cfg
- /etc/llvm/clang.cfg
- clang.cfg in the directory where clang executable resides.

Configuration file is the sequence of compiler options specified in one or several lines. Lines started with '#' possibly prepended with space characters are comments, they are ignored. Lines started with '#" in the first column immediately followed by ':' are reserved for directives. The file may reference other files using usual syntax for response files: @included_file. Example of configuration file:

# Frontend options
-std=c++14 -fcxx-exceptions
# Directories
@include-dirs.cfg
@library-dirs
Name of the active configuration file and options it provided can be obtained by call of clang with options '-v' or '-###'.

------ End of user documentation ------

Notes:

1. What should be the name of default config file, 'default.cfg' or 'clang.cfg'? If some tool other than clang will use the same mechanism, name 'clang.cfg' looks more appropriate.
2. Should compiler emit a warning (or even an error) if the specified configuration file was not found? Obviously absence of default config file should be silently ignored, as well as empty CLANGCFG variable. But what if configuration specified by --config is not found? This looks like a severe error, as user expectation are broken.
3. Default config file may be searched for in  the directory where clang executable resides. Should configuration specified by --config be searched in this directory as well? I would say no, because a user who need tuning configurations may prepare them in home directory or use those provided by installation in system directories. Ability to place default config into binary directory is convenient for compiler developer or CI tools, they may uses several variants of compiler simultaneously.
4. Format of proposed config file is in fact gnu response file. YAML format mentioned by Richard looks nice but it requires development of proper parser. I would propose at first implement the simple format. In future other format can be supported, we can distinguish formats by putting a directive like '#:format=yaml' or even automatically.
5. Some comments may be reserved for future use. Format directive mentioned above is an example. The sequence '#:' proposed as marker is absolutely arbitrary, any other prefix may be used. Now compiler may warn if it sees such comment. Probably we do not need to bother about the future extensions now.
6. Using response files in the form @file can imitate include directive. If a user do not want to copy-and-past pieces of config files, he may structure them using this feature.


Thanks,
--Serge


_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev



_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Configuration files

Ivan Garramona via cfe-dev
This feature addresses at least two different use cases:

1. Config file as a way to override compiler defaults.

Compiler already has some default settings and it is hardly possible to find the settings that suit everybody. For instance, the warning -Wundefined-var-template is helpful for some users but annoys others. With config file a user could put new defaults there and thus customize the compiler. No changes to build script is required. Note, this scenario requires just default config files.

Setting config file via environment variable is a variant of this scenario, it allows modification of compiler options without touching build scripts.

Intel compiler implements both default config and environment variable as a way to modify compiler option and practice proved their convenience.

2. Config file as an "option macro".

Some users, especially those using cross-compilation, want to have possibility to define an "option" that when applied specifies a set of other options. An example: switching from one target processor to another which requires change of sysroot, libraries, triple etc. By organizing these options into config files, work with several build variants becomes more convenient.

This scenario requires to specify particular config file, either by option --config or by encoding it in compiler executable like clang-conf.

Clang based ELLCC Compiler uses these kind of config files and they are proved to be convenient (http://lists.llvm.org/pipermail/cfe-dev/2016-September/050783.html).


2016-09-28 2:18 GMT+07:00 James Y Knight <[hidden email]>:
I really don't see what the purpose of this feature is, if it's only going to work effectively identically to an "@args-file" option.

If that's all you want, it doesn't even need to be in the clang driver, a 5 line shell script that looks at $0 to choose a config file to read can work just as well.

 
For "option macro" scenario it is probably a matter of convenience. For "defaults override" it is not suitable as requires change in build scripts.
 
But I don't think that's really what people want -- I believe that any workable config-file scheme *must* interact with the various target specification command-line arguments, and allow for setting options dependent upon the actually-selected target.

 
You probably mean something like gcc spec file. This is definitely more extensive work. The simple scheme proposed here may be used to implement more elaborated solutions.

2016-09-28 1:45 GMT+07:00 Matthias Braun <[hidden email]>:
Sorry for being late to the discussion,

I think automatic user defaults are a bad idea: Right now when you invoke clang without any additional options you know that you are in a well known state. Build systems and projects rely on this having unknown compiler options active because the user put them into his configuration file is a recipe for disaster IMO! The example below "-std=c++14 -fcxx-exception" already illustrates a situation in which settings are used that can break the compilation of many projects!

 
In fact invocation of clang without additional argument does not provide a well known state. Actual state is represented by options passed from clang driver to clang compiler, but these options depend on used OS, CPU. In theory a system upgrade could cause compilation change. Hardly bare driver arguments can represent compilation options.

Default config file is just a way to influence the magic of option calculation made by driver. Config files can be used by clang based SDK suppliers to customize clang without need to modify compiler sources. If end user plays with default config files, he must know what he does.
 
- I'm fine with a commandline option triggering reading a config file (@file)
- Using a different executable name/symlinks to trigger loading of config files may be fine as well (build systems should not pick up myarch-clang by accident)
- Automatically picking up options from user or system configuration files is too dangerous IMO!

Automatic picking up options:
- is already made by driver,
- is necessary in "defaults override" scenario.

2016-09-28 1:55 GMT+07:00 Joerg Sonnenberger via cfe-dev <[hidden email]>:
On Tue, Sep 20, 2016 at 02:13:42PM +0700, Serge Pavlov wrote:
> If full path is specified, options are read from that file. If
> configuration is specified by name with optional suffix ".cfg",
> corresponding configuration file is searched in the directories in the
> following order:
> - ~/.llvm
> - /etc/llvm

I see no advantage in the additional complexity of new directory
searching logic when the user explicitly provides a config name.

 
A config name is in fact a config file, which need to be found somewhere. It makes sense if:
- config files are kept in a well known place.
- SDK supplier provides some set of config files. These may be considered as "system" configs and in multi-user environment they cannot be changed by regular users.
- A user has possibility to override system configurations.
As a result we end up with two well-known places: one for "system" configs, one for "user".
 
> If the option --config is absent, and environment variable CLANGCFG is set,
> content of CLANGCFG is used as full path to configuration file. If CLANGCFG
> is empty, config file is not used, no diagnostic produced.

I'm not a fan of environment variables for compilers either. You
normally can't specify empty environment variables: they are undefined.

Yes, it is too Linux specific. It looks like Windows does not support empty variables. This sentence must be removed.
 

> If neither --config nor CLANGCFG are specified, default config file is
> searched in the following order:
> - ~/.llvm/clang.cfg
> - /etc/llvm/clang.cfg
> - clang.cfg in the directory where clang executable resides.

I'm against a default configuration look up in the home directory. As
stated earlier, that would essentially force me to always provide a
--config argument in build systems to get consistent rules. /etc/llvm
should be the equivalent of --sysconfdir to configure in cmake. I'm
against the last as the question of where an executable is located is
quite problematic in many situations. If anything, the -B option should
be honored here.

Based on feedback I would limit the search for default config file by:
> - clang.cfg in the directory where clang executable resides.
Reasons:
- It still allows "defaults override" scenario,
- A regular user still is able to override compiler defaults if he uses local installation,
- It allows work with several compilers (compiler developer, CI system),
- It is the only place which Intel Compiler searches for config files and in practice it is enough.


> Configuration file is the sequence of compiler options specified in one or
> several lines. Lines started with '#' possibly prepended with space
> characters are comments, they are ignored. Lines started with '#" in the
> first column immediately followed by ':' are reserved for directives. The
> file may reference other files using usual syntax for response files:
> @included_file. Example of configuration file:

Please don't mix comments with semantics in one syntax form.

This place is not good, agree.

Thanks,
--Serge
 

_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Configuration files

Ivan Garramona via cfe-dev
On Wed, Sep 28, 2016 at 01:39:59PM +0700, Serge Pavlov via cfe-dev wrote:

> This feature addresses at least two different use cases:
>
> 1. Config file as a way to override compiler defaults.
>
> Compiler already has some default settings and it is hardly possible to
> find the settings that suit everybody. For instance, the warning
> -Wundefined-var-template
> is helpful for some users but annoys others. With config file a user could
> put new defaults there and thus customize the compiler. No changes to build
> script is required. Note, this scenario requires just default config files.

This is *exactly* why I am *strongly* against this. Yes, users do that.
Afterwards, they start to fill random bug reports against projects that
break, because the invocation includes -Werror. This is even worse than
setting a global CFLAGS or CXXFLAGS variable since the expansion of
those are at least generally visible. Hidden user-modifable sources of
flags are a support nightware.

> Setting config file via environment variable is a variant of this scenario,
> it allows modification of compiler options without touching build scripts.

See above. Same deal.

> Intel compiler implements both default config and environment variable as a
> way to modify compiler option and practice proved their convenience.

I dare to say that ICC tends to be used in a much more limited
environment. If you want to allow magic environmental variables in your
build script that are invisible for the casual observer, it is not
difficult to add. Well written build systems already support that
anyway.

> 2. Config file as an "option macro".
>
> Some users, especially those using cross-compilation, want to have
> possibility to define an "option" that when applied specifies a set of
> other options. An example: switching from one target processor to another
> which requires change of sysroot, libraries, triple etc. By organizing
> these options into config files, work with several build variants becomes
> more convenient.
>
> This scenario requires to specify particular config file, either by option
> --config or by encoding it in compiler executable like clang-conf.
>
> Clang based ELLCC Compiler uses these kind of config files and they are
> proved to be convenient (
> http://lists.llvm.org/pipermail/cfe-dev/2016-September/050783.html).

There was a reasonable proposal when this came up in the past. If you
have an invocation like ${target}-clang, look for a ${target}.cfg option
file and parse that. The lookup should be limited to something derived
from the static configuration and -B options.

Joerg

PS: It is a PITA to reasonable respond to mails that aggregate feedback
to a complete thread.
_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Configuration files

Ivan Garramona via cfe-dev
In reply to this post by Ivan Garramona via cfe-dev
On Wed, Sep 28, 2016 at 01:39:59PM +0700, Serge Pavlov via cfe-dev wrote:
> > But I don't think that's really what people want -- I believe that any
> > workable config-file scheme *must* interact with the various target
> > specification command-line arguments, and allow for setting options
> > dependent upon the actually-selected target.
> >
> >
> You probably mean something like gcc spec file. This is definitely more
> extensive work. The simple scheme proposed here may be used to implement
> more elaborated solutions.

No, nothing like spec files. As I said, you want to be able to specify
that the arm-linux target uses --sysroot=/somewhere/arm-linux and that
the arm-netbsd target uses --sysroot/somewhere/arm-netbsd. That and
maybe some -B option to help finding the linker likely already covers a
good 90% of what people need for setting up cross-compilation, if the
automatic triple-based search is not enough.

Joerg
_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Configuration files

Ivan Garramona via cfe-dev
In reply to this post by Ivan Garramona via cfe-dev
On Wed, Sep 28, 2016 at 01:39:59PM +0700, Serge Pavlov via cfe-dev wrote:

> > Sorry for being late to the discussion,
> >
> > I think automatic user defaults are a bad idea: Right now when you invoke
> > clang without any additional options you know that you are in a well known
> > state. Build systems and projects rely on this having unknown compiler
> > options active because the user put them into his configuration file is a
> > recipe for disaster IMO! The example below "-std=c++14 -fcxx-exception"
> > already illustrates a situation in which settings are used that can break
> > the compilation of many projects!
> >
> >
> In fact invocation of clang without additional argument does not provide a
> well known state. Actual state is represented by options passed from clang
> driver to clang compiler, but these options depend on used OS, CPU. In
> theory a system upgrade could cause compilation change. Hardly bare driver
> arguments can represent compilation options.

This is frankly FUD. While a compiler update can sometimes introduce new
default warnings (or add them to -Wall), those are governed by pretty
strict rules. The far majority of the options, especially the per-OS
per-CPU configuration changes very rarely and few of those changes are
user-visible in the sense that they break valid input.

> Default config file is just a way to influence the magic of option
> calculation made by driver. Config files can be used by clang based SDK
> suppliers to customize clang without need to modify compiler sources. If
> end user plays with default config files, he must know what he does.

But that's the point. Users often don't know or forget. There are long
rants to be found by Google of people forcing -OMG -ffast-math for
everything because they don't understand the implications. I strongly
hope that any vendor shipping a customized default config puts a strong
"Do not change this file without fully understanding what you are doing"
comment in it. The important part is that others are the ones that pay
the price.

Joerg
_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Configuration files

Ivan Garramona via cfe-dev
2016-09-28 16:22 GMT+07:00 Joerg Sonnenberger via cfe-dev <[hidden email]>:
On Wed, Sep 28, 2016 at 01:39:59PM +0700, Serge Pavlov via cfe-dev wrote:
> > Sorry for being late to the discussion,
> >
> > I think automatic user defaults are a bad idea: Right now when you invoke
> > clang without any additional options you know that you are in a well known
> > state. Build systems and projects rely on this having unknown compiler
> > options active because the user put them into his configuration file is a
> > recipe for disaster IMO! The example below "-std=c++14 -fcxx-exception"
> > already illustrates a situation in which settings are used that can break
> > the compilation of many projects!
> >
> >
> In fact invocation of clang without additional argument does not provide a
> well known state. Actual state is represented by options passed from clang
> driver to clang compiler, but these options depend on used OS, CPU. In
> theory a system upgrade could cause compilation change. Hardly bare driver
> arguments can represent compilation options.

This is frankly FUD. While a compiler update can sometimes introduce new
default warnings (or add them to -Wall), those are governed by pretty
strict rules. The far majority of the options, especially the per-OS
per-CPU configuration changes very rarely and few of those changes are
user-visible in the sense that they break valid input.

 
You are right, the mapping from driver options to compiler ones is relatively stable now.
My statement was inspired by ICC option -xHost, which selects options optimal
for the host processor. In any case the set of options passed from driver to compiler is
more robust characteristic and it differs from user supplied options substantially.
 
> Default config file is just a way to influence the magic of option
> calculation made by driver. Config files can be used by clang based SDK
> suppliers to customize clang without need to modify compiler sources. If
> end user plays with default config files, he must know what he does.

But that's the point. Users often don't know or forget. There are long
rants to be found by Google of people forcing -OMG -ffast-math for
everything because they don't understand the implications. I strongly
hope that any vendor shipping a customized default config puts a strong
"Do not change this file without fully understanding what you are doing"
comment in it. The important part is that others are the ones that pay
the price.


 Absolutely agree!

Thanks,
--Serge


_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Configuration files

Ivan Garramona via cfe-dev
In reply to this post by Ivan Garramona via cfe-dev
2016-09-28 16:13 GMT+07:00 Joerg Sonnenberger via cfe-dev <[hidden email]>:
On Wed, Sep 28, 2016 at 01:39:59PM +0700, Serge Pavlov via cfe-dev wrote:
> This feature addresses at least two different use cases:
>
> 1. Config file as a way to override compiler defaults.
>
> Compiler already has some default settings and it is hardly possible to
> find the settings that suit everybody. For instance, the warning
> -Wundefined-var-template
> is helpful for some users but annoys others. With config file a user could
> put new defaults there and thus customize the compiler. No changes to build
> script is required. Note, this scenario requires just default config files.

This is *exactly* why I am *strongly* against this. Yes, users do that.
Afterwards, they start to fill random bug reports against projects that
break, because the invocation includes -Werror. This is even worse than
setting a global CFLAGS or CXXFLAGS variable since the expansion of
those are at least generally visible. Hidden user-modifable sources of
flags are a support nightware.

 
Users are of different categories. What you say is true for end users, who just use
compiler as a tool. But there are other users, who develop clang based SDKs for
these end users. They would use this feature with more care because support problems
would be their problems. Otherwise such user may want to turn off unwanted warnings by
modifying compiler sources, this is much worse case.


I dare to say that ICC tends to be used in a much more limited
environment. If you want to allow magic environmental variables in your
build script that are invisible for the casual observer, it is not
difficult to add. Well written build systems already support that
anyway.

Environment variable is just another way to specify configuration file, it does not
need modification of build scripts. Convenient for experiments. If we support config
file in any form, there is no additional danger to allow setting up configuration via
environmental variables. It cannot add any option automatically.

Thanks,
--Serge


_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Configuration files

Ivan Garramona via cfe-dev
On Wed, Sep 28, 2016 at 07:57:19PM +0700, Serge Pavlov via cfe-dev wrote:

> 2016-09-28 16:13 GMT+07:00 Joerg Sonnenberger via cfe-dev <
> [hidden email]>:
>
> > On Wed, Sep 28, 2016 at 01:39:59PM +0700, Serge Pavlov via cfe-dev wrote:
> > > This feature addresses at least two different use cases:
> > >
> > > 1. Config file as a way to override compiler defaults.
> > >
> > > Compiler already has some default settings and it is hardly possible to
> > > find the settings that suit everybody. For instance, the warning
> > > -Wundefined-var-template
> > > is helpful for some users but annoys others. With config file a user
> > could
> > > put new defaults there and thus customize the compiler. No changes to
> > build
> > > script is required. Note, this scenario requires just default config
> > files.
> >
> > This is *exactly* why I am *strongly* against this. Yes, users do that.
> > Afterwards, they start to fill random bug reports against projects that
> > break, because the invocation includes -Werror. This is even worse than
> > setting a global CFLAGS or CXXFLAGS variable since the expansion of
> > those are at least generally visible. Hidden user-modifable sources of
> > flags are a support nightware.
> >
> >
> Users are of different categories. What you say is true for end users, who
> just use
> compiler as a tool. But there are other users, who develop clang based SDKs
> for
> these end users. They would use this feature with more care because support
> problems
> would be their problems. Otherwise such user may want to turn off unwanted
> warnings by
> modifying compiler sources, this is much worse case.

That's why I am not completely opposed to providing target-specific
configuration files.

> > I dare to say that ICC tends to be used in a much more limited
> > environment. If you want to allow magic environmental variables in your
> > build script that are invisible for the casual observer, it is not
> > difficult to add. Well written build systems already support that
> > anyway.
> >
>
> Environment variable is just another way to specify configuration file, it
> does not
> need modification of build scripts. Convenient for experiments. If we
> support config
> file in any form, there is no additional danger to allow setting up
> configuration via
> environmental variables. It cannot add any option automatically.

If the configuration file is (normally) provided via the command line
that gives others at least a chance of finding out what is going wrong.
That's quite different from environment variables, that are often
invisible until you explicitly look for them. History of UNIX is full of
problems with environment variables.

Joerg
_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Configuration files

Ivan Garramona via cfe-dev
In reply to this post by Ivan Garramona via cfe-dev
My main concern with configuration files is that they should interact with regular command line flags in a predictable and maintainable way. Your approach of extending response files seems like it's going in the right direction.

I'm not excited about adding something like ELLC's YAML format to Clang (http://ellcc.org/viewvc/svn/ellcc/trunk/libecc/config/ppc64el-linux?view=markup) because then we will have to document and maintain the semantics of things like multiple configuration files, in what order they override each other, and how they interact with flags. Our command line flag logic is already too complicated. We have one public interface, and it's flags, and I'm hesitant to add a new one.

On Mon, Sep 19, 2016 at 1:06 AM, Serge Pavlov via cfe-dev <[hidden email]> wrote:
Hi all,

I would like to propose implementing configuration files in clang. The idea is to read some set of options prior to the options specified in a command line in every tool invocation. These options are taken from a file which is searched either in predefined place or its location is specified by environmental variable.

Few words about particular problem this facility can solve. Clang issues many warnings, some refer to potential errors, some merely attract attention to code that can cause problems in some circumstances but otherwise is pretty innocent. For example, warning -Wundefined-var-template is issued for template usages, which nor cannot be instantiated implicitly neither are mentioned in explicit instantiation declarations. Such usage is not an error or bad style, but the warning may be helpful for a user in a complex case, such as described in PR24425. For other users this warning may be annoying and they would prefer to turn it off by default (see discussion in http://lists.llvm.org/pipermail/cfe-commits/Week-of-Mon-20160808/167354.html). Different categories of users like to have different set of warnings enabled by default.

If configuration files were supported by clang, a user can put -Wno-undefined-var-template to such file and this would have the same effect as if the unwanted warning were turned off by default. No changes to build scripts would be required.

Configuration files are supported by the Intel Compiler (https://software.intel.com/en-us/node/522780) and similar behavior is proposed to be implemented by clang.

By default the configuration file is searched in the same directory where tool executable resides. In the case of clang the file would be 'clang.cfg', for other tool named 'foo'- 'foo.cfg'. User can specify any location for this file if he sets up environmental variable 'CLANGCFG' (or 'FOOCFG'). The variable should contain path to configuration file. If the configuration file does not exist, it is silently ignored. Content of configuration file is treated as a part of command line of each tool invocation preceding any arguments specified in command line.

Does implementation of this facility makes sense?

Thanks,
--Serge

_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev



_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Configuration files

Ivan Garramona via cfe-dev
The proposal has been evolved due to feedback. Below is a summary of updated version and feedback from various sources.

**Problems**

The two main problems this feature addresses are:

1. Ability to modify default compiler settings.

As an example, the warning `-Wundefined-var-template` can be helpful for people doing module-enabled builds   (https://llvm.org/bugs/show_bug.cgi?id=24425), but it makes headache for others, who even come with ideas to turn this warning off by default (http://lists.llvm.org/pipermail/cfe-commits/Week-of-Mon-20160808/167354.html).

To cope with this and similar problem clang need a mechanism that effectively change default settings hardcoded in compiler sources. 
2. Support of cross builds.

Doing cross builds requires to specify lots of options that set target, backend options, include and library directories etc. These options could be placed into config files and then be specified with single option or even without such. This feature could make cross compilation with clang more convenient.
**Solution**

Configuration file groups related options together and allow to specify them in more simple and less error prone way than just listing them somewhere in build scripts. According to the two problems to solve, there are two use patterns of configuration files:

- Default config file is applied without need to specify anything in compiler invocation. It can be read from predefined location or be specified by environmental variable. It allows to override compiler defaults.
- Named config file is specified explicitly by a user, either by command line option `--config`, or by using special executable file names, such as `armv7l-clang`. Such configuration may be thought as a "macro" that designates an option set and is expanded when compiler is called.
Only one config file is used, if it is specified explicitly, default config file is not searched. Config file may contain comments (# style), references to other config files using syntax `@file`.

Config file are searched in the directory where clang executable resides. User may specify any config file using option `--config` followed by full path of the file. Alternatively the path to config file may be specified in environmental variable CLANGCFG.
**Existing experience**

Configuration file is not a new concept, it has been used in various form in many compilers including clang.

1. Intel compiler supports default configuration files (https://software.intel.com/en-us/node/522780), the file is searched for in the directory where executable resides or can be specified by setting environmental variable. The feature is pretty convenient in practice.
2. Clang allows to specify target by using special executable names, like `armv7l-clang`. Only target name can be specified by this way, which is not sufficient to tune compiler for a new target. Obvious question arise - why not extend this mechanism for other target specific options too.
3. Clang allows modifying options using environmental variable `CCC_OVERRIDE_OPTIONS`. This is undocumented way and is looks more like a hack for internal use. Its existence however indicates that such functionality is useful.
4. Cross compilation environment ELLCC (http://ellcc.org), based on clang, supports advanced config files in YAML format. It has proven to be very handy to support a wide variety of targets.
5. GCC has support of spec files (https://gcc.gnu.org/onlinedocs/gcc/Spec-Files.html), which can be taken from default location or be specified by command line option `--spec`. Spec files are more powerful tool than just config files as they not only set options but also program driver logic. Using custom spec files allows GCC to be tuned to particular target without need to specify lots of options in command line.
6. Visual Studio supports custom project templates (https://msdn.microsoft.com/en-us/library/xkh1wxd8.aspx), which may specify any compiler flags as default setting. Although it can be considered as feature of build system rather than compiler, it allows to override default compiler settings.
7. Keil C compilers allow to specify compiler options in environmental variables, ARMCOMPILER6_CLANGOPT (http://www.keil.com/support/man/docs/armclang_intro/armclang_intro_chr1374139991387.htm) or ARMCC5_CCOPT (http://infocenter.arm.com/help/topic/com.arm.doc.dui0592e/pge1399635613963.html) depending on compiler version.

**Concerns**

Somewhat random set of concerns extracted from discussions with answers to them.

1. With default config files the options specified for clang in command line do not represent definite state anymore, because some options may come from the config file. A user that prepares bug report can forget to specify content of the used default config file, it can make difficult to reproduce failure conditions.
Answer: Actual set of compiler options is provided in error dump files, in output of `clang -v` and `clang -###`, these invocations also report used config file. If a project wants to disable some warnings by default, add it to the Makefile.
2. It is not a business of compiler to adjust options. What flags to pass to the compiler should be configured in a project's build system.
Answer: A project may include dozens of components each come with own build systems, which are tuned differently. Retargeting such build system can be time consuming and error prone. On the other hand, setting compiler and target specific options is not a business of a component, ideally it may be unaware of using in cross compilation environment. Also, not all component build systems are flexible enough. As a result some users do not want modify their build files, they would prefer customized compiler.
Config files brings more power and convenience to the compiler.
3. Clang already has `@file` for bundling up compiler flags.
Answer: Config file make this feature a bit more convenient by providing a way to choose proper option set and making files containing options more user friendly by allowing comments and nested `@file`. Also `@file` cannot solve the problem of overriding compiler defaults.
4. The idea of having the compiler search for config files in random places on the filesystem seems scary.
Answer: Default config files are searched only in the directory where compiler executable file resides. If compiler is installed from package, this file is placed in system directories and should not be changed by a simple user. It is assumed that such file is a part of SDK or some package prepared by experienced person, it is not changed frequently. The same applies to target-specific config files. A user may specify full path to config file, in this case the file may be anywhere but this explicit action.
Most of elaborated environments allows search for libraries, packages, classes etc in some predefined places. Running simple command `gcc file.c` make compiler search set of various places, and the set may change depending on version and OS variant.
5. There is already a lot of magic in the driver to figure out how to invoke the tools, this feature makes driver more complex.
Answer: Orchestrating tools made by clang driver is already a complex task and hardly it can be made simple. But it can be made more structured, config file may be a step in this direction making existing target selection in clang more powerful and useful.
6. This is too primitive solution:
- A good solution must be able to deduce target from arguments (-m32).
- We should also probably automatically silence unused options from config files.  For example, if my cross-compile toolchain passes some extra linker arguments with -Wl,-foo, then I don’t want every single line in my build to complain that -Wl is unused in conjunction with -c.
- It should provide default mapping from targets to sysroots and associated paths; we currently hardcode a lot of this into the compiler itself.
- Any workable config-file scheme *must* interact with the various target specification command-line arguments, and allow for setting options dependent upon the actually-selected target.
- Maybe it'd be better to create a regular, .ini/.conf-style configuration file.
Answer: This is a simple solution, that solves a good deal of problems. It does not require big changes in documentation and substantial maintenance efforts. It can be extended in future by allowing more complex constructs in config files.

Thanks,
--Serge

2016-10-05 0:29 GMT+07:00 Reid Kleckner <[hidden email]>:
My main concern with configuration files is that they should interact with regular command line flags in a predictable and maintainable way. Your approach of extending response files seems like it's going in the right direction.

I'm not excited about adding something like ELLC's YAML format to Clang (http://ellcc.org/viewvc/svn/ellcc/trunk/libecc/config/ppc64el-linux?view=markup) because then we will have to document and maintain the semantics of things like multiple configuration files, in what order they override each other, and how they interact with flags. Our command line flag logic is already too complicated. We have one public interface, and it's flags, and I'm hesitant to add a new one.

On Mon, Sep 19, 2016 at 1:06 AM, Serge Pavlov via cfe-dev <[hidden email]> wrote:
Hi all,

I would like to propose implementing configuration files in clang. The idea is to read some set of options prior to the options specified in a command line in every tool invocation. These options are taken from a file which is searched either in predefined place or its location is specified by environmental variable.

Few words about particular problem this facility can solve. Clang issues many warnings, some refer to potential errors, some merely attract attention to code that can cause problems in some circumstances but otherwise is pretty innocent. For example, warning -Wundefined-var-template is issued for template usages, which nor cannot be instantiated implicitly neither are mentioned in explicit instantiation declarations. Such usage is not an error or bad style, but the warning may be helpful for a user in a complex case, such as described in PR24425. For other users this warning may be annoying and they would prefer to turn it off by default (see discussion in http://lists.llvm.org/pipermail/cfe-commits/Week-of-Mon-20160808/167354.html). Different categories of users like to have different set of warnings enabled by default.

If configuration files were supported by clang, a user can put -Wno-undefined-var-template to such file and this would have the same effect as if the unwanted warning were turned off by default. No changes to build scripts would be required.

Configuration files are supported by the Intel Compiler (https://software.intel.com/en-us/node/522780) and similar behavior is proposed to be implemented by clang.

By default the configuration file is searched in the same directory where tool executable resides. In the case of clang the file would be 'clang.cfg', for other tool named 'foo'- 'foo.cfg'. User can specify any location for this file if he sets up environmental variable 'CLANGCFG' (or 'FOOCFG'). The variable should contain path to configuration file. If the configuration file does not exist, it is silently ignored. Content of configuration file is treated as a part of command line of each tool invocation preceding any arguments specified in command line.

Does implementation of this facility makes sense?

Thanks,
--Serge

_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev




_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Configuration files

Ivan Garramona via cfe-dev
On 9 Nov 2016, at 07:59, Serge Pavlov via cfe-dev <[hidden email]> wrote:
>
> 5. There is already a lot of magic in the driver to figure out how to invoke the tools, this feature makes driver more complex.
>
> Answer: Orchestrating tools made by clang driver is already a complex task and hardly it can be made simple. But it can be made more structured, config file may be a step in this direction making existing target selection in clang more powerful and useful.

It would be nice if a proposed solution resulted in a net decrease in total complexity for the driver, with more of the code moved into default config files.

David


_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Configuration files

Ivan Garramona via cfe-dev
Hi all,

Implementation is presented in https://reviews.llvm.org/D24933. It is limited to the case in which user explicitly specifies config file, either by option `--config` or by using mangled compiler name, like `armv7l-clang`. Its main goal is to simplify use of clang when command line contains many options that are the same for many invocations. The other use case is default config file, which is applied automatically. This case is not presented in this implementation to simplify review process.

To move forward additional feedback is necessary, mainly to decide whether this feature is worth implementing. Config files may be convenient in the cases in cross-compilation. So far, there is a positive opinion about using config files in clang based cross compilation environment: http://lists.llvm.org/pipermail/cfe-dev/2016-September/050783.html.

Some express concern that this is not a useful feature, the main points are listed below.

1. Config files vs `@file`
There is an opinion that the functionality provided by config files is already available through the directive `@file`. Both config file and `@file` specify a set of command line options, but config file has some advantages:
- It searches well-known places for files, while `@file` must search only the current directory,
- It may include other config file, they are resolved relative to including file, making it possible to move set of related files across file system. Nested `@file` searches only current directory. Coupled with above note it means that using `@file` directive requires all files be specified by absolute paths, if we want to reference the same files during build process.
- Config files may contain comments,
- Options from config file do not produce warning 'unused option' even if they are indeed unused.

Ability to contain comments is not only a convenience for maintenance. It can be useful for extending functionality. We can assume, for instance, that if the first line is a comment of some special form, such as `#!yaml` or `# format = xml` or some other, then it specifies format of the config file. This would allow  extending functionality of config files in future without breaking compatibility.

2. More elaborated solutions

Many are of opinion that config file in this form is too primitive solution, for instance:
- A good solution must be able to deduce target from arguments (-m32),
- Any workable config-file scheme *must* interact with the various target specification command-line arguments, and allow for setting options dependent upon the actually-selected target,
- Config file must reduce complexity of clang driver.

The presented implementation of config files is only a way to specify set of options, it cannot solve these problems. However it can be extended in future so that, for instance, the following construct could be allowed (particular syntax does not matter):

    …
    #if (argv.contains("-m32"))
    -target i686-unknown-linux-gnu
    #else
    -targer x86_64-unknown-linux-gnu
    #endif
    …

This syntax would require to support `if/else/endif` control construct, and builtin function `argv.contains`. Ability to include an option conditionally is likely to be sufficient to solve large part of the problems mentioned above and can indeed reduce complexity of the driver.

Speaking about ways to make clang driver more flexible and less complex, we need to mention an approach implemented in gcc as `spec files` (https://gcc.gnu.org/onlinedocs/gcc/Spec-Files.html). A config file, even in the extended form, is only a way to specify a set of options for the driver. A spec file is a set of instructions for the driver how to handle a particular input file. These two solution have different ecological niches and can coexist. A regular user may prepare a set of config files for their needs, these files would capture user-specific information, such as where various components are placed in the user environment. A spec file is more compiler specific, if it needs user information, it would get it from options or environmental variables. As a part of spec file interpretation, the driver must build command lines for compiler, linker and other tools, functionality of config file may be useful in this case too.

In comparison with the more elaborated solutions mentioned above, the proposed implementation of config files is much simpler and if it is useful, it can be a first step toward more powerful and convenient driver.


Thanks,
--Serge


_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Configuration files

Ivan Garramona via cfe-dev


On 02/06/2017 10:05 AM, Serge Pavlov via cfe-dev wrote:
Hi all,

Implementation is presented in https://reviews.llvm.org/D24933. It is limited to the case in which user explicitly specifies config file, either by option `--config` or by using mangled compiler name, like `armv7l-clang`. Its main goal is to simplify use of clang when command line contains many options that are the same for many invocations. The other use case is default config file, which is applied automatically. This case is not presented in this implementation to simplify review process.

To move forward additional feedback is necessary, mainly to decide whether this feature is worth implementing. Config files may be convenient in the cases in cross-compilation. So far, there is a positive opinion about using config files in clang based cross compilation environment: http://lists.llvm.org/pipermail/cfe-dev/2016-September/050783.html.

Some express concern that this is not a useful feature, the main points are listed below.

1. Config files vs `@file`
There is an opinion that the functionality provided by config files is already available through the directive `@file`. Both config file and `@file` specify a set of command line options, but config file has some advantages:
- It searches well-known places for files, while `@file` must search only the current directory,
- It may include other config file, they are resolved relative to including file, making it possible to move set of related files across file system. Nested `@file` searches only current directory. Coupled with above note it means that using `@file` directive requires all files be specified by absolute paths, if we want to reference the same files during build process.
- Config files may contain comments,
- Options from config file do not produce warning 'unused option' even if they are indeed unused.

I chatted with Richard about this in person a couple of weeks ago, and he seems happy with the general direction here. One comment he had was that it seems like `@file` processing is a restricted case of config-file processing. As a result, it might make sense to replace the current `@file`-processing logic with the config-file processing logic (just with a bunch of features turned off).


Ability to contain comments is not only a convenience for maintenance. It can be useful for extending functionality. We can assume, for instance, that if the first line is a comment of some special form, such as `#!yaml` or `# format = xml` or some other, then it specifies format of the config file. This would allow  extending functionality of config files in future without breaking compatibility.

2. More elaborated solutions

Many are of opinion that config file in this form is too primitive solution, for instance:
- A good solution must be able to deduce target from arguments (-m32),
- Any workable config-file scheme *must* interact with the various target specification command-line arguments, and allow for setting options dependent upon the actually-selected target,
- Config file must reduce complexity of clang driver.

The presented implementation of config files is only a way to specify set of options, it cannot solve these problems. However it can be extended in future so that, for instance, the following construct could be allowed (particular syntax does not matter):

    …
    #if (argv.contains("-m32"))
    -target i686-unknown-linux-gnu
    #else
    -targer x86_64-unknown-linux-gnu
    #endif
    …

This syntax would require to support `if/else/endif` control construct, and builtin function `argv.contains`. Ability to include an option conditionally is likely to be sufficient to solve large part of the problems mentioned above and can indeed reduce complexity of the driver.

This is appealing, but I think that we really need to be careful here. We don't want to create a system where people are encouraged to replicate complex driver logic in the config files in order to independently figure out which features might be enabled. Of course, we already do this, but the logic ends up in wrapper scripts (and is just as likely to be broken, if not more so). However, if we're going to create a solution here, we should do so in a way that does not encourage the confg-file writer to duplicate driver logic.

In any case, I definitely consider this follow-up work.

Thanks again,
Hal


Speaking about ways to make clang driver more flexible and less complex, we need to mention an approach implemented in gcc as `spec files` (https://gcc.gnu.org/onlinedocs/gcc/Spec-Files.html). A config file, even in the extended form, is only a way to specify a set of options for the driver. A spec file is a set of instructions for the driver how to handle a particular input file. These two solution have different ecological niches and can coexist. A regular user may prepare a set of config files for their needs, these files would capture user-specific information, such as where various components are placed in the user environment. A spec file is more compiler specific, if it needs user information, it would get it from options or environmental variables. As a part of spec file interpretation, the driver must build command lines for compiler, linker and other tools, functionality of config file may be useful in this case too.

In comparison with the more elaborated solutions mentioned above, the proposed implementation of config files is much simpler and if it is useful, it can be a first step toward more powerful and convenient driver.


Thanks,
--Serge



_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev

-- 
Hal Finkel
Lead, Compiler Technology and Programming Languages
Leadership Computing Facility
Argonne National Laboratory

_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Configuration files

Ivan Garramona via cfe-dev
Hi Hal,

Thank you very much for your interest and help!

2017-03-15 2:08 GMT+07:00 Hal Finkel <[hidden email]>:
 
I chatted with Richard about this in person a couple of weeks ago, and he seems happy with the general direction here. One comment he had was that it seems like `@file` processing is a restricted case of config-file processing. As a result, it might make sense to replace the current `@file`-processing logic with the config-file processing logic (just with a bunch of features turned off).


The main issue of using `@file` for loading configuration is compatibility. The construct `@file` has been used for ages and there must be lots of build scripts using it, the change of semantics must not break them. It looks like the features that may be added to the file format in `@file` construct without disturbing existing functionality are:
- comments,
- splitting long lines with trailing '\',
- suppressing warnings on unused options.
The remaining feature is resolving names in nested `@file` constructs, it must keep current behavior (resolve names relative to current directory). This makes nesting constructs `@file` inconvenient it not useless, but it does not look like a big loss.

Configuration encoded in compiler file name, as in `armv7l-clang` is not related to `@file`, so this functionality implemented in https://reviews.llvm.org/D24933 remains unchanged. Explicit specification of config file in command line should be made by `@file` option, there are several issues as compared to the proposed `--config` option:
- config file must be specified by its full path otherwise it won't be found if the current directory is changed by build script. This limitation should not be a big problem if options are specified in build scripts, but for manual invocations in command line it is inconvenient.
- command line may contain several `@file` constructs, there may be options before `@file`. If we want to distinguish config file from others (for instance, we want to suppress unused option warnings only in such files), we need to assume that only the first option may load it.

So if we are ready to sacrifice some convenience, config files can be implemented on top of `@file` construct. If there are no objections I'll update the patch.
 

2. More elaborated solutions

Many are of opinion that config file in this form is too primitive solution, for instance:
- A good solution must be able to deduce target from arguments (-m32),
- Any workable config-file scheme *must* interact with the various target specification command-line arguments, and allow for setting options dependent upon the actually-selected target,
- Config file must reduce complexity of clang driver.

The presented implementation of config files is only a way to specify set of options, it cannot solve these problems. However it can be extended in future so that, for instance, the following construct could be allowed (particular syntax does not matter):

    …
    #if (argv.contains("-m32"))
    -target i686-unknown-linux-gnu
    #else
    -targer x86_64-unknown-linux-gnu
    #endif
    …

This syntax would require to support `if/else/endif` control construct, and builtin function `argv.contains`. Ability to include an option conditionally is likely to be sufficient to solve large part of the problems mentioned above and can indeed reduce complexity of the driver.

This is appealing, but I think that we really need to be careful here. We don't want to create a system where people are encouraged to replicate complex driver logic in the config files in order to independently figure out which features might be enabled. Of course, we already do this, but the logic ends up in wrapper scripts (and is just as likely to be broken, if not more so). However, if we're going to create a solution here, we should do so in a way that does not encourage the confg-file writer to duplicate driver logic.

In any case, I definitely consider this follow-up work.

 This was a bad example. To obtain more powerful solution clang need something like gcc spec files. This is completely different topic.

Thanks,
--Serge


_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Configuration files

Ivan Garramona via cfe-dev


On 03/15/2017 10:59 AM, Serge Pavlov wrote:
Hi Hal,

Thank you very much for your interest and help!

2017-03-15 2:08 GMT+07:00 Hal Finkel <[hidden email]>:
 
I chatted with Richard about this in person a couple of weeks ago, and he seems happy with the general direction here. One comment he had was that it seems like `@file` processing is a restricted case of config-file processing. As a result, it might make sense to replace the current `@file`-processing logic with the config-file processing logic (just with a bunch of features turned off).


The main issue of using `@file` for loading configuration is compatibility. The construct `@file` has been used for ages and there must be lots of build scripts using it, the change of semantics must not break them.

Right.

It looks like the features that may be added to the file format in `@file` construct without disturbing existing functionality are:
- comments,
- splitting long lines with trailing '\',
- suppressing warnings on unused options.
The remaining feature is resolving names in nested `@file` constructs, it must keep current behavior (resolve names relative to current directory). This makes nesting constructs `@file` inconvenient it not useless, but it does not look like a big loss.

Can't you just make it configurable? If parsing an `@file`, do the old thing, otherwise, do the new thing?


Configuration encoded in compiler file name, as in `armv7l-clang` is not related to `@file`, so this functionality implemented in https://reviews.llvm.org/D24933 remains unchanged. Explicit specification of config file in command line should be made by `@file` option, there are several issues as compared to the proposed `--config` option:
- config file must be specified by its full path otherwise it won't be found if the current directory is changed by build script. This limitation should not be a big problem if options are specified in build scripts, but for manual invocations in command line it is inconvenient.
- command line may contain several `@file` constructs, there may be options before `@file`. If we want to distinguish config file from others (for instance, we want to suppress unused option warnings only in such files), we need to assume that only the first option may load it.

So if we are ready to sacrifice some convenience, config files can be implemented on top of `@file` construct. If there are no objections I'll update the patch.

Maybe I'm missing something, but I think you're taking this the wrong way. I don't recommend giving up features in this sense. I think the question is: can we replace the current `@file` processing with the new config-file logic (just with some of the new features turned off)?

If that can't be done reasonably, then I think that's okay too. We just should understand (and document) why.

 -Hal

 

2. More elaborated solutions

Many are of opinion that config file in this form is too primitive solution, for instance:
- A good solution must be able to deduce target from arguments (-m32),
- Any workable config-file scheme *must* interact with the various target specification command-line arguments, and allow for setting options dependent upon the actually-selected target,
- Config file must reduce complexity of clang driver.

The presented implementation of config files is only a way to specify set of options, it cannot solve these problems. However it can be extended in future so that, for instance, the following construct could be allowed (particular syntax does not matter):

    …
    #if (argv.contains("-m32"))
    -target i686-unknown-linux-gnu
    #else
    -targer x86_64-unknown-linux-gnu
    #endif
    …

This syntax would require to support `if/else/endif` control construct, and builtin function `argv.contains`. Ability to include an option conditionally is likely to be sufficient to solve large part of the problems mentioned above and can indeed reduce complexity of the driver.

This is appealing, but I think that we really need to be careful here. We don't want to create a system where people are encouraged to replicate complex driver logic in the config files in order to independently figure out which features might be enabled. Of course, we already do this, but the logic ends up in wrapper scripts (and is just as likely to be broken, if not more so). However, if we're going to create a solution here, we should do so in a way that does not encourage the confg-file writer to duplicate driver logic.

In any case, I definitely consider this follow-up work.

 This was a bad example. To obtain more powerful solution clang need something like gcc spec files. This is completely different topic.

Thanks,
--Serge


-- 
Hal Finkel
Lead, Compiler Technology and Programming Languages
Leadership Computing Facility
Argonne National Laboratory

_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Configuration files

Ivan Garramona via cfe-dev


On 03/15/2017 11:21 AM, Hal Finkel via cfe-dev wrote:


On 03/15/2017 10:59 AM, Serge Pavlov wrote:
Hi Hal,

Thank you very much for your interest and help!

2017-03-15 2:08 GMT+07:00 Hal Finkel <[hidden email]>:
 
I chatted with Richard about this in person a couple of weeks ago, and he seems happy with the general direction here. One comment he had was that it seems like `@file` processing is a restricted case of config-file processing. As a result, it might make sense to replace the current `@file`-processing logic with the config-file processing logic (just with a bunch of features turned off).


The main issue of using `@file` for loading configuration is compatibility. The construct `@file` has been used for ages and there must be lots of build scripts using it, the change of semantics must not break them.

Right.

It looks like the features that may be added to the file format in `@file` construct without disturbing existing functionality are:
- comments,
- splitting long lines with trailing '\',
- suppressing warnings on unused options.
The remaining feature is resolving names in nested `@file` constructs, it must keep current behavior (resolve names relative to current directory). This makes nesting constructs `@file` inconvenient it not useless, but it does not look like a big loss.

Can't you just make it configurable? If parsing an `@file`, do the old thing, otherwise, do the new thing?


Configuration encoded in compiler file name, as in `armv7l-clang` is not related to `@file`, so this functionality implemented in https://reviews.llvm.org/D24933 remains unchanged. Explicit specification of config file in command line should be made by `@file` option, there are several issues as compared to the proposed `--config` option:
- config file must be specified by its full path otherwise it won't be found if the current directory is changed by build script. This limitation should not be a big problem if options are specified in build scripts, but for manual invocations in command line it is inconvenient.
- command line may contain several `@file` constructs, there may be options before `@file`. If we want to distinguish config file from others (for instance, we want to suppress unused option warnings only in such files), we need to assume that only the first option may load it.

So if we are ready to sacrifice some convenience, config files can be implemented on top of `@file` construct. If there are no objections I'll update the patch.

Maybe I'm missing something, but I think you're taking this the wrong way. I don't recommend giving up features in this sense. I think the question is: can we replace the current `@file` processing with the new config-file logic (just with some of the new features turned off)?

If that can't be done reasonably, then I think that's okay too. We just should understand (and document) why.

To elaborate slightly, we currently support `@file` in Clang by calling llvm::cl::ExpandResponseFiles in Clang's driver. Since this is general LLVM functionality, and the config-file paring is more Clang-specific, maybe this is not worth changing. However, what if we processed `@file` like config files, but in a mode which was backward compatible with existing `@file` semantics?

 -Hal


 -Hal

 

2. More elaborated solutions

Many are of opinion that config file in this form is too primitive solution, for instance:
- A good solution must be able to deduce target from arguments (-m32),
- Any workable config-file scheme *must* interact with the various target specification command-line arguments, and allow for setting options dependent upon the actually-selected target,
- Config file must reduce complexity of clang driver.

The presented implementation of config files is only a way to specify set of options, it cannot solve these problems. However it can be extended in future so that, for instance, the following construct could be allowed (particular syntax does not matter):

    …
    #if (argv.contains("-m32"))
    -target i686-unknown-linux-gnu
    #else
    -targer x86_64-unknown-linux-gnu
    #endif
    …

This syntax would require to support `if/else/endif` control construct, and builtin function `argv.contains`. Ability to include an option conditionally is likely to be sufficient to solve large part of the problems mentioned above and can indeed reduce complexity of the driver.

This is appealing, but I think that we really need to be careful here. We don't want to create a system where people are encouraged to replicate complex driver logic in the config files in order to independently figure out which features might be enabled. Of course, we already do this, but the logic ends up in wrapper scripts (and is just as likely to be broken, if not more so). However, if we're going to create a solution here, we should do so in a way that does not encourage the confg-file writer to duplicate driver logic.

In any case, I definitely consider this follow-up work.

 This was a bad example. To obtain more powerful solution clang need something like gcc spec files. This is completely different topic.

Thanks,
--Serge


-- 
Hal Finkel
Lead, Compiler Technology and Programming Languages
Leadership Computing Facility
Argonne National Laboratory


_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev

-- 
Hal Finkel
Lead, Compiler Technology and Programming Languages
Leadership Computing Facility
Argonne National Laboratory

_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Configuration files

Ivan Garramona via cfe-dev
In reply to this post by Ivan Garramona via cfe-dev


Thanks,
--Serge

2017-03-15 23:21 GMT+07:00 Hal Finkel <[hidden email]>:

On 03/15/2017 10:59 AM, Serge Pavlov wrote:

It looks like the features that may be added to the file format in `@file` construct without disturbing existing functionality are:
- comments,
- splitting long lines with trailing '\',
- suppressing warnings on unused options.
The remaining feature is resolving names in nested `@file` constructs, it must keep current behavior (resolve names relative to current directory). This makes nesting constructs `@file` inconvenient it not useless, but it does not look like a big loss.

Can't you just make it configurable? If parsing an `@file`, do the old thing, otherwise, do the new thing?


Configuration encoded in compiler file name, as in `armv7l-clang` is not related to `@file`, so this functionality implemented in https://reviews.llvm.org/D24933 remains unchanged. Explicit specification of config file in command line should be made by `@file` option, there are several issues as compared to the proposed `--config` option:
- config file must be specified by its full path otherwise it won't be found if the current directory is changed by build script. This limitation should not be a big problem if options are specified in build scripts, but for manual invocations in command line it is inconvenient.
- command line may contain several `@file` constructs, there may be options before `@file`. If we want to distinguish config file from others (for instance, we want to suppress unused option warnings only in such files), we need to assume that only the first option may load it.

So if we are ready to sacrifice some convenience, config files can be implemented on top of `@file` construct. If there are no objections I'll update the patch.

Maybe I'm missing something, but I think you're taking this the wrong way. I don't recommend giving up features in this sense. I think the question is: can we replace the current `@file` processing with the new config-file logic (just with some of the new features turned off)?

If that can't be done reasonably, then I think that's okay too. We just should understand (and document) why.


I understood the question as "can we load config files with @file not with new option", there were such opinions in this thread above. If the question is only about enhancements to processing of `@file`, the reply is yes. In the LLVM part of this patch config file parsing in implemented on top of `@file` processing (https://reviews.llvm.org/D24926, function `cl::tokenizeConfigFile`). It seems to me that comments and using '\' to glue lines may be enabled in `@file` as well, as an extension. Other features may be enabled for config files only but clang must somehow know that the file contains configuration.

Thanks,
--Serge

_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Configuration files

Ivan Garramona via cfe-dev


On 03/15/2017 12:04 PM, Serge Pavlov wrote:


Thanks,
--Serge

2017-03-15 23:21 GMT+07:00 Hal Finkel <[hidden email]>:

On 03/15/2017 10:59 AM, Serge Pavlov wrote:

It looks like the features that may be added to the file format in `@file` construct without disturbing existing functionality are:
- comments,
- splitting long lines with trailing '\',
- suppressing warnings on unused options.
The remaining feature is resolving names in nested `@file` constructs, it must keep current behavior (resolve names relative to current directory). This makes nesting constructs `@file` inconvenient it not useless, but it does not look like a big loss.

Can't you just make it configurable? If parsing an `@file`, do the old thing, otherwise, do the new thing?


Configuration encoded in compiler file name, as in `armv7l-clang` is not related to `@file`, so this functionality implemented in https://reviews.llvm.org/D24933 remains unchanged. Explicit specification of config file in command line should be made by `@file` option, there are several issues as compared to the proposed `--config` option:
- config file must be specified by its full path otherwise it won't be found if the current directory is changed by build script. This limitation should not be a big problem if options are specified in build scripts, but for manual invocations in command line it is inconvenient.
- command line may contain several `@file` constructs, there may be options before `@file`. If we want to distinguish config file from others (for instance, we want to suppress unused option warnings only in such files), we need to assume that only the first option may load it.

So if we are ready to sacrifice some convenience, config files can be implemented on top of `@file` construct. If there are no objections I'll update the patch.

Maybe I'm missing something, but I think you're taking this the wrong way. I don't recommend giving up features in this sense. I think the question is: can we replace the current `@file` processing with the new config-file logic (just with some of the new features turned off)?

If that can't be done reasonably, then I think that's okay too. We just should understand (and document) why.


I understood the question as "can we load config files with @file not with new option", there were such opinions in this thread above. If the question is only about enhancements to processing of `@file`, the reply is yes. In the LLVM part of this patch config file parsing in implemented on top of `@file` processing (https://reviews.llvm.org/D24926, function `cl::tokenizeConfigFile`). It seems to me that comments and using '\' to glue lines may be enabled in `@file` as well, as an extension. Other features may be enabled for config files only but clang must somehow know that the file contains configuration.
Sounds good. I'll look at the patches.

 -Hal


Thanks,
--Serge

-- 
Hal Finkel
Lead, Compiler Technology and Programming Languages
Leadership Computing Facility
Argonne National Laboratory

_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Configuration files

Ivan Garramona via cfe-dev
In reply to this post by Ivan Garramona via cfe-dev
On Tue, Mar 14, 2017 at 3:08 PM, Hal Finkel via cfe-dev <[hidden email]> wrote:
2. More elaborated solutions

Many are of opinion that config file in this form is too primitive solution, for instance:
- A good solution must be able to deduce target from arguments (-m32),
- Any workable config-file scheme *must* interact with the various target specification command-line arguments, and allow for setting options dependent upon the actually-selected target,
- Config file must reduce complexity of clang driver.

The presented implementation of config files is only a way to specify set of options, it cannot solve these problems. However it can be extended in future so that, for instance, the following construct could be allowed (particular syntax does not matter):

    …
    #if (argv.contains("-m32"))
    -target i686-unknown-linux-gnu
    #else
    -targer x86_64-unknown-linux-gnu
    #endif
    …

This syntax would require to support `if/else/endif` control construct, and builtin function `argv.contains`. Ability to include an option conditionally is likely to be sufficient to solve large part of the problems mentioned above and can indeed reduce complexity of the driver.

This is appealing, but I think that we really need to be careful here. We don't want to create a system where people are encouraged to replicate complex driver logic in the config files in order to independently figure out which features might be enabled. Of course, we already do this, but the logic ends up in wrapper scripts (and is just as likely to be broken, if not more so). However, if we're going to create a solution here, we should do so in a way that does not encourage the confg-file writer to duplicate driver logic.

 
I'd really like to at least have a *design* for how this can eventually incorporate target-specific options before moving forward with adding a --config option, even if the initial commit won't implement the full design.

I don't believe hand-wavy "maybe we'll add syntax that looks kinda like a comment so older versions will ignore it" is good enough there.

I'd like to again keep in mind the use-case I mentioned a while ago. Approximately every linux distro configures GCC to set their default target cpu levels. E.g., Debian seems to set the following:
- On x86, the default CPU should be i686.
- But on x86-64, the default CPU isn't changed (aka it's left as "x86-64").
- For ppc32/ppc64, the default CPU should be power7 but tune for power8.
- For ppc64le, the default CPU should be power8.
- On ARM (hf), armv7-a should be the default cpu, vfpv3-d16 the default fpu, and it should default to thumb mode.
etc...

Note that those defaults are different on different releases of the distro.

The way you do this with GCC is via options passed to the configure script: --with-arch-32= --with-arch-64= --with-fpu= --with-mode= etc. which turn into values used in the target-specific OPTION_DEFAULT_SPECS macro. Since GCC only builds for one target at a time (or two, if you count 32/64 separately), and you're expected to need to build a new gcc any time you want to cross-compile, that's sufficient.

Clang is intrinsically a cross-compiler, so gcc's solution isn't good enough for clang (nor is clang's current behavior enough). So, what's the plan to actually solve this?

1. There needs to a way to be able to configure the defaults for all supported architectures of the platform (either in a single config, or in multiple that clang knows how to select).

2. Those platform defaults should, somehow, avoid interfering with the use of clang to cross-compile TO a different platform, and be easy to use FROM a different host platform. (find default config via sysroot, maybe?)

3. How do we recommend users select a target?
Do we need to look for the proper defaults when the user specifies "--target $TARGET" to clang?
Or maybe we favor of the "$TARGET-clang" symlink method?
Or maybe "--target" is a low-level feature not recommended for end-users, and we should steer people to using something like "-arch", to select a named architecture within the current platform configuration, like apple does with darwin-based platforms now?

Again, I'm not saying the first patch needs to implement everything, but I do think a design should exist on how this is going to work.

_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
123
Loading...