RFC: refactoring libclangDriver/libclangFrontend to share with Flang

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

RFC: refactoring libclangDriver/libclangFrontend to share with Flang

Keane, Erich via cfe-dev
Hello,

This is a refined design of how the new Flang driver will re-use the
code currently available in Clang without depending on Clang (long-term
goal). This was initially discussed in [1]. Based on the feedback and
after a few weeks of prototyping [2] we are proposing a much smaller set
of changes. Below is detailed summary of the design and how it will
affect Clang. Your input is much appreciated!

# *SUMMARY OF CHANGES FOR CLANG*:
* Make libclangDriver independent of Clang by:
    ** Creating a higher-level, reduced interface over DiagnosticEngine
for compiler drivers to use that does not require Clang's SourceManager
    ** Lifting the TableGen backend for DiagnosticDriverKinds
* Move libclangDriver (together with the TableGen backend for
DiagnosticDriverKinds) out of Clang

# *THINGS TO RE-USE*: libclangDriver
The Flang driver (i.e. "flang"), like Clang driver (i.e. "clang"), will
be implemented in terms of libclangDriver. libclangDriver can already
distinguish between various driver modes [3], including a dedicated mode
for flang: clang::driver::Driver::FlangMode. We already use this
mechanism (via ParsedClangName [4]) to start the driver in
Driver::FlangMode (this is done inside "flang").

One of the key tasks of the driver is to parse the command line options
and translate them into Actions. This part seems re-usable as is.

Next, based on the generated Actions the driver creates jobs (i.e.
instances of clang::driver::Command). At this point (assuming that the
driver is in
Driver::FlangMode):
* a particular ToolChain/Tool is selected (for the preprocess phase [5],
the ToolChain is implemented in Flang.cpp [6] and the selected tool is
simply "flang -fc1")
* compiler driver options (e.g. options for "flang") are translated into
options for the selected tool (e.g. options for "flang -fc1" in the
preprocess phase)

The required top-level logic for this is already available in
libclangDriver [6]. Any new logic that will apply only to Flang will be
implemented in clang::driver::tools::Flang.

# *THINGS NOT TO RE-USE*: libclangFrontend
Once a job representing a call to the Flang frontend driver is
constructed and a dedicate subprocesses is created, the Flang frontend
driver (i.e. "flang -fc1") takes care of the rest.  "flang -fc1" will be:
* a seperate entity (akin "clang -cc1" [7])
* independent of libclangFrontend (and "clang -cc1")
* implemented in terms of libflangFrontend (this library will be part of
the Flang subproject)

So far the implementation of "flang -fc1" and libflangFrontend
(available in our fork [2]) have been heavily inspired by "clang -cc1"
and libclangFrontend, but otherwise are written from scratch. The Flang
frontend driver is unlikely to re-use any code from Clang's frontend
driver at this stage. This seems consistent with what people suggested
in the past (in particular, see this reply from Richard Smith [8]).

Clang's SourceManager is only really needed by DiagnosticsEngine, but
with libclangDriver limited usage of DiagnosticsEngine, we should be
able to remove the dependency on SourceManager completely. From what we
can see, libclangDriver doesn't really need it.

# *DEPENDENCIES ON CLANG*
The end goal is to have a Flang compiler driver implemented in terms of
libclangDriver that does not depend on Clang. This means extracting
libclangDriver from Clang and moving it to a separate sub-project. To
this end we have to make sure that libclangDriver no longer depends on
Clang. This is the list of dependencies that we have identified:
* DiagnosticsEngine (+DiagnotsicOptions + DiagnosticIDs +
DiagnosticConsumer)
* TableGen backend for generating error/warning definitions for
DiagnosticsEngine

Although this list is short (perhaps we missed something?), it contains
some rather complex and pervasive Clang classes that belong in
libclangBasic. Fortunately, libclangDriver uses these classes to a
rather limited extent.

DiagnosticsEngine is used by the driver to print warnings about user
errors made in the options supplied. This is rather basic usage compared
to reporting errors/warning generating by the parser or semantic
analysis (e.g. we don't care about specific locations in files, macro
expansions, etc). We propose creating a thin layer above
DiagnosticsEngine to satisfy the dependencies of libclangDriver. This
seems feasible and shouldn't be too disruptive.

The TableGen backend is required to generate DiagnosticDriverKinds.inc,
i.e. the libclangDriver specific errors/warnings. Moving the
corresponding TableGen backend out of Clang (together with
libclangDriver) seems like the most straightforward approach to this.
Any frontend specific diagnostic definitions should remain in Clang. Any
use of these within libclangDriver can be dealt with on a case-by-case
basis.

# *COMPILER DRIVER OPTIONS*
To handle Flang options we propose to:
* Use ClangFlags [9] to identify Flang options (we will add a dedicated
enum for Flang, e.g. FlangOption)
* Tweak Driver::PrintHelp [10] to only display the appropriate options
depending on the driver mode
* Add new Flang options for libClangDriver to the main DriverOptTable
[11] table, perhaps via a separate *.td file

We think this has the benefit of being simple and extending existing
interfaces. It may be worth investigating a way to make this scale out a
bit more - cf. [12] - and we propose that as a future enhancement. We
should emphasise that currently libclangDriver creates only one instance
of DriverOptTable [11] that holds all available options. In our design
this table will hold options for both Clang and Flang.

Flang will re-use many of the options already available via
libclangDriver. C and C++ specific options are also relevant. A common
pattern in HPC apps is mixed C++ and Fortran use in the same source
base. In such mixed-source cases, it is useful for the compiler driver
to be able to handle both at the same time. We will also add some new
options, but it's unlikely to be a long list. Taking gfortran as a
reference, the new options would be a very small fraction of what
libclangDriver already supports.

# *NEXT STEPS*
The proposed changes (summarized at the top) are relatively small and
will only affect libclangDriver.  We would like to start upstreaming our
patches into Flang at the same as lifting libclangDriver out of Clang
into a separate project. This means submitting some patches into Clang
while libclangDriver is still part of Clang. If the overall plan sounds
sensible then shortly we'll prepare a separate, more detailed RFC that
focuses on the usage of DiagnosticsEngine in libclangDriver.

All input appreciated.

Thanks for reading.

Andrzej Warzynski
On behalf on the Arm Fortran Team


[1] http://lists.llvm.org/pipermail/llvm-dev/2020-June/141994.html
[2] https://github.com/banach-space/llvm-project
[3]
https://github.com/llvm/llvm-project/blob/cbb3571b0df5a0948602aa4d2b913b64270143ff/clang/include/clang/Driver/Driver.h#L64
[4]
https://github.com/llvm/llvm-project/blob/cbb3571b0df5a0948602aa4d2b913b64270143ff/clang/include/clang/Driver/ToolChain.h#L63
[5]
https://github.com/llvm/llvm-project/blob/cbb3571b0df5a0948602aa4d2b913b64270143ff/clang/include/clang/Driver/Phases.h#L14
[6]
https://github.com/llvm/llvm-project/blob/cbb3571b0df5a0948602aa4d2b913b64270143ff/clang/lib/Driver/ToolChains/Flang.cpp
[7]
https://github.com/llvm/llvm-project/blob/cbb3571b0df5a0948602aa4d2b913b64270143ff/clang/tools/driver/cc1_main.cpp#L184
[8] http://lists.llvm.org/pipermail/llvm-dev/2020-June/142024.html
[9]
https://github.com/llvm/llvm-project/blob/cbb3571b0df5a0948602aa4d2b913b64270143ff/clang/include/clang/Driver/Options.h#L26
[10]
https://github.com/llvm/llvm-project/blob/cbb3571b0df5a0948602aa4d2b913b64270143ff/clang/lib/Driver/Driver.cpp#L1559
[11]
https://github.com/llvm/llvm-project/blob/cbb3571b0df5a0948602aa4d2b913b64270143ff/clang/lib/Driver/DriverOptions.cpp#L43
[12] http://lists.llvm.org/pipermail/llvm-dev/2020-July/143745.html
_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: RFC: refactoring libclangDriver/libclangFrontend to share with Flang

Keane, Erich via cfe-dev

 

Hi Andrzej,

 

Per below references …

 

> [9] https://github.com/llvm/llvm-project/blob/cbb3571b0df5a0948602aa4d2b913b64270143ff/clang/include/clang/Driver/Options.h#L26

> [10] https://github.com/llvm/llvm-project/blob/cbb3571b0df5a0948602aa4d2b913b64270143ff/clang/lib/Driver/Driver.cpp#L1559

> [11] https://github.com/llvm/llvm-project/blob/cbb3571b0df5a0948602aa4d2b913b64270143ff/clang/lib/Driver/DriverOptions.cpp#L43

 

> # *COMPILER DRIVER OPTIONS*

> To handle Flang options we propose to:

> * Use ClangFlags [9] to identify Flang options (we will add a dedicated

enum for Flang, e.g. FlangOption)

> * Tweak Driver::PrintHelp [10] to only display the appropriate options

depending on the driver mode

 

What about downstream compiler products that want to provide a more succinct –help output (clang –help will write 900+ lines to stdout)?

It seems like there should be a toolchain-specific way to filter the DriverOptTable further than driver mode and existing groups and flags.

 

> * Add new Flang options for libClangDriver to the main DriverOptTable

[11] table, perhaps via a separate *.td file

 

This would still emit a unified Options.inc file, yes?

 

As my immediate interest is in customizing the –help output in a toolchain specific way, I think there are (at least) two concerns:

 

-          Adding toolchain specific groups to the Options.inc output (maybe via a separate .td file like you suggest for flang)

-          A mechanism the toolchain can latch onto to filter the DriverOptTable in a toolchain specific way

o   Would like to be able to provide a custom HelpText field and associate an option with a toolchain-specific group

o   Maybe add mutable fields to the OptTable::Info that the toolchain can use to annotate the DriverOptTable?

 

I’m interested to learn what other downstream compiler products like armclang have done to solve this problem. Do they implement something entirely disjoint from the upstream code base? or do they use the OptTable infrastructure?

 

~ Todd Snider


_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: RFC: refactoring libclangDriver/libclangFrontend to share with Flang

Keane, Erich via cfe-dev
Todd,

Thank you for your email!

At this stage we are only focusing on a relatively basic driver
supporting a very small amount of options compared to Clang. Our main
goal for now is to leverage libclangDriver without introducing the
dependency on Clang. This means refactoring libclangDriver enough so
that it can be moved to a dedicated subproject.

I agree that the way options are handled in libclangDriver could be a
bit more sympathetic to customising the output of `-help`. Thank you for
raising this! We feel that this could be improved at later time. Knowing
that others face similar issue is re-assuring and will make any future
refactoring worthwhile.

More comments inline.


On 06/08/2020 18:14, Snider, Todd wrote:
>
> What about downstream compiler products that want to provide a more
> succinct �help output (clang �help will write 900+ lines to stdout)?

In this RFC I only focus on the Flang driver (i.e. one specific customer
of libclangDriver), which want to implement upstream.

>
> It seems like there should be a toolchain-specific way to filter the
> DriverOptTable further than driver mode and existing groups and flags.

I agree, but this is out-of-scope of this RFC and what we want to
achieve at this stage. We propose this as a future enhancement.

>
>  > * Add new Flang options for libClangDriver to the main DriverOptTable
>
> [11] table, perhaps via a separate *.td file
>
> This would still emit a unified Options.inc file, yes?
>

Correct, nothing changes in this respect.

> As my immediate interest is in customizing the �help output in a
> toolchain specific way, I think there are (at least) two concerns:
>

I think that these concerns relate to how options are _currently_
handled in libclangDriver in general. We have no plans to change that in
the near future. Instead, we want to leverage the current implementation
and filter options in printHelp via option groups/flags.
>
> I�m interested to learn what other downstream compiler products like
> armclang have done to solve this problem.


In armclang that I'm familiar with (AFAIK, there's more than 1),
filtering is done at the printHelp level via option groups (i.e.
similarly that what we intend to use for Flang).
-Andrzej
_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: [EXTERNAL] Re: RFC: refactoring libclangDriver/libclangFrontend to share with Flang

Keane, Erich via cfe-dev
Hi Andrzej,

Thanks for the quick response.

I realize that customization of the DriverOptTable is not quite in the scope of your RFC, but regarding ...

>> I�m interested to learn what other downstream compiler products like
>> armclang have done to solve this problem.

> In armclang that I'm familiar with (AFAIK, there's more than 1),
> filtering is done at the printHelp level via option groups (i.e.
> similarly that what we intend to use for Flang).

It sounds like the armclang solution might freely be adding to/editing the Options.td file (special group filter + custom HelpText fields), yes?

If that is the case, it is not something that would likely be upstreamed.

Is there someone familiar with how armclang implements a custom help display that could provide more details?

~ Todd

-----Original Message-----
From: Andrzej Warzynski [mailto:[hidden email]]
Sent: Friday, August 7, 2020 10:55 AM
To: Snider, Todd
Cc: '[hidden email]'
Subject: [EXTERNAL] Re: [cfe-dev] RFC: refactoring libclangDriver/libclangFrontend to share with Flang

Todd,

Thank you for your email!

At this stage we are only focusing on a relatively basic driver
supporting a very small amount of options compared to Clang. Our main
goal for now is to leverage libclangDriver without introducing the
dependency on Clang. This means refactoring libclangDriver enough so
that it can be moved to a dedicated subproject.

I agree that the way options are handled in libclangDriver could be a
bit more sympathetic to customising the output of `-help`. Thank you for
raising this! We feel that this could be improved at later time. Knowing
that others face similar issue is re-assuring and will make any future
refactoring worthwhile.

More comments inline.


On 06/08/2020 18:14, Snider, Todd wrote:
>
> What about downstream compiler products that want to provide a more
> succinct �help output (clang �help will write 900+ lines to stdout)?

In this RFC I only focus on the Flang driver (i.e. one specific customer
of libclangDriver), which want to implement upstream.

>
> It seems like there should be a toolchain-specific way to filter the
> DriverOptTable further than driver mode and existing groups and flags.

I agree, but this is out-of-scope of this RFC and what we want to
achieve at this stage. We propose this as a future enhancement.

>
>  > * Add new Flang options for libClangDriver to the main DriverOptTable
>
> [11] table, perhaps via a separate *.td file
>
> This would still emit a unified Options.inc file, yes?
>

Correct, nothing changes in this respect.

> As my immediate interest is in customizing the �help output in a
> toolchain specific way, I think there are (at least) two concerns:
>

I think that these concerns relate to how options are _currently_
handled in libclangDriver in general. We have no plans to change that in
the near future. Instead, we want to leverage the current implementation
and filter options in printHelp via option groups/flags.
>
> I�m interested to learn what other downstream compiler products like
> armclang have done to solve this problem.


In armclang that I'm familiar with (AFAIK, there's more than 1),
filtering is done at the printHelp level via option groups (i.e.
similarly that what we intend to use for Flang).
-Andrzej
_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: RFC: refactoring libclangDriver/libclangFrontend to share with Flang

Keane, Erich via cfe-dev
In reply to this post by Keane, Erich via cfe-dev
Thanks for the writeup, this seems like a good direction to me. Do you have a concrete proposal for where in the LLVM project the Driver will move to?

On Fri, 31 Jul 2020 at 06:24, Andrzej Warzynski via cfe-dev <[hidden email]> wrote:
Hello,

This is a refined design of how the new Flang driver will re-use the
code currently available in Clang without depending on Clang (long-term
goal). This was initially discussed in [1]. Based on the feedback and
after a few weeks of prototyping [2] we are proposing a much smaller set
of changes. Below is detailed summary of the design and how it will
affect Clang. Your input is much appreciated!

# *SUMMARY OF CHANGES FOR CLANG*:
* Make libclangDriver independent of Clang by:
    ** Creating a higher-level, reduced interface over DiagnosticEngine
for compiler drivers to use that does not require Clang's SourceManager
    ** Lifting the TableGen backend for DiagnosticDriverKinds
* Move libclangDriver (together with the TableGen backend for
DiagnosticDriverKinds) out of Clang

# *THINGS TO RE-USE*: libclangDriver
The Flang driver (i.e. "flang"), like Clang driver (i.e. "clang"), will
be implemented in terms of libclangDriver. libclangDriver can already
distinguish between various driver modes [3], including a dedicated mode
for flang: clang::driver::Driver::FlangMode. We already use this
mechanism (via ParsedClangName [4]) to start the driver in
Driver::FlangMode (this is done inside "flang").

One of the key tasks of the driver is to parse the command line options
and translate them into Actions. This part seems re-usable as is.

Next, based on the generated Actions the driver creates jobs (i.e.
instances of clang::driver::Command). At this point (assuming that the
driver is in
Driver::FlangMode):
* a particular ToolChain/Tool is selected (for the preprocess phase [5],
the ToolChain is implemented in Flang.cpp [6] and the selected tool is
simply "flang -fc1")
* compiler driver options (e.g. options for "flang") are translated into
options for the selected tool (e.g. options for "flang -fc1" in the
preprocess phase)

The required top-level logic for this is already available in
libclangDriver [6]. Any new logic that will apply only to Flang will be
implemented in clang::driver::tools::Flang.

# *THINGS NOT TO RE-USE*: libclangFrontend
Once a job representing a call to the Flang frontend driver is
constructed and a dedicate subprocesses is created, the Flang frontend
driver (i.e. "flang -fc1") takes care of the rest.  "flang -fc1" will be:
* a seperate entity (akin "clang -cc1" [7])
* independent of libclangFrontend (and "clang -cc1")
* implemented in terms of libflangFrontend (this library will be part of
the Flang subproject)

So far the implementation of "flang -fc1" and libflangFrontend
(available in our fork [2]) have been heavily inspired by "clang -cc1"
and libclangFrontend, but otherwise are written from scratch. The Flang
frontend driver is unlikely to re-use any code from Clang's frontend
driver at this stage. This seems consistent with what people suggested
in the past (in particular, see this reply from Richard Smith [8]).

Clang's SourceManager is only really needed by DiagnosticsEngine, but
with libclangDriver limited usage of DiagnosticsEngine, we should be
able to remove the dependency on SourceManager completely. From what we
can see, libclangDriver doesn't really need it.

# *DEPENDENCIES ON CLANG*
The end goal is to have a Flang compiler driver implemented in terms of
libclangDriver that does not depend on Clang. This means extracting
libclangDriver from Clang and moving it to a separate sub-project. To
this end we have to make sure that libclangDriver no longer depends on
Clang. This is the list of dependencies that we have identified:
* DiagnosticsEngine (+DiagnotsicOptions + DiagnosticIDs +
DiagnosticConsumer)
* TableGen backend for generating error/warning definitions for
DiagnosticsEngine

Although this list is short (perhaps we missed something?), it contains
some rather complex and pervasive Clang classes that belong in
libclangBasic. Fortunately, libclangDriver uses these classes to a
rather limited extent.

DiagnosticsEngine is used by the driver to print warnings about user
errors made in the options supplied. This is rather basic usage compared
to reporting errors/warning generating by the parser or semantic
analysis (e.g. we don't care about specific locations in files, macro
expansions, etc). We propose creating a thin layer above
DiagnosticsEngine to satisfy the dependencies of libclangDriver. This
seems feasible and shouldn't be too disruptive.

The TableGen backend is required to generate DiagnosticDriverKinds.inc,
i.e. the libclangDriver specific errors/warnings. Moving the
corresponding TableGen backend out of Clang (together with
libclangDriver) seems like the most straightforward approach to this.
Any frontend specific diagnostic definitions should remain in Clang. Any
use of these within libclangDriver can be dealt with on a case-by-case
basis.

# *COMPILER DRIVER OPTIONS*
To handle Flang options we propose to:
* Use ClangFlags [9] to identify Flang options (we will add a dedicated
enum for Flang, e.g. FlangOption)
* Tweak Driver::PrintHelp [10] to only display the appropriate options
depending on the driver mode
* Add new Flang options for libClangDriver to the main DriverOptTable
[11] table, perhaps via a separate *.td file

We think this has the benefit of being simple and extending existing
interfaces. It may be worth investigating a way to make this scale out a
bit more - cf. [12] - and we propose that as a future enhancement. We
should emphasise that currently libclangDriver creates only one instance
of DriverOptTable [11] that holds all available options. In our design
this table will hold options for both Clang and Flang.

Flang will re-use many of the options already available via
libclangDriver. C and C++ specific options are also relevant. A common
pattern in HPC apps is mixed C++ and Fortran use in the same source
base. In such mixed-source cases, it is useful for the compiler driver
to be able to handle both at the same time. We will also add some new
options, but it's unlikely to be a long list. Taking gfortran as a
reference, the new options would be a very small fraction of what
libclangDriver already supports.

# *NEXT STEPS*
The proposed changes (summarized at the top) are relatively small and
will only affect libclangDriver.  We would like to start upstreaming our
patches into Flang at the same as lifting libclangDriver out of Clang
into a separate project. This means submitting some patches into Clang
while libclangDriver is still part of Clang. If the overall plan sounds
sensible then shortly we'll prepare a separate, more detailed RFC that
focuses on the usage of DiagnosticsEngine in libclangDriver.

All input appreciated.

Thanks for reading.

Andrzej Warzynski
On behalf on the Arm Fortran Team


[1] http://lists.llvm.org/pipermail/llvm-dev/2020-June/141994.html
[2] https://github.com/banach-space/llvm-project
[3]
https://github.com/llvm/llvm-project/blob/cbb3571b0df5a0948602aa4d2b913b64270143ff/clang/include/clang/Driver/Driver.h#L64
[4]
https://github.com/llvm/llvm-project/blob/cbb3571b0df5a0948602aa4d2b913b64270143ff/clang/include/clang/Driver/ToolChain.h#L63
[5]
https://github.com/llvm/llvm-project/blob/cbb3571b0df5a0948602aa4d2b913b64270143ff/clang/include/clang/Driver/Phases.h#L14
[6]
https://github.com/llvm/llvm-project/blob/cbb3571b0df5a0948602aa4d2b913b64270143ff/clang/lib/Driver/ToolChains/Flang.cpp
[7]
https://github.com/llvm/llvm-project/blob/cbb3571b0df5a0948602aa4d2b913b64270143ff/clang/tools/driver/cc1_main.cpp#L184
[8] http://lists.llvm.org/pipermail/llvm-dev/2020-June/142024.html
[9]
https://github.com/llvm/llvm-project/blob/cbb3571b0df5a0948602aa4d2b913b64270143ff/clang/include/clang/Driver/Options.h#L26
[10]
https://github.com/llvm/llvm-project/blob/cbb3571b0df5a0948602aa4d2b913b64270143ff/clang/lib/Driver/Driver.cpp#L1559
[11]
https://github.com/llvm/llvm-project/blob/cbb3571b0df5a0948602aa4d2b913b64270143ff/clang/lib/Driver/DriverOptions.cpp#L43
[12] http://lists.llvm.org/pipermail/llvm-dev/2020-July/143745.html
_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev

_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: RFC: refactoring libclangDriver/libclangFrontend to share with Flang

Keane, Erich via cfe-dev
Hi Richard,

On 11/08/2020 22:25, Richard Smith wrote:
> Do you have a concrete proposal for where in the LLVM project the Driver will
> move to?

When we discussed this last time, I proposed [1] to move this new
library to a dedicated subproject: `frontend-support`. However, since
then scope of this has been narrowed down.

I think that first we need to make libclangDriver independent of any
Clang libraries. Once that's done, moving the new library to a dedicated
sub-project should be relatively easy. The actual name/location could be
discussed then - with a much shorter RFC :) Until then everything stays
in Clang.

Btw, do you have any preferences?

-Andrzej

[1] http://lists.llvm.org/pipermail/llvm-dev/2020-June/142186.html

>
> On Fri, 31 Jul 2020 at 06:24, Andrzej Warzynski via cfe-dev
> <[hidden email] <mailto:[hidden email]>> wrote:
>
>     Hello,
>
>     This is a refined design of how the new Flang driver will re-use the
>     code currently available in Clang without depending on Clang (long-term
>     goal). This was initially discussed in [1]. Based on the feedback and
>     after a few weeks of prototyping [2] we are proposing a much smaller
>     set
>     of changes. Below is detailed summary of the design and how it will
>     affect Clang. Your input is much appreciated!
>
>     # *SUMMARY OF CHANGES FOR CLANG*:
>     * Make libclangDriver independent of Clang by:
>          ** Creating a higher-level, reduced interface over
>     DiagnosticEngine
>     for compiler drivers to use that does not require Clang's SourceManager
>          ** Lifting the TableGen backend for DiagnosticDriverKinds
>     * Move libclangDriver (together with the TableGen backend for
>     DiagnosticDriverKinds) out of Clang
>
>     # *THINGS TO RE-USE*: libclangDriver
>     The Flang driver (i.e. "flang"), like Clang driver (i.e. "clang"), will
>     be implemented in terms of libclangDriver. libclangDriver can already
>     distinguish between various driver modes [3], including a dedicated
>     mode
>     for flang: clang::driver::Driver::FlangMode. We already use this
>     mechanism (via ParsedClangName [4]) to start the driver in
>     Driver::FlangMode (this is done inside "flang").
>
>     One of the key tasks of the driver is to parse the command line options
>     and translate them into Actions. This part seems re-usable as is.
>
>     Next, based on the generated Actions the driver creates jobs (i.e.
>     instances of clang::driver::Command). At this point (assuming that the
>     driver is in
>     Driver::FlangMode):
>     * a particular ToolChain/Tool is selected (for the preprocess phase
>     [5],
>     the ToolChain is implemented in Flang.cpp [6] and the selected tool is
>     simply "flang -fc1")
>     * compiler driver options (e.g. options for "flang") are translated
>     into
>     options for the selected tool (e.g. options for "flang -fc1" in the
>     preprocess phase)
>
>     The required top-level logic for this is already available in
>     libclangDriver [6]. Any new logic that will apply only to Flang will be
>     implemented in clang::driver::tools::Flang.
>
>     # *THINGS NOT TO RE-USE*: libclangFrontend
>     Once a job representing a call to the Flang frontend driver is
>     constructed and a dedicate subprocesses is created, the Flang frontend
>     driver (i.e. "flang -fc1") takes care of the rest.  "flang -fc1"
>     will be:
>     * a seperate entity (akin "clang -cc1" [7])
>     * independent of libclangFrontend (and "clang -cc1")
>     * implemented in terms of libflangFrontend (this library will be
>     part of
>     the Flang subproject)
>
>     So far the implementation of "flang -fc1" and libflangFrontend
>     (available in our fork [2]) have been heavily inspired by "clang -cc1"
>     and libclangFrontend, but otherwise are written from scratch. The Flang
>     frontend driver is unlikely to re-use any code from Clang's frontend
>     driver at this stage. This seems consistent with what people suggested
>     in the past (in particular, see this reply from Richard Smith [8]).
>
>     Clang's SourceManager is only really needed by DiagnosticsEngine, but
>     with libclangDriver limited usage of DiagnosticsEngine, we should be
>     able to remove the dependency on SourceManager completely. From what we
>     can see, libclangDriver doesn't really need it.
>
>     # *DEPENDENCIES ON CLANG*
>     The end goal is to have a Flang compiler driver implemented in terms of
>     libclangDriver that does not depend on Clang. This means extracting
>     libclangDriver from Clang and moving it to a separate sub-project. To
>     this end we have to make sure that libclangDriver no longer depends on
>     Clang. This is the list of dependencies that we have identified:
>     * DiagnosticsEngine (+DiagnotsicOptions + DiagnosticIDs +
>     DiagnosticConsumer)
>     * TableGen backend for generating error/warning definitions for
>     DiagnosticsEngine
>
>     Although this list is short (perhaps we missed something?), it contains
>     some rather complex and pervasive Clang classes that belong in
>     libclangBasic. Fortunately, libclangDriver uses these classes to a
>     rather limited extent.
>
>     DiagnosticsEngine is used by the driver to print warnings about user
>     errors made in the options supplied. This is rather basic usage
>     compared
>     to reporting errors/warning generating by the parser or semantic
>     analysis (e.g. we don't care about specific locations in files, macro
>     expansions, etc). We propose creating a thin layer above
>     DiagnosticsEngine to satisfy the dependencies of libclangDriver. This
>     seems feasible and shouldn't be too disruptive.
>
>     The TableGen backend is required to generate DiagnosticDriverKinds.inc,
>     i.e. the libclangDriver specific errors/warnings. Moving the
>     corresponding TableGen backend out of Clang (together with
>     libclangDriver) seems like the most straightforward approach to this.
>     Any frontend specific diagnostic definitions should remain in Clang.
>     Any
>     use of these within libclangDriver can be dealt with on a case-by-case
>     basis.
>
>     # *COMPILER DRIVER OPTIONS*
>     To handle Flang options we propose to:
>     * Use ClangFlags [9] to identify Flang options (we will add a dedicated
>     enum for Flang, e.g. FlangOption)
>     * Tweak Driver::PrintHelp [10] to only display the appropriate options
>     depending on the driver mode
>     * Add new Flang options for libClangDriver to the main DriverOptTable
>     [11] table, perhaps via a separate *.td file
>
>     We think this has the benefit of being simple and extending existing
>     interfaces. It may be worth investigating a way to make this scale
>     out a
>     bit more - cf. [12] - and we propose that as a future enhancement. We
>     should emphasise that currently libclangDriver creates only one
>     instance
>     of DriverOptTable [11] that holds all available options. In our design
>     this table will hold options for both Clang and Flang.
>
>     Flang will re-use many of the options already available via
>     libclangDriver. C and C++ specific options are also relevant. A common
>     pattern in HPC apps is mixed C++ and Fortran use in the same source
>     base. In such mixed-source cases, it is useful for the compiler driver
>     to be able to handle both at the same time. We will also add some new
>     options, but it's unlikely to be a long list. Taking gfortran as a
>     reference, the new options would be a very small fraction of what
>     libclangDriver already supports.
>
>     # *NEXT STEPS*
>     The proposed changes (summarized at the top) are relatively small and
>     will only affect libclangDriver.  We would like to start upstreaming
>     our
>     patches into Flang at the same as lifting libclangDriver out of Clang
>     into a separate project. This means submitting some patches into Clang
>     while libclangDriver is still part of Clang. If the overall plan sounds
>     sensible then shortly we'll prepare a separate, more detailed RFC that
>     focuses on the usage of DiagnosticsEngine in libclangDriver.
>
>     All input appreciated.
>
>     Thanks for reading.
>
>     Andrzej Warzynski
>     On behalf on the Arm Fortran Team
>
>
>     [1] http://lists.llvm.org/pipermail/llvm-dev/2020-June/141994.html
>     [2] https://github.com/banach-space/llvm-project
>     [3]
>     https://github.com/llvm/llvm-project/blob/cbb3571b0df5a0948602aa4d2b913b64270143ff/clang/include/clang/Driver/Driver.h#L64
>     [4]
>     https://github.com/llvm/llvm-project/blob/cbb3571b0df5a0948602aa4d2b913b64270143ff/clang/include/clang/Driver/ToolChain.h#L63
>     [5]
>     https://github.com/llvm/llvm-project/blob/cbb3571b0df5a0948602aa4d2b913b64270143ff/clang/include/clang/Driver/Phases.h#L14
>     [6]
>     https://github.com/llvm/llvm-project/blob/cbb3571b0df5a0948602aa4d2b913b64270143ff/clang/lib/Driver/ToolChains/Flang.cpp
>     [7]
>     https://github.com/llvm/llvm-project/blob/cbb3571b0df5a0948602aa4d2b913b64270143ff/clang/tools/driver/cc1_main.cpp#L184
>     [8] http://lists.llvm.org/pipermail/llvm-dev/2020-June/142024.html
>     [9]
>     https://github.com/llvm/llvm-project/blob/cbb3571b0df5a0948602aa4d2b913b64270143ff/clang/include/clang/Driver/Options.h#L26
>     [10]
>     https://github.com/llvm/llvm-project/blob/cbb3571b0df5a0948602aa4d2b913b64270143ff/clang/lib/Driver/Driver.cpp#L1559
>     [11]
>     https://github.com/llvm/llvm-project/blob/cbb3571b0df5a0948602aa4d2b913b64270143ff/clang/lib/Driver/DriverOptions.cpp#L43
>     [12] http://lists.llvm.org/pipermail/llvm-dev/2020-July/143745.html
>     _______________________________________________
>     cfe-dev mailing list
>     [hidden email] <mailto:[hidden email]>
>     https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>
_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev