Controlling instantiation of templates from PCH

classic Classic list List threaded Threaded
25 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Re: Controlling instantiation of templates from PCH

Gavin Cui via cfe-dev
Sorry for the delayed response.

I'm getting a bit confused by the layers of this thread, so I'll see if I can sum up my understanding & see if that's any use.

Currently there's a difference between PCH and Clang Header Modules semantics with regard to pending instantiations
In PCH, pending instantiations are all performed at any use of the PCH.
In modules pending instantiations are done at the end of building the header module (so they're performed once).
This difference is the cause of the original compile time cost you were investigating (somewhat independent of the duplicate code in object files)

So, step 1) perhaps we can just remove the PCH mode ^ here, and do the header module thing (perform pending instantiations at the end of building the PCH) for PCH as well.
If you want to do this ^ work, I think a patch would be welcome (I don't know much about where the distinction is between PCH and modules that causes this difference in how pending instantiations are handled - I might be able to find the code a little easier than you, but perhaps you can have a look around & see if you can find it - or Richard might be able to tell you/us)


Separate from that, there's some desire to be able to build an object file from a PCH to remove duplicate template instantiations in object files. (reducing compile time, reducing object size, maybe reducing link time).
To do this - it sounds like there's already a flag (-building-pch-with-obj) for building PCHs that will have a separate object file. I've no idea what this flag does today - does it already do what you want in terms of moving template instantiations and inline functions into that object? Or does it do something else (what else?)? If it doesn't do what you want already, and you want to extend it to do what you want - yes, I'd recommend piggy-backing on the exsiting Clang Header Modules modular code generation features for this.

& yes, potentially the pending instantiations could be moved over from the PCH-or-header-module generation step, to the PCH-or-header-module->Object step. (to benefit both PCH and header modules)

- Dave


On Fri, May 31, 2019 at 2:36 PM Lubos Lunak <[hidden email]> wrote:
On Friday 31 of May 2019, David Blaikie wrote:
> On Thu, May 30, 2019 at 6:04 AM Lubos Lunak <[hidden email]> wrote:
> > >  Two questions come to mind here:
> > > - is it reasonably ready for use?
> > > - how much work would it be to use it?
> > >
> > >  I tried Clang modules after they were mentioned in the first reply,
> > > and I got the impression that they need preparation for every header
> > > file used by the project, even external ones. Unless that can be
> > > automated, I don't
> > > quite see that happening for something the size and complexity of
> > > LibreOffice (we don't manually create our headers-to-become-PCHs
> > > either). And an unofficial tongue-in-cheek motton of LibreOffice is
> > > "proudly breaking your toolchain since 1985", so unless modules are
> > > reasonably usable, we'll run into all the bugs there and nobody will
> > > want to use it.
> > >
> > >  If doing this is good technically, long-term, fine, do it. But I'd
> > > like to
> > > have something that works this summer, and my rather simple patch can
> > > deliver that.
> >
> >  And so this part would be irrelevant in that case I hope?
>
> Right. (I mean, a separate question is whether you'd want to use modules -
> but yes, at the very least it does involve standardizing your headers on
> "well behaved" sort of restraints (basically "you can include the header
> anywhere, any time, and it always behaves the same way" - so not having
> headers that depend on macros locally defined to different values in
> different translation units, etc) - but yeah, it's a lot more work than the
> PCH situation)


 We can try modules eventually, but as said above, I expect the switch to
those would be possibly quite some work, and there's also the question how
build tools like ccache and icecream would cope with modules, so unlikely
now.

>
> > > > * Moving the pending instantiation processing to the end of the PCH
> > > > would make PCH generation a little slower, but given a project would
> > > > only have one PCH that might not be a huge problem.
> > >
> > >  I think that would be very well worth it.
> >
> >  This should be the same either way.

 Just to make it clear, this means "this should be worth it for either case of
whether I understood correctly or not that we should use modules". However
I'm later basically contradicting this by saying that if the build mode is
switched to generate an object file to aacompany the PCH then this shouldn't
be done.

> Yeah, I think I'd misunderstood your proposal - I had assumed there was a
> separation between PCH generation and the PCH->Object step (& in that
> latter step, the pending instantiations would be done). Sounds like your
> .h->PCH step also generates the object?


 My current plan is still to have a separate PCH->object step, but the idea
now is to use -building-pch-with-obj. So the build steps now should be:

# generate PCH and mark it as having accompanying .o
clang++ -c precompiled.hxx -o precompiled.pch -Xclang -building-pch-with-obj
# generate that .o
clang++ -c empty.cxx -include-pch
precompiled.pch -Xclang -building-pch-with-obj
# compile the rest
clang++ -c whatever.cxx -include-pch precompiled.pch

 So the first step will mark the PCH, the second step will generate all the
shared code, and the remaining steps will see the PCH as marked and will skip
generating things again. I like this separation better than /Yu merging the
first two steps into one, here each step generates just one output and
compilations depending on the PCH can already run alongside generating the
PCH's object. And keeping instantiating PCH templates the way it is now
instead of moving it to the end of PCH creation would mean that work is done
only in the second step, so the PCH generation wouldn't get slower.

 And this could be easily later extended to e.g. non-template inline functions
that get out-of-line copies in debug mode:
- Are we using a marked PCH and -building-pch-with-obj is set? => generate
shared copies
- Are we using a marked PCH but -building-pch-with-obj is not set? => skip
generation
- The PCH is not marked as having an object? => work normally

 I don't know how this relates to modules, but for PCHs I expect this should
work fine.

> But I still suspect whatever that implements isn't quite modules-codegen,
> but I could be wrong.


 I still don't quite get how modules relate to my patch, so I can't really
comment on this.

--
 Lubos Lunak

_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: Controlling instantiation of templates from PCH

Gavin Cui via cfe-dev
Hi everyone,

Just wanted to chime in and mention another mode of operations for PCHs that has not been discussed yet.

> So, step 1) perhaps we can just remove the PCH mode ^ here, and do the header module thing (perform pending instantiations at the end of building the PCH) for PCH as well.

Preamble optimization in code completion (used by libclang and clangd) is where this mode is actually useful, so we should probably avoid removing this completely without prior investigation of how this affects the time until initial completion (I would expect the optimization to be significant and useful).
That being said, changing the clang binary behavior seems ok, but we probably want to keep this option in the C++ API.

On Tue, Jun 18, 2019 at 12:18 AM David Blaikie via cfe-dev <[hidden email]> wrote:
Sorry for the delayed response.

I'm getting a bit confused by the layers of this thread, so I'll see if I can sum up my understanding & see if that's any use.

Currently there's a difference between PCH and Clang Header Modules semantics with regard to pending instantiations
In PCH, pending instantiations are all performed at any use of the PCH.
In modules pending instantiations are done at the end of building the header module (so they're performed once).
This difference is the cause of the original compile time cost you were investigating (somewhat independent of the duplicate code in object files)

So, step 1) perhaps we can just remove the PCH mode ^ here, and do the header module thing (perform pending instantiations at the end of building the PCH) for PCH as well.
If you want to do this ^ work, I think a patch would be welcome (I don't know much about where the distinction is between PCH and modules that causes this difference in how pending instantiations are handled - I might be able to find the code a little easier than you, but perhaps you can have a look around & see if you can find it - or Richard might be able to tell you/us)


Separate from that, there's some desire to be able to build an object file from a PCH to remove duplicate template instantiations in object files. (reducing compile time, reducing object size, maybe reducing link time).
To do this - it sounds like there's already a flag (-building-pch-with-obj) for building PCHs that will have a separate object file. I've no idea what this flag does today - does it already do what you want in terms of moving template instantiations and inline functions into that object? Or does it do something else (what else?)? If it doesn't do what you want already, and you want to extend it to do what you want - yes, I'd recommend piggy-backing on the exsiting Clang Header Modules modular code generation features for this.

& yes, potentially the pending instantiations could be moved over from the PCH-or-header-module generation step, to the PCH-or-header-module->Object step. (to benefit both PCH and header modules)

- Dave


On Fri, May 31, 2019 at 2:36 PM Lubos Lunak <[hidden email]> wrote:
On Friday 31 of May 2019, David Blaikie wrote:
> On Thu, May 30, 2019 at 6:04 AM Lubos Lunak <[hidden email]> wrote:
> > >  Two questions come to mind here:
> > > - is it reasonably ready for use?
> > > - how much work would it be to use it?
> > >
> > >  I tried Clang modules after they were mentioned in the first reply,
> > > and I got the impression that they need preparation for every header
> > > file used by the project, even external ones. Unless that can be
> > > automated, I don't
> > > quite see that happening for something the size and complexity of
> > > LibreOffice (we don't manually create our headers-to-become-PCHs
> > > either). And an unofficial tongue-in-cheek motton of LibreOffice is
> > > "proudly breaking your toolchain since 1985", so unless modules are
> > > reasonably usable, we'll run into all the bugs there and nobody will
> > > want to use it.
> > >
> > >  If doing this is good technically, long-term, fine, do it. But I'd
> > > like to
> > > have something that works this summer, and my rather simple patch can
> > > deliver that.
> >
> >  And so this part would be irrelevant in that case I hope?
>
> Right. (I mean, a separate question is whether you'd want to use modules -
> but yes, at the very least it does involve standardizing your headers on
> "well behaved" sort of restraints (basically "you can include the header
> anywhere, any time, and it always behaves the same way" - so not having
> headers that depend on macros locally defined to different values in
> different translation units, etc) - but yeah, it's a lot more work than the
> PCH situation)


 We can try modules eventually, but as said above, I expect the switch to
those would be possibly quite some work, and there's also the question how
build tools like ccache and icecream would cope with modules, so unlikely
now.

>
> > > > * Moving the pending instantiation processing to the end of the PCH
> > > > would make PCH generation a little slower, but given a project would
> > > > only have one PCH that might not be a huge problem.
> > >
> > >  I think that would be very well worth it.
> >
> >  This should be the same either way.

 Just to make it clear, this means "this should be worth it for either case of
whether I understood correctly or not that we should use modules". However
I'm later basically contradicting this by saying that if the build mode is
switched to generate an object file to aacompany the PCH then this shouldn't
be done.

> Yeah, I think I'd misunderstood your proposal - I had assumed there was a
> separation between PCH generation and the PCH->Object step (& in that
> latter step, the pending instantiations would be done). Sounds like your
> .h->PCH step also generates the object?


 My current plan is still to have a separate PCH->object step, but the idea
now is to use -building-pch-with-obj. So the build steps now should be:

# generate PCH and mark it as having accompanying .o
clang++ -c precompiled.hxx -o precompiled.pch -Xclang -building-pch-with-obj
# generate that .o
clang++ -c empty.cxx -include-pch
precompiled.pch -Xclang -building-pch-with-obj
# compile the rest
clang++ -c whatever.cxx -include-pch precompiled.pch

 So the first step will mark the PCH, the second step will generate all the
shared code, and the remaining steps will see the PCH as marked and will skip
generating things again. I like this separation better than /Yu merging the
first two steps into one, here each step generates just one output and
compilations depending on the PCH can already run alongside generating the
PCH's object. And keeping instantiating PCH templates the way it is now
instead of moving it to the end of PCH creation would mean that work is done
only in the second step, so the PCH generation wouldn't get slower.

 And this could be easily later extended to e.g. non-template inline functions
that get out-of-line copies in debug mode:
- Are we using a marked PCH and -building-pch-with-obj is set? => generate
shared copies
- Are we using a marked PCH but -building-pch-with-obj is not set? => skip
generation
- The PCH is not marked as having an object? => work normally

 I don't know how this relates to modules, but for PCHs I expect this should
work fine.

> But I still suspect whatever that implements isn't quite modules-codegen,
> but I could be wrong.


 I still don't quite get how modules relate to my patch, so I can't really
comment on this.

--
 Lubos Lunak
_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev


--
Regards,
Ilya Biryukov

_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: Controlling instantiation of templates from PCH

Gavin Cui via cfe-dev


On Tue, Jun 18, 2019 at 1:37 AM Ilya Biryukov <[hidden email]> wrote:
Hi everyone,

Just wanted to chime in and mention another mode of operations for PCHs that has not been discussed yet.

> So, step 1) perhaps we can just remove the PCH mode ^ here, and do the header module thing (perform pending instantiations at the end of building the PCH) for PCH as well.

Preamble optimization in code completion (used by libclang and clangd) is where this mode is actually useful, so we should probably avoid removing this completely without prior investigation of how this affects the time until initial completion (I would expect the optimization to be significant and useful).
That being said, changing the clang binary behavior seems ok, but we probably want to keep this option in the C++ API.

OK, so I know basically nothing about this - you mean code completion essentially/does something like building a PCH (without actually writing out a file or anything, I guess) of the prefix up to the completion point? (& so not doing pending instantiations at that point would be important for performance)
 

On Tue, Jun 18, 2019 at 12:18 AM David Blaikie via cfe-dev <[hidden email]> wrote:
Sorry for the delayed response.

I'm getting a bit confused by the layers of this thread, so I'll see if I can sum up my understanding & see if that's any use.

Currently there's a difference between PCH and Clang Header Modules semantics with regard to pending instantiations
In PCH, pending instantiations are all performed at any use of the PCH.
In modules pending instantiations are done at the end of building the header module (so they're performed once).
This difference is the cause of the original compile time cost you were investigating (somewhat independent of the duplicate code in object files)

So, step 1) perhaps we can just remove the PCH mode ^ here, and do the header module thing (perform pending instantiations at the end of building the PCH) for PCH as well.
If you want to do this ^ work, I think a patch would be welcome (I don't know much about where the distinction is between PCH and modules that causes this difference in how pending instantiations are handled - I might be able to find the code a little easier than you, but perhaps you can have a look around & see if you can find it - or Richard might be able to tell you/us)


Separate from that, there's some desire to be able to build an object file from a PCH to remove duplicate template instantiations in object files. (reducing compile time, reducing object size, maybe reducing link time).
To do this - it sounds like there's already a flag (-building-pch-with-obj) for building PCHs that will have a separate object file. I've no idea what this flag does today - does it already do what you want in terms of moving template instantiations and inline functions into that object? Or does it do something else (what else?)? If it doesn't do what you want already, and you want to extend it to do what you want - yes, I'd recommend piggy-backing on the exsiting Clang Header Modules modular code generation features for this.

& yes, potentially the pending instantiations could be moved over from the PCH-or-header-module generation step, to the PCH-or-header-module->Object step. (to benefit both PCH and header modules)

- Dave


On Fri, May 31, 2019 at 2:36 PM Lubos Lunak <[hidden email]> wrote:
On Friday 31 of May 2019, David Blaikie wrote:
> On Thu, May 30, 2019 at 6:04 AM Lubos Lunak <[hidden email]> wrote:
> > >  Two questions come to mind here:
> > > - is it reasonably ready for use?
> > > - how much work would it be to use it?
> > >
> > >  I tried Clang modules after they were mentioned in the first reply,
> > > and I got the impression that they need preparation for every header
> > > file used by the project, even external ones. Unless that can be
> > > automated, I don't
> > > quite see that happening for something the size and complexity of
> > > LibreOffice (we don't manually create our headers-to-become-PCHs
> > > either). And an unofficial tongue-in-cheek motton of LibreOffice is
> > > "proudly breaking your toolchain since 1985", so unless modules are
> > > reasonably usable, we'll run into all the bugs there and nobody will
> > > want to use it.
> > >
> > >  If doing this is good technically, long-term, fine, do it. But I'd
> > > like to
> > > have something that works this summer, and my rather simple patch can
> > > deliver that.
> >
> >  And so this part would be irrelevant in that case I hope?
>
> Right. (I mean, a separate question is whether you'd want to use modules -
> but yes, at the very least it does involve standardizing your headers on
> "well behaved" sort of restraints (basically "you can include the header
> anywhere, any time, and it always behaves the same way" - so not having
> headers that depend on macros locally defined to different values in
> different translation units, etc) - but yeah, it's a lot more work than the
> PCH situation)


 We can try modules eventually, but as said above, I expect the switch to
those would be possibly quite some work, and there's also the question how
build tools like ccache and icecream would cope with modules, so unlikely
now.

>
> > > > * Moving the pending instantiation processing to the end of the PCH
> > > > would make PCH generation a little slower, but given a project would
> > > > only have one PCH that might not be a huge problem.
> > >
> > >  I think that would be very well worth it.
> >
> >  This should be the same either way.

 Just to make it clear, this means "this should be worth it for either case of
whether I understood correctly or not that we should use modules". However
I'm later basically contradicting this by saying that if the build mode is
switched to generate an object file to aacompany the PCH then this shouldn't
be done.

> Yeah, I think I'd misunderstood your proposal - I had assumed there was a
> separation between PCH generation and the PCH->Object step (& in that
> latter step, the pending instantiations would be done). Sounds like your
> .h->PCH step also generates the object?


 My current plan is still to have a separate PCH->object step, but the idea
now is to use -building-pch-with-obj. So the build steps now should be:

# generate PCH and mark it as having accompanying .o
clang++ -c precompiled.hxx -o precompiled.pch -Xclang -building-pch-with-obj
# generate that .o
clang++ -c empty.cxx -include-pch
precompiled.pch -Xclang -building-pch-with-obj
# compile the rest
clang++ -c whatever.cxx -include-pch precompiled.pch

 So the first step will mark the PCH, the second step will generate all the
shared code, and the remaining steps will see the PCH as marked and will skip
generating things again. I like this separation better than /Yu merging the
first two steps into one, here each step generates just one output and
compilations depending on the PCH can already run alongside generating the
PCH's object. And keeping instantiating PCH templates the way it is now
instead of moving it to the end of PCH creation would mean that work is done
only in the second step, so the PCH generation wouldn't get slower.

 And this could be easily later extended to e.g. non-template inline functions
that get out-of-line copies in debug mode:
- Are we using a marked PCH and -building-pch-with-obj is set? => generate
shared copies
- Are we using a marked PCH but -building-pch-with-obj is not set? => skip
generation
- The PCH is not marked as having an object? => work normally

 I don't know how this relates to modules, but for PCHs I expect this should
work fine.

> But I still suspect whatever that implements isn't quite modules-codegen,
> but I could be wrong.


 I still don't quite get how modules relate to my patch, so I can't really
comment on this.

--
 Lubos Lunak
_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev


--
Regards,
Ilya Biryukov

_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: Controlling instantiation of templates from PCH

Gavin Cui via cfe-dev
Exactly, code completion would build the PCH up to a first non-preprocessor token (this typically covers all top #include directives).
This PCH would later be used when reparsing the file for code completion.

A parse for code completion does not need pending instantiations and does not perform them. Doing pending instantiations when building the PCH for the file preamble would potentially mean significantly increased latency until first completion.

On Tue, Jun 18, 2019 at 8:05 PM David Blaikie <[hidden email]> wrote:


On Tue, Jun 18, 2019 at 1:37 AM Ilya Biryukov <[hidden email]> wrote:
Hi everyone,

Just wanted to chime in and mention another mode of operations for PCHs that has not been discussed yet.

> So, step 1) perhaps we can just remove the PCH mode ^ here, and do the header module thing (perform pending instantiations at the end of building the PCH) for PCH as well.

Preamble optimization in code completion (used by libclang and clangd) is where this mode is actually useful, so we should probably avoid removing this completely without prior investigation of how this affects the time until initial completion (I would expect the optimization to be significant and useful).
That being said, changing the clang binary behavior seems ok, but we probably want to keep this option in the C++ API.

OK, so I know basically nothing about this - you mean code completion essentially/does something like building a PCH (without actually writing out a file or anything, I guess) of the prefix up to the completion point? (& so not doing pending instantiations at that point would be important for performance)
 

On Tue, Jun 18, 2019 at 12:18 AM David Blaikie via cfe-dev <[hidden email]> wrote:
Sorry for the delayed response.

I'm getting a bit confused by the layers of this thread, so I'll see if I can sum up my understanding & see if that's any use.

Currently there's a difference between PCH and Clang Header Modules semantics with regard to pending instantiations
In PCH, pending instantiations are all performed at any use of the PCH.
In modules pending instantiations are done at the end of building the header module (so they're performed once).
This difference is the cause of the original compile time cost you were investigating (somewhat independent of the duplicate code in object files)

So, step 1) perhaps we can just remove the PCH mode ^ here, and do the header module thing (perform pending instantiations at the end of building the PCH) for PCH as well.
If you want to do this ^ work, I think a patch would be welcome (I don't know much about where the distinction is between PCH and modules that causes this difference in how pending instantiations are handled - I might be able to find the code a little easier than you, but perhaps you can have a look around & see if you can find it - or Richard might be able to tell you/us)


Separate from that, there's some desire to be able to build an object file from a PCH to remove duplicate template instantiations in object files. (reducing compile time, reducing object size, maybe reducing link time).
To do this - it sounds like there's already a flag (-building-pch-with-obj) for building PCHs that will have a separate object file. I've no idea what this flag does today - does it already do what you want in terms of moving template instantiations and inline functions into that object? Or does it do something else (what else?)? If it doesn't do what you want already, and you want to extend it to do what you want - yes, I'd recommend piggy-backing on the exsiting Clang Header Modules modular code generation features for this.

& yes, potentially the pending instantiations could be moved over from the PCH-or-header-module generation step, to the PCH-or-header-module->Object step. (to benefit both PCH and header modules)

- Dave


On Fri, May 31, 2019 at 2:36 PM Lubos Lunak <[hidden email]> wrote:
On Friday 31 of May 2019, David Blaikie wrote:
> On Thu, May 30, 2019 at 6:04 AM Lubos Lunak <[hidden email]> wrote:
> > >  Two questions come to mind here:
> > > - is it reasonably ready for use?
> > > - how much work would it be to use it?
> > >
> > >  I tried Clang modules after they were mentioned in the first reply,
> > > and I got the impression that they need preparation for every header
> > > file used by the project, even external ones. Unless that can be
> > > automated, I don't
> > > quite see that happening for something the size and complexity of
> > > LibreOffice (we don't manually create our headers-to-become-PCHs
> > > either). And an unofficial tongue-in-cheek motton of LibreOffice is
> > > "proudly breaking your toolchain since 1985", so unless modules are
> > > reasonably usable, we'll run into all the bugs there and nobody will
> > > want to use it.
> > >
> > >  If doing this is good technically, long-term, fine, do it. But I'd
> > > like to
> > > have something that works this summer, and my rather simple patch can
> > > deliver that.
> >
> >  And so this part would be irrelevant in that case I hope?
>
> Right. (I mean, a separate question is whether you'd want to use modules -
> but yes, at the very least it does involve standardizing your headers on
> "well behaved" sort of restraints (basically "you can include the header
> anywhere, any time, and it always behaves the same way" - so not having
> headers that depend on macros locally defined to different values in
> different translation units, etc) - but yeah, it's a lot more work than the
> PCH situation)


 We can try modules eventually, but as said above, I expect the switch to
those would be possibly quite some work, and there's also the question how
build tools like ccache and icecream would cope with modules, so unlikely
now.

>
> > > > * Moving the pending instantiation processing to the end of the PCH
> > > > would make PCH generation a little slower, but given a project would
> > > > only have one PCH that might not be a huge problem.
> > >
> > >  I think that would be very well worth it.
> >
> >  This should be the same either way.

 Just to make it clear, this means "this should be worth it for either case of
whether I understood correctly or not that we should use modules". However
I'm later basically contradicting this by saying that if the build mode is
switched to generate an object file to aacompany the PCH then this shouldn't
be done.

> Yeah, I think I'd misunderstood your proposal - I had assumed there was a
> separation between PCH generation and the PCH->Object step (& in that
> latter step, the pending instantiations would be done). Sounds like your
> .h->PCH step also generates the object?


 My current plan is still to have a separate PCH->object step, but the idea
now is to use -building-pch-with-obj. So the build steps now should be:

# generate PCH and mark it as having accompanying .o
clang++ -c precompiled.hxx -o precompiled.pch -Xclang -building-pch-with-obj
# generate that .o
clang++ -c empty.cxx -include-pch
precompiled.pch -Xclang -building-pch-with-obj
# compile the rest
clang++ -c whatever.cxx -include-pch precompiled.pch

 So the first step will mark the PCH, the second step will generate all the
shared code, and the remaining steps will see the PCH as marked and will skip
generating things again. I like this separation better than /Yu merging the
first two steps into one, here each step generates just one output and
compilations depending on the PCH can already run alongside generating the
PCH's object. And keeping instantiating PCH templates the way it is now
instead of moving it to the end of PCH creation would mean that work is done
only in the second step, so the PCH generation wouldn't get slower.

 And this could be easily later extended to e.g. non-template inline functions
that get out-of-line copies in debug mode:
- Are we using a marked PCH and -building-pch-with-obj is set? => generate
shared copies
- Are we using a marked PCH but -building-pch-with-obj is not set? => skip
generation
- The PCH is not marked as having an object? => work normally

 I don't know how this relates to modules, but for PCHs I expect this should
work fine.

> But I still suspect whatever that implements isn't quite modules-codegen,
> but I could be wrong.


 I still don't quite get how modules relate to my patch, so I can't really
comment on this.

--
 Lubos Lunak
_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev


--
Regards,
Ilya Biryukov


--
Regards,
Ilya Biryukov

_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: Controlling instantiation of templates from PCH

Lubos Lunak-2
In reply to this post by Gavin Cui via cfe-dev
On Tuesday 18 of June 2019, David Blaikie wrote:

> I'm getting a bit confused by the layers of this thread, so I'll see if I
> can sum up my understanding & see if that's any use.
>
> Currently there's a difference between PCH and Clang Header Modules
> semantics with regard to pending instantiations
> In PCH, pending instantiations are all performed at any use of the PCH.
> In modules pending instantiations are done at the end of building the
> header module (so they're performed once).
> This difference is the cause of the original compile time cost you were
> investigating (somewhat independent of the duplicate code in object files)


 Yes (I don't know about modules, but presumably those parts are correct too).

> So, step 1) perhaps we can just remove the PCH mode ^ here, and do the
> header module thing (perform pending instantiations at the end of building
> the PCH) for PCH as well.
> If you want to do this ^ work,


 Theoretically yes, but in practice no. Besides what Ilya Biryukov commented,
this would also prevent the case of instantiating those templates only in the
PCH's object file, and I consider that approach to be a better one.

> Separate from that, there's some desire to be able to build an object file
> from a PCH to remove duplicate template instantiations in object files.
> (reducing compile time, reducing object size, maybe reducing link time).
> To do this - it sounds like there's already a flag (-building-pch-with-obj)
> for building PCHs that will have a separate object file.

 Yes.

> I've no idea what
> this flag does today - does it already do what you want in terms of moving
> template instantiations and inline functions into that object? Or does it
> do something else (what else?)?


 The flag marks the PCH with a flag that the PCH comes with an object file and
that's it. Code in Clang then can check the flag and act based on that.
Currently it is used only for avoiding repeatedly emitting exported inline
functions (https://reviews.llvm.org/D48426).

> If it doesn't do what you want already, and
> you want to extend it to do what you want - yes, I'd recommend
> piggy-backing on the exsiting Clang Header Modules modular code generation
> features for this.


 I think taking Clang Header Modules into this is a needless complication. My
patch is basically a glorified "if(special_condition) skip_instantiating;".


> & yes, potentially the pending instantiations could be moved over from the
> PCH-or-header-module generation step, to the PCH-or-header-module->Object
> step. (to benefit both PCH and header modules)


 I have updated my patch, made it based on -building-pch-with-obj and resolved
some issues: https://reviews.llvm.org/D64284 . It is now capable of handling
an entire LibreOffice debug build. I still need help with some of the
problems described in the review submission summary.

>  (to benefit both PCH and header modules)


 Sorry, but I still have no idea how header modules are actually related to my
patch.

--
 Lubos Lunak
_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
12