Update on generating cc1 command line from CompilerInvocation

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Update on generating cc1 command line from CompilerInvocation

Dimitry Andric via cfe-dev
Hi, I work on explicitly building Clang modules via pre-scanning at Apple along with Michael Spencer and Duncan Exon Smith.

As part of our effort, Daniel Grumberg posted an RFC to Generate CC1 Command Lines from a CompilerInvocation instance <https://lists.llvm.org/pipermail/cfe-dev/2020-May/065421.html> some time ago, which got some positive feedback. The implementation is almost complete and we want to check if everyone is okay with our direction.

The plan

The RFC proposed the following steps:

1. Build an infrastructure for automatic command line option marshalling.
2. Use the infrastructure to automatically parse and generate simple options.
3. Manually implement code for generating non-trivial options.
4. Turn on round-tripping in assert builds to ensure correctness.

Steps 1, 2 are already finished and upstream, step 3 is almost done.

Round-tripping in step 4 means that CompilerInvocation::CreateFromArgs does not create the instance directly from the original command line. Instead, it parses the arguments into a dummy CompilerInvocation instance and uses it to generate new command line. The real CompilerInvocation is then parsed from the generated command line. When exercised by Clang’s test suite, this setup validates the generated command line has semantics identical to the original.

Currently, round-tripping can be enabled by building Clang with -DCLANG_ROUND_TRIP_CC1_ARGS=ON or by invoking cc1 with -round-trip-args. The generated command line can be printed by using the -Rround-trip-cc1-args option.

Next steps

After all patches are upstream and the whole CompilerInvocation is serializable, it would be good to enable round-tripping by default for assert builds. This would ensure bots building with assertions fail whenever new cc1 option is not being generated correctly. The process of adding new command line option and using the marshalling infrastructure is documented in the Clang Internals Manual <https://clang.llvm.org/docs/InternalsManual.html#adding-new-command-line-option>. I’d be watching patches related to command line options for a while and helping folks with implementing generation for new options.

Special considerations for downstream

Downstream projects that need to adopt this feature in their own timeframe could temporarily disable round-tripping by building with -DCLANG_ROUND_TRIP_CC1_ARGS=OFF or always passing -no-round-trip-args to cc1.

Below is a non-exhaustive list of related patches:

* <https://reviews.llvm.org/D94472> implements round-tripping,
* <https://reviews.llvm.org/D96056> manually generates complex CodeGenOptions,
* <https://reviews.llvm.org/D96280> changes the round-trip to handle whole CompilerInvocation at once instead of handling individual members separately.

Does the proposed plan of enabling round-trip in assert builds sound good?

Cheers,
Jan Svoboda
_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: RFC: Update on generating cc1 command line from CompilerInvocation

Dimitry Andric via cfe-dev
Ping.

The patch that enables command line round-trip for assert builds is here: https://reviews.llvm.org/D97462

-Jan

> On Feb 17, 2021, at 3:01 PM, Jan Svoboda <[hidden email]> wrote:
>
> Hi, I work on explicitly building Clang modules via pre-scanning at Apple along with Michael Spencer and Duncan Exon Smith.
>
> As part of our effort, Daniel Grumberg posted an RFC to Generate CC1 Command Lines from a CompilerInvocation instance <https://lists.llvm.org/pipermail/cfe-dev/2020-May/065421.html> some time ago, which got some positive feedback. The implementation is almost complete and we want to check if everyone is okay with our direction.
>
> The plan
>
> The RFC proposed the following steps:
>
> 1. Build an infrastructure for automatic command line option marshalling.
> 2. Use the infrastructure to automatically parse and generate simple options.
> 3. Manually implement code for generating non-trivial options.
> 4. Turn on round-tripping in assert builds to ensure correctness.
>
> Steps 1, 2 are already finished and upstream, step 3 is almost done.
>
> Round-tripping in step 4 means that CompilerInvocation::CreateFromArgs does not create the instance directly from the original command line. Instead, it parses the arguments into a dummy CompilerInvocation instance and uses it to generate new command line. The real CompilerInvocation is then parsed from the generated command line. When exercised by Clang’s test suite, this setup validates the generated command line has semantics identical to the original.
>
> Currently, round-tripping can be enabled by building Clang with -DCLANG_ROUND_TRIP_CC1_ARGS=ON or by invoking cc1 with -round-trip-args. The generated command line can be printed by using the -Rround-trip-cc1-args option.
>
> Next steps
>
> After all patches are upstream and the whole CompilerInvocation is serializable, it would be good to enable round-tripping by default for assert builds. This would ensure bots building with assertions fail whenever new cc1 option is not being generated correctly. The process of adding new command line option and using the marshalling infrastructure is documented in the Clang Internals Manual <https://clang.llvm.org/docs/InternalsManual.html#adding-new-command-line-option>. I’d be watching patches related to command line options for a while and helping folks with implementing generation for new options.
>
> Special considerations for downstream
>
> Downstream projects that need to adopt this feature in their own timeframe could temporarily disable round-tripping by building with -DCLANG_ROUND_TRIP_CC1_ARGS=OFF or always passing -no-round-trip-args to cc1.
>
> Below is a non-exhaustive list of related patches:
>
> * <https://reviews.llvm.org/D94472> implements round-tripping,
> * <https://reviews.llvm.org/D96056> manually generates complex CodeGenOptions,
> * <https://reviews.llvm.org/D96280> changes the round-trip to handle whole CompilerInvocation at once instead of handling individual members separately.
>
> Does the proposed plan of enabling round-trip in assert builds sound good?
>
> Cheers,
> Jan Svoboda

_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: RFC: Update on generating cc1 command line from CompilerInvocation

Dimitry Andric via cfe-dev
Ping.

> On Feb 26, 2021, at 6:25 PM, Jan Svoboda <[hidden email]> wrote:
>
> Ping.
>
> The patch that enables command line round-trip for assert builds is here: https://reviews.llvm.org/D97462
>
> -Jan
>
>> On Feb 17, 2021, at 3:01 PM, Jan Svoboda <[hidden email]> wrote:
>>
>> Hi, I work on explicitly building Clang modules via pre-scanning at Apple along with Michael Spencer and Duncan Exon Smith.
>>
>> As part of our effort, Daniel Grumberg posted an RFC to Generate CC1 Command Lines from a CompilerInvocation instance <https://lists.llvm.org/pipermail/cfe-dev/2020-May/065421.html> some time ago, which got some positive feedback. The implementation is almost complete and we want to check if everyone is okay with our direction.
>>
>> The plan
>>
>> The RFC proposed the following steps:
>>
>> 1. Build an infrastructure for automatic command line option marshalling.
>> 2. Use the infrastructure to automatically parse and generate simple options.
>> 3. Manually implement code for generating non-trivial options.
>> 4. Turn on round-tripping in assert builds to ensure correctness.
>>
>> Steps 1, 2 are already finished and upstream, step 3 is almost done.
>>
>> Round-tripping in step 4 means that CompilerInvocation::CreateFromArgs does not create the instance directly from the original command line. Instead, it parses the arguments into a dummy CompilerInvocation instance and uses it to generate new command line. The real CompilerInvocation is then parsed from the generated command line. When exercised by Clang’s test suite, this setup validates the generated command line has semantics identical to the original.
>>
>> Currently, round-tripping can be enabled by building Clang with -DCLANG_ROUND_TRIP_CC1_ARGS=ON or by invoking cc1 with -round-trip-args. The generated command line can be printed by using the -Rround-trip-cc1-args option.
>>
>> Next steps
>>
>> After all patches are upstream and the whole CompilerInvocation is serializable, it would be good to enable round-tripping by default for assert builds. This would ensure bots building with assertions fail whenever new cc1 option is not being generated correctly. The process of adding new command line option and using the marshalling infrastructure is documented in the Clang Internals Manual <https://clang.llvm.org/docs/InternalsManual.html#adding-new-command-line-option>. I’d be watching patches related to command line options for a while and helping folks with implementing generation for new options.
>>
>> Special considerations for downstream
>>
>> Downstream projects that need to adopt this feature in their own timeframe could temporarily disable round-tripping by building with -DCLANG_ROUND_TRIP_CC1_ARGS=OFF or always passing -no-round-trip-args to cc1.
>>
>> Below is a non-exhaustive list of related patches:
>>
>> * <https://reviews.llvm.org/D94472> implements round-tripping,
>> * <https://reviews.llvm.org/D96056> manually generates complex CodeGenOptions,
>> * <https://reviews.llvm.org/D96280> changes the round-trip to handle whole CompilerInvocation at once instead of handling individual members separately.
>>
>> Does the proposed plan of enabling round-trip in assert builds sound good?
>>
>> Cheers,
>> Jan Svoboda
>

_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: RFC: Update on generating cc1 command line from CompilerInvocation

Dimitry Andric via cfe-dev
Hi Jan,

Given that this RFC has been quiet for a few weeks and it doesn't diverge significantly from Daniel's RFC in the summer, I think it's safe to move forward.

Looking forward to having this in place!

Duncan

> On 2021 Mar  11, at 13:27, Jan Svoboda via cfe-dev <[hidden email]> wrote:
>
> Ping.
>
>> On Feb 26, 2021, at 6:25 PM, Jan Svoboda <[hidden email]> wrote:
>>
>> Ping.
>>
>> The patch that enables command line round-trip for assert builds is here: https://reviews.llvm.org/D97462
>>
>> -Jan
>>
>>> On Feb 17, 2021, at 3:01 PM, Jan Svoboda <[hidden email]> wrote:
>>>
>>> Hi, I work on explicitly building Clang modules via pre-scanning at Apple along with Michael Spencer and Duncan Exon Smith.
>>>
>>> As part of our effort, Daniel Grumberg posted an RFC to Generate CC1 Command Lines from a CompilerInvocation instance <https://lists.llvm.org/pipermail/cfe-dev/2020-May/065421.html> some time ago, which got some positive feedback. The implementation is almost complete and we want to check if everyone is okay with our direction.
>>>
>>> The plan
>>>
>>> The RFC proposed the following steps:
>>>
>>> 1. Build an infrastructure for automatic command line option marshalling.
>>> 2. Use the infrastructure to automatically parse and generate simple options.
>>> 3. Manually implement code for generating non-trivial options.
>>> 4. Turn on round-tripping in assert builds to ensure correctness.
>>>
>>> Steps 1, 2 are already finished and upstream, step 3 is almost done.
>>>
>>> Round-tripping in step 4 means that CompilerInvocation::CreateFromArgs does not create the instance directly from the original command line. Instead, it parses the arguments into a dummy CompilerInvocation instance and uses it to generate new command line. The real CompilerInvocation is then parsed from the generated command line. When exercised by Clang’s test suite, this setup validates the generated command line has semantics identical to the original.
>>>
>>> Currently, round-tripping can be enabled by building Clang with -DCLANG_ROUND_TRIP_CC1_ARGS=ON or by invoking cc1 with -round-trip-args. The generated command line can be printed by using the -Rround-trip-cc1-args option.
>>>
>>> Next steps
>>>
>>> After all patches are upstream and the whole CompilerInvocation is serializable, it would be good to enable round-tripping by default for assert builds. This would ensure bots building with assertions fail whenever new cc1 option is not being generated correctly. The process of adding new command line option and using the marshalling infrastructure is documented in the Clang Internals Manual <https://clang.llvm.org/docs/InternalsManual.html#adding-new-command-line-option>. I’d be watching patches related to command line options for a while and helping folks with implementing generation for new options.
>>>
>>> Special considerations for downstream
>>>
>>> Downstream projects that need to adopt this feature in their own timeframe could temporarily disable round-tripping by building with -DCLANG_ROUND_TRIP_CC1_ARGS=OFF or always passing -no-round-trip-args to cc1.
>>>
>>> Below is a non-exhaustive list of related patches:
>>>
>>> * <https://reviews.llvm.org/D94472> implements round-tripping,
>>> * <https://reviews.llvm.org/D96056> manually generates complex CodeGenOptions,
>>> * <https://reviews.llvm.org/D96280> changes the round-trip to handle whole CompilerInvocation at once instead of handling individual members separately.
>>>
>>> Does the proposed plan of enabling round-trip in assert builds sound good?
>>>
>>> Cheers,
>>> Jan Svoboda
>>
>
> _______________________________________________
> cfe-dev mailing list
> [hidden email]
> https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev

_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: RFC: Update on generating cc1 command line from CompilerInvocation

Dimitry Andric via cfe-dev
Hi Duncan,

That sounds reasonable to me. I'll commit the patch later this week.

-Jan

> On Mar 15, 2021, at 10:44 PM, Duncan P. N. Exon Smith <[hidden email]> wrote:
>
> Hi Jan,
>
> Given that this RFC has been quiet for a few weeks and it doesn't diverge significantly from Daniel's RFC in the summer, I think it's safe to move forward.
>
> Looking forward to having this in place!
>
> Duncan
>
>> On 2021 Mar  11, at 13:27, Jan Svoboda via cfe-dev <[hidden email]> wrote:
>>
>> Ping.
>>
>>> On Feb 26, 2021, at 6:25 PM, Jan Svoboda <[hidden email]> wrote:
>>>
>>> Ping.
>>>
>>> The patch that enables command line round-trip for assert builds is here: https://reviews.llvm.org/D97462
>>>
>>> -Jan
>>>
>>>> On Feb 17, 2021, at 3:01 PM, Jan Svoboda <[hidden email]> wrote:
>>>>
>>>> Hi, I work on explicitly building Clang modules via pre-scanning at Apple along with Michael Spencer and Duncan Exon Smith.
>>>>
>>>> As part of our effort, Daniel Grumberg posted an RFC to Generate CC1 Command Lines from a CompilerInvocation instance <https://lists.llvm.org/pipermail/cfe-dev/2020-May/065421.html> some time ago, which got some positive feedback. The implementation is almost complete and we want to check if everyone is okay with our direction.
>>>>
>>>> The plan
>>>>
>>>> The RFC proposed the following steps:
>>>>
>>>> 1. Build an infrastructure for automatic command line option marshalling.
>>>> 2. Use the infrastructure to automatically parse and generate simple options.
>>>> 3. Manually implement code for generating non-trivial options.
>>>> 4. Turn on round-tripping in assert builds to ensure correctness.
>>>>
>>>> Steps 1, 2 are already finished and upstream, step 3 is almost done.
>>>>
>>>> Round-tripping in step 4 means that CompilerInvocation::CreateFromArgs does not create the instance directly from the original command line. Instead, it parses the arguments into a dummy CompilerInvocation instance and uses it to generate new command line. The real CompilerInvocation is then parsed from the generated command line. When exercised by Clang’s test suite, this setup validates the generated command line has semantics identical to the original.
>>>>
>>>> Currently, round-tripping can be enabled by building Clang with -DCLANG_ROUND_TRIP_CC1_ARGS=ON or by invoking cc1 with -round-trip-args. The generated command line can be printed by using the -Rround-trip-cc1-args option.
>>>>
>>>> Next steps
>>>>
>>>> After all patches are upstream and the whole CompilerInvocation is serializable, it would be good to enable round-tripping by default for assert builds. This would ensure bots building with assertions fail whenever new cc1 option is not being generated correctly. The process of adding new command line option and using the marshalling infrastructure is documented in the Clang Internals Manual <https://clang.llvm.org/docs/InternalsManual.html#adding-new-command-line-option>. I’d be watching patches related to command line options for a while and helping folks with implementing generation for new options.
>>>>
>>>> Special considerations for downstream
>>>>
>>>> Downstream projects that need to adopt this feature in their own timeframe could temporarily disable round-tripping by building with -DCLANG_ROUND_TRIP_CC1_ARGS=OFF or always passing -no-round-trip-args to cc1.
>>>>
>>>> Below is a non-exhaustive list of related patches:
>>>>
>>>> * <https://reviews.llvm.org/D94472> implements round-tripping,
>>>> * <https://reviews.llvm.org/D96056> manually generates complex CodeGenOptions,
>>>> * <https://reviews.llvm.org/D96280> changes the round-trip to handle whole CompilerInvocation at once instead of handling individual members separately.
>>>>
>>>> Does the proposed plan of enabling round-trip in assert builds sound good?
>>>>
>>>> Cheers,
>>>> Jan Svoboda
>>>
>>
>> _______________________________________________
>> cfe-dev mailing list
>> [hidden email]
>> https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>

_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev