Reflection/metaparsing implem

classic Classic list List threaded Threaded
20 messages Options
Reply | Threaded
Open this post in threaded view
|

Reflection/metaparsing implem

Tom Stellard via cfe-dev
Please enjoy this clang 7.0.0 fork:

https://github.com/drec357/clang-meta

This reflection implementation reflects just about the entire clang AST, and lets you use clang syntax/names in *regular, non-clang* C++ code:

namespace ns { class MyClass { float f; }; }
constexpr {
ce_assert(reflexpr(ns)->isInline());
ce_assert(!reflexpr(ns)->isAnonymousNamespace())
FOR ( (Decl *D) : reflexpr(ns::MyClass)->decls()) {
if (auto_ FD = dyn_cast<FieldDecl>(D))
FD->dump();
}
//As for meta-parsing:
__queue_metaparse("static const int i = ");
constexpr int ival = __metaparse_expr(__concatenate("3", 2+2), int);
__queue_metaparse(jval);
__queue_metaparse(";")

} // queued metatparses performed here…

static_assert(i = 34);

Plus you can define clang-style diagnostics and FixItHints, using reflected SourceLocations, in your regular, non-clang code.

Please adapt to any uses you see fit!  I would love to see this way in the C++ standards, much more general and low-maintenance than the proposals I’ve seen, but I sense that will be an uphill battle.  Please let me know if you would support this way of doing reflection/metaprogramming, or have suggestions on how to proceed.

David Rector

_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: Reflection/metaparsing implem

Tom Stellard via cfe-dev


On Mon, Nov 11, 2019 at 8:43 AM David Rector via cfe-dev <[hidden email]> wrote:
Please enjoy this clang 7.0.0 fork:

https://github.com/drec357/clang-meta

This reflection implementation reflects just about the entire clang AST, and lets you use clang syntax/names in *regular, non-clang* C++ code:

namespace ns { class MyClass { float f; }; }
constexpr {
ce_assert(reflexpr(ns)->isInline());
ce_assert(!reflexpr(ns)->isAnonymousNamespace())
FOR ( (Decl *D) : reflexpr(ns::MyClass)->decls()) {
if (auto_ FD = dyn_cast<FieldDecl>(D))
FD->dump();
}
//As for meta-parsing:
__queue_metaparse("static const int i = ");
constexpr int ival = __metaparse_expr(__concatenate("3", 2+2), int);
__queue_metaparse(jval);
__queue_metaparse(";")

} // queued metatparses performed here…

static_assert(i = 34);

Plus you can define clang-style diagnostics and FixItHints, using reflected SourceLocations, in your regular, non-clang code.

Please adapt to any uses you see fit!  I would love to see this way in the C++ standards, much more general and low-maintenance than the proposals I’ve seen,

One quick caveat: If it's general and lower maintenance by exposing Clang's AST directly, that is probably a significant deterrent to this being standardized - Clang's AST changes and it's not the same as other compilers ASTs, so generalizing this across compilers and across time might be quite difficult.
 
but I sense that will be an uphill battle.  Please let me know if you would support this way of doing reflection/metaprogramming, or have suggestions on how to proceed.

David Rector
_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev

_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: Reflection/metaparsing implem

Tom Stellard via cfe-dev
I argue the standards should be based on the clang AST.  Clean it up first of course — and yes that alone would be extraordinarily difficult.  But it’s also very difficult to build a gigantic parallel naming system for reflection properties.  And then to maintain it long term…

Best to look hard at what lies ahead, and bite the bullet now.  Clean up the AST, define the standard via it.  (And c’mon we all know those AST nodes have a lot of flab to go with all their muscle; clang would benefit too after we all finish grumbling about the changes.)  Other compilers can do a parallel naming system or adjust their own ASTs to match clang’s naming system — doesn’t need to add any work for them.

Dave

On Nov 11, 2019, at 12:49 PM, David Blaikie <[hidden email]> wrote:



On Mon, Nov 11, 2019 at 8:43 AM David Rector via cfe-dev <[hidden email]> wrote:
Please enjoy this clang 7.0.0 fork:

https://github.com/drec357/clang-meta

This reflection implementation reflects just about the entire clang AST, and lets you use clang syntax/names in *regular, non-clang* C++ code:

namespace ns { class MyClass { float f; }; }
constexpr {
ce_assert(reflexpr(ns)->isInline());
ce_assert(!reflexpr(ns)->isAnonymousNamespace())
FOR ( (Decl *D) : reflexpr(ns::MyClass)->decls()) {
if (auto_ FD = dyn_cast<FieldDecl>(D))
FD->dump();
}
//As for meta-parsing:
__queue_metaparse("static const int i = ");
constexpr int ival = __metaparse_expr(__concatenate("3", 2+2), int);
__queue_metaparse(jval);
__queue_metaparse(";")

} // queued metatparses performed here…

static_assert(i = 34);

Plus you can define clang-style diagnostics and FixItHints, using reflected SourceLocations, in your regular, non-clang code.

Please adapt to any uses you see fit!  I would love to see this way in the C++ standards, much more general and low-maintenance than the proposals I’ve seen,

One quick caveat: If it's general and lower maintenance by exposing Clang's AST directly, that is probably a significant deterrent to this being standardized - Clang's AST changes and it's not the same as other compilers ASTs, so generalizing this across compilers and across time might be quite difficult.
 
but I sense that will be an uphill battle.  Please let me know if you would support this way of doing reflection/metaprogramming, or have suggestions on how to proceed.

David Rector
_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev


_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: Reflection/metaparsing implem

Tom Stellard via cfe-dev
In reply to this post by Tom Stellard via cfe-dev
On Mon, Nov 11, 2019 at 7:50 AM David Blaikie via cfe-dev <[hidden email]> wrote:
On Mon, Nov 11, 2019 at 8:43 AM David Rector via cfe-dev <[hidden email]> wrote:
Please enjoy this clang 7.0.0 fork:

https://github.com/drec357/clang-meta

This reflection implementation reflects just about the entire clang AST, and lets you use clang syntax/names in *regular, non-clang* C++ code:

namespace ns { class MyClass { float f; }; }
constexpr {
ce_assert(reflexpr(ns)->isInline());
ce_assert(!reflexpr(ns)->isAnonymousNamespace())
FOR ( (Decl *D) : reflexpr(ns::MyClass)->decls()) {
if (auto_ FD = dyn_cast<FieldDecl>(D))
FD->dump();
}
//As for meta-parsing:
__queue_metaparse("static const int i = ");
constexpr int ival = __metaparse_expr(__concatenate("3", 2+2), int);
__queue_metaparse(jval);
__queue_metaparse(";")

} // queued metatparses performed here…

static_assert(i = 34);

Plus you can define clang-style diagnostics and FixItHints, using reflected SourceLocations, in your regular, non-clang code.

Please adapt to any uses you see fit!  I would love to see this way in the C++ standards, much more general and low-maintenance than the proposals I’ve seen,

One quick caveat: If it's general and lower maintenance by exposing Clang's AST directly, that is probably a significant deterrent to this being standardized - Clang's AST changes and it's not the same as other compilers ASTs, so generalizing this across compilers and across time might be quite difficult.

Very true. However, this looks like a very promising Clang-specific feature that might make life easier for Clang users who want to use reflection features. If people started using this feature, and then GCC and MSVC got on board with their own GCC-specific and MSVC-specific reflection facilities, then we might end up in a situation where we had both
- implementation experience and
- motivation
to consolidate and standardize the implementations into a standard reflection facility.

The current WG21 standardization process puts the cart before the horse — they have a Reflection subgroup attempting to design a standard reflection facility prior to any vendor implementation experience and without any body of existing practice (motivation) to guide their design.  It might be worth seriously considering "practical" works like this one, at least far enough to see what's wrong with them besides "lack of ISO involvement."

my $.02,
–Arthur




_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: Reflection/metaparsing implem

Tom Stellard via cfe-dev


On Mon, Nov 11, 2019 at 10:29 AM Arthur O'Dwyer via cfe-dev <[hidden email]> wrote:
On Mon, Nov 11, 2019 at 7:50 AM David Blaikie via cfe-dev <[hidden email]> wrote:
On Mon, Nov 11, 2019 at 8:43 AM David Rector via cfe-dev <[hidden email]> wrote:
Please enjoy this clang 7.0.0 fork:

https://github.com/drec357/clang-meta

This reflection implementation reflects just about the entire clang AST, and lets you use clang syntax/names in *regular, non-clang* C++ code:

namespace ns { class MyClass { float f; }; }
constexpr {
ce_assert(reflexpr(ns)->isInline());
ce_assert(!reflexpr(ns)->isAnonymousNamespace())
FOR ( (Decl *D) : reflexpr(ns::MyClass)->decls()) {
if (auto_ FD = dyn_cast<FieldDecl>(D))
FD->dump();
}
//As for meta-parsing:
__queue_metaparse("static const int i = ");
constexpr int ival = __metaparse_expr(__concatenate("3", 2+2), int);
__queue_metaparse(jval);
__queue_metaparse(";")

} // queued metatparses performed here…

static_assert(i = 34);

Plus you can define clang-style diagnostics and FixItHints, using reflected SourceLocations, in your regular, non-clang code.

Please adapt to any uses you see fit!  I would love to see this way in the C++ standards, much more general and low-maintenance than the proposals I’ve seen,

One quick caveat: If it's general and lower maintenance by exposing Clang's AST directly, that is probably a significant deterrent to this being standardized - Clang's AST changes and it's not the same as other compilers ASTs, so generalizing this across compilers and across time might be quite difficult.

Very true. However, this looks like a very promising Clang-specific feature that might make life easier for Clang users who want to use reflection features.

I think that'd still be a pretty hard sell - Clang's APIs and AST are volatile & intentionally so. People writing against the Clang APIs themselves with things like clang-tidy are aware of that. If we sold this as a feature to general C++ users, I don't think there would be a way to sufficiently communicate how volatile this will be - and end up backing ourselves into a corner & limiting the ability to change Clang.
 
If people started using this feature, and then GCC and MSVC got on board with their own GCC-specific and MSVC-specific reflection facilities, then we might end up in a situation where we had both
- implementation experience and
- motivation
to consolidate and standardize the implementations into a standard reflection facility.

The current WG21 standardization process puts the cart before the horse — they have a Reflection subgroup attempting to design a standard reflection facility prior to any vendor implementation experience and without any body of existing practice (motivation) to guide their design.  It might be worth seriously considering "practical" works like this one, at least far enough to see what's wrong with them besides "lack of ISO involvement."

my $.02,
–Arthur



_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev

_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: Reflection/metaparsing implem

Tom Stellard via cfe-dev
On Mon, Nov 11, 2019 at 3:03 PM David Blaikie via cfe-dev <[hidden email]> wrote: 
I think that'd still be a pretty hard sell - Clang's APIs and AST are volatile & intentionally so. People writing against the Clang APIs themselves with things like clang-tidy are aware of that. If we sold this as a feature to general C++ users, I don't think there would be a way to sufficiently communicate how volatile this will be - and end up backing ourselves into a corner & limiting the ability to change Clang.

Does Clang's AST API _need_ to be unstable in order to keep up with language changes, or is it just a convenience? Is this concern indicative of problems that a standardized AST API would also have, or is unique to Clang's current API?

_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: Reflection/metaparsing implem

Tom Stellard via cfe-dev


On Mon, Nov 11, 2019 at 1:01 PM James Y Knight <[hidden email]> wrote:
On Mon, Nov 11, 2019 at 3:03 PM David Blaikie via cfe-dev <[hidden email]> wrote: 
I think that'd still be a pretty hard sell - Clang's APIs and AST are volatile & intentionally so. People writing against the Clang APIs themselves with things like clang-tidy are aware of that. If we sold this as a feature to general C++ users, I don't think there would be a way to sufficiently communicate how volatile this will be - and end up backing ourselves into a corner & limiting the ability to change Clang.

Does Clang's AST API _need_ to be unstable in order to keep up with language changes, or is it just a convenience?

I imagine with sufficient heroics the AST could be maintained for a particular version - and /probably/ version-over-version (so the C++23 AST was backwards compatible with the C++20 AST, etc). In the same way that the language itself is mostly backwards compatible version over version.
 
Is this concern indicative of problems that a standardized AST API would also have, or is unique to Clang's current API?

The concern is mostly indicative of a very high fidelity API - the lower level/more bare the API, the more brittle it'll be to changes (both refactoring improvements to Clang, and necessary changes to accommodate new language features). Standardizing something a little higher level could provide a lot more flexibility under the hood.
 

_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: Reflection/metaparsing implem

Tom Stellard via cfe-dev
In reply to this post by Tom Stellard via cfe-dev
> The current WG21 standardization process puts the cart before the horse — they have a Reflection subgroup attempting to design a standard reflection facility prior to any vendor implementation experience and without any body of existing practice (motivation) to guide their design.

This is wildly inaccurate.

SG7 has been discussing static reflection systems for at least 5
years. There is well-established motivation on top of decades of
practical experience including template metaprogramming and languages
with dynamic reflection.

As far as implementation experience goes, quoting P1240R1:

"Two implementations of this proposal are underway. The first and most
mature/complete is a fork of Clang by Lock3 Software (by, among
others, Andrew and Wyatt, authors of this paper). It includes a large
portion of the capabilities presented here, albeit not always with the
exact syntax or interfaces proposed. The second is based on the EDG
front end (by Faisal and Daveed) and is in very early stages, which
just a few cases of the reflexpr operator, a few reifiers, and a few
meta-library interfaces implemented"

There was a prototype of the template-based approach by Matus. That
probably goes back 4 years. The first work on the Clang version
(https://gitlab.com/lock3/clang) started more than 3 years ago. I
know... I wrote it. I also gave talks on this work at CppCon in 2017
and 2019. Herb Sutter has given a number of talks about this since at
least 2017 (usually related to metaclasses). We've had active
experimenters for quite a while.

None of this work has been secret. It's well documented. This
information is easy to find, the people involved even more so. Any
claims that SG7 isn't working hard on this feature is either ignorant
or an aspersion.

Andrew
_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: Reflection/metaparsing implem

Tom Stellard via cfe-dev
In reply to this post by Tom Stellard via cfe-dev


On Nov 11, 2019, at 4:09 PM, David Blaikie <[hidden email]> wrote:



On Mon, Nov 11, 2019 at 1:01 PM James Y Knight <[hidden email]> wrote:
On Mon, Nov 11, 2019 at 3:03 PM David Blaikie via cfe-dev <[hidden email]> wrote: 
I think that'd still be a pretty hard sell - Clang's APIs and AST are volatile & intentionally so. People writing against the Clang APIs themselves with things like clang-tidy are aware of that. If we sold this as a feature to general C++ users, I don't think there would be a way to sufficiently communicate how volatile this will be - and end up backing ourselves into a corner & limiting the ability to change Clang.

Does Clang's AST API _need_ to be unstable in order to keep up with language changes, or is it just a convenience?

I imagine with sufficient heroics the AST could be maintained for a particular version - and /probably/ version-over-version (so the C++23 AST was backwards compatible with the C++20 AST, etc). In the same way that the language itself is mostly backwards compatible version over version.
 
Is this concern indicative of problems that a standardized AST API would also have, or is unique to Clang's current API?

The concern is mostly indicative of a very high fidelity API - the lower level/more bare the API, the more brittle it'll be to changes (both refactoring improvements to Clang, and necessary changes to accommodate new language features). Standardizing something a little higher level could provide a lot more flexibility under the hood.
 

Volatility issues could be managed. E.g. I’ve already implem’d so that CLANG_NO_REFLECT turns off reflection for any AST public const method or field — but note that it is a crutch; any public const method is either a) eventually going to be asked to be reflected or b) probably shouldn’t be a public const method.  (Reflection is an opportunity to look in the mirror — better to hit the gym than to design a funhouse mirror to fix the issue.)

I imagine Other AST attributes — e.g. CLANG_REFLECT_DEPRECATED — could be defined so that the tool generates warnings and fixits whenever they are used.

_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: Reflection/metaparsing implem

Tom Stellard via cfe-dev


On Mon, Nov 11, 2019 at 2:08 PM David Rector <[hidden email]> wrote:


On Nov 11, 2019, at 4:09 PM, David Blaikie <[hidden email]> wrote:



On Mon, Nov 11, 2019 at 1:01 PM James Y Knight <[hidden email]> wrote:
On Mon, Nov 11, 2019 at 3:03 PM David Blaikie via cfe-dev <[hidden email]> wrote: 
I think that'd still be a pretty hard sell - Clang's APIs and AST are volatile & intentionally so. People writing against the Clang APIs themselves with things like clang-tidy are aware of that. If we sold this as a feature to general C++ users, I don't think there would be a way to sufficiently communicate how volatile this will be - and end up backing ourselves into a corner & limiting the ability to change Clang.

Does Clang's AST API _need_ to be unstable in order to keep up with language changes, or is it just a convenience?

I imagine with sufficient heroics the AST could be maintained for a particular version - and /probably/ version-over-version (so the C++23 AST was backwards compatible with the C++20 AST, etc). In the same way that the language itself is mostly backwards compatible version over version.
 
Is this concern indicative of problems that a standardized AST API would also have, or is unique to Clang's current API?

The concern is mostly indicative of a very high fidelity API - the lower level/more bare the API, the more brittle it'll be to changes (both refactoring improvements to Clang, and necessary changes to accommodate new language features). Standardizing something a little higher level could provide a lot more flexibility under the hood.
 

Volatility issues could be managed. E.g. I’ve already implem’d so that CLANG_NO_REFLECT turns off reflection for any AST public const method or field — but note that it is a crutch; any public const method is either a) eventually going to be asked to be reflected or b) probably shouldn’t be a public const method.

Having something be "public" within Clang's codebase and users of Clang's APIs that are explicitly documented as volatile is quite different from it being public across significant amounts of end-user C++ code.

It makes a fair bit of sense for a lot of things to be public within the former scope that aren't suitable to be public within the latter scope.

The goal is that people may ask for (a) but we may reasonably push back and say that that level of fidelity maybe isn't worth the maintenance burden to all C++ compilers indefinitely into the future.
 
 (Reflection is an opportunity to look in the mirror — better to hit the gym than to design a funhouse mirror to fix the issue.)

I imagine Other AST attributes — e.g. CLANG_REFLECT_DEPRECATED — could be defined so that the tool generates warnings and fixits whenever they are used.

Historically the C++ standard hasn't been terribly open to significant deprecation/feature removal - it's probably not practical to plan on a reflection API that would need that to maintain... maintainability. 

_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: Reflection/metaparsing implem

Tom Stellard via cfe-dev
Well then for any such "volatile" stuff you’d either CLANG_NO_REFLECT it or define a CLANG_RELFECT_VOLATILE so that you’d get always get a warning if it was used.

Surely a full parallel reflection property processing system to do such filtering is going to require more maintenance than anything we’re talking about here.

On Nov 11, 2019, at 5:25 PM, David Blaikie <[hidden email]> wrote:



On Mon, Nov 11, 2019 at 2:08 PM David Rector <[hidden email]> wrote:


On Nov 11, 2019, at 4:09 PM, David Blaikie <[hidden email]> wrote:



On Mon, Nov 11, 2019 at 1:01 PM James Y Knight <[hidden email]> wrote:
On Mon, Nov 11, 2019 at 3:03 PM David Blaikie via cfe-dev <[hidden email]> wrote: 
I think that'd still be a pretty hard sell - Clang's APIs and AST are volatile & intentionally so. People writing against the Clang APIs themselves with things like clang-tidy are aware of that. If we sold this as a feature to general C++ users, I don't think there would be a way to sufficiently communicate how volatile this will be - and end up backing ourselves into a corner & limiting the ability to change Clang.

Does Clang's AST API _need_ to be unstable in order to keep up with language changes, or is it just a convenience?

I imagine with sufficient heroics the AST could be maintained for a particular version - and /probably/ version-over-version (so the C++23 AST was backwards compatible with the C++20 AST, etc). In the same way that the language itself is mostly backwards compatible version over version.
 
Is this concern indicative of problems that a standardized AST API would also have, or is unique to Clang's current API?

The concern is mostly indicative of a very high fidelity API - the lower level/more bare the API, the more brittle it'll be to changes (both refactoring improvements to Clang, and necessary changes to accommodate new language features). Standardizing something a little higher level could provide a lot more flexibility under the hood.
 

Volatility issues could be managed. E.g. I’ve already implem’d so that CLANG_NO_REFLECT turns off reflection for any AST public const method or field — but note that it is a crutch; any public const method is either a) eventually going to be asked to be reflected or b) probably shouldn’t be a public const method.

Having something be "public" within Clang's codebase and users of Clang's APIs that are explicitly documented as volatile is quite different from it being public across significant amounts of end-user C++ code.

It makes a fair bit of sense for a lot of things to be public within the former scope that aren't suitable to be public within the latter scope.

The goal is that people may ask for (a) but we may reasonably push back and say that that level of fidelity maybe isn't worth the maintenance burden to all C++ compilers indefinitely into the future.
 
 (Reflection is an opportunity to look in the mirror — better to hit the gym than to design a funhouse mirror to fix the issue.)

I imagine Other AST attributes — e.g. CLANG_REFLECT_DEPRECATED — could be defined so that the tool generates warnings and fixits whenever they are used.

Historically the C++ standard hasn't been terribly open to significant deprecation/feature removal - it's probably not practical to plan on a reflection API that would need that to maintain... maintainability. 


_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: Reflection/metaparsing implem

Tom Stellard via cfe-dev
On 11/11/2019 23:17, David Rector via cfe-dev wrote:
> Well then for any such "volatile" stuff you’d either CLANG_NO_REFLECT it
> or define a CLANG_RELFECT_VOLATILE so that you’d get always get a
> warning if it was used.

If you know in advance that something is implemented in a way that is
not the right long-term solution, you probably wouldn't implement it
that way in the first place.  A lot of the volatility in the Clang ASTs
comes from experiences trying to use the APIs in different ways and
learning which design decisions work and which ones don't.  If you want
to freeze these APIs in perpetuity, then you are likely to also freeze
the set of things that they're useful for.

David
_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: Reflection/metaparsing implem

Tom Stellard via cfe-dev
In reply to this post by Tom Stellard via cfe-dev
On 11/11/2019 18:04, David Rector via cfe-dev wrote:
> Best to look hard at what lies ahead, and bite the bullet now.  Clean up
> the AST, define the standard via it.  (And c’mon we all know those AST
> nodes have a lot of flab to go with all their muscle; clang would
> benefit too after we all finish grumbling about the changes.)  Other
> compilers can do a parallel naming system or adjust their own ASTs to
> match clang’s naming system — doesn’t need to add any work for them.

This is the second time in this thread that you have made insulting
comments about other people's work.  Please read the LLVM Code of
Conduct, particularly the section 'Be respectful':

https://llvm.org/docs/CodeOfConduct.html

This is a public discussion and insults do not help convincing anyone
that you are right.

David
_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: Reflection/metaparsing implem

Tom Stellard via cfe-dev
In reply to this post by Tom Stellard via cfe-dev

I argue the standards should be based on the clang AST.  Clean it up first of course — and yes that alone would be extraordinarily difficult.  But it’s also very difficult to build a gigantic parallel naming system for reflection properties.  And then to maintain it long term…

Best to look hard at what lies ahead, and bite the bullet now.  Clean up the AST, define the standard via it.  (And c’mon we all know those AST nodes have a lot of flab to go with all their muscle; clang would benefit too after we all finish grumbling about the changes.)  Other compilers can do a parallel naming system or adjust their own ASTs to match clang’s naming system — doesn’t need to add any work for them.

This is a non-starter for standardization. There are significant and generally incompatible differences in the internal representations of C++ among the different vendors and no will to standardize or even informally adopt a common representation. Even if there were interest in doing so, the engineering cost would be prohibitive, as it would essentially require gutting and rewriting huge swaths of the compiler (and in GCC's case, possibly the middle-end). Parallel naming is also unlikely because of memory requirements.

You have the same problem with compiled module interfaces, which serialize the internal representation.

Andrew

_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: Reflection/metaparsing implem

Tom Stellard via cfe-dev
Fair enough.  

As I say in my readme, I recommend those who need full reflection facilities — up to their class templates, down to the individual IfStmts etc. in their function defs — to use my implementation and simply isolate dependencies on future standards into their own helper functions.  You can do everything now!  No need to complain to or pressure Andrew (who has done some extraordinary work by the way) about what isn’t yet reflected.

*Eventually* the standard will support reflecting everything, years hence, whatever anyone says now.  The demands will just never stop until everything is reflected.  Start writing your algorithms now, switch your helper functions to whatever standards they decide later.

Regarding parallel naming I think we’re talking about different things — what we’re doing now IS a parallel naming structure.  meta::is_constexpr(reflinfo) is a parallel of FunctionDecl::isConstexpr() etc.  A full parallel structure of every property is what we’re looking at building.  If everyone’s looked hard at that eventuality and are okay with maintaining that, then I suppose I have no cause to complain — maybe it’s not as hard as I imagine.



On Nov 12, 2019, at 8:01 AM, Andrew Sutton <[hidden email]> wrote:


I argue the standards should be based on the clang AST.  Clean it up first of course — and yes that alone would be extraordinarily difficult.  But it’s also very difficult to build a gigantic parallel naming system for reflection properties.  And then to maintain it long term…

Best to look hard at what lies ahead, and bite the bullet now.  Clean up the AST, define the standard via it.  (And c’mon we all know those AST nodes have a lot of flab to go with all their muscle; clang would benefit too after we all finish grumbling about the changes.)  Other compilers can do a parallel naming system or adjust their own ASTs to match clang’s naming system — doesn’t need to add any work for them.

This is a non-starter for standardization. There are significant and generally incompatible differences in the internal representations of C++ among the different vendors and no will to standardize or even informally adopt a common representation. Even if there were interest in doing so, the engineering cost would be prohibitive, as it would essentially require gutting and rewriting huge swaths of the compiler (and in GCC's case, possibly the middle-end). Parallel naming is also unlikely because of memory requirements.

You have the same problem with compiled module interfaces, which serialize the internal representation.

Andrew


_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: Reflection/metaparsing implem

Tom Stellard via cfe-dev
Also: if we want to make this a clang-specific feature, all we need to do is define a different version of reflexpr that doesn’t conflict with Andrew’s — e.g. __creflexpr.  

Then anyone can either use either

meta::is_constexpr(reflexpr(myfunc))

OR 

__creflexpr(myfunc)->isConstexpr()
__creflexpr(myfunc)->getBody()->… //non-ISO reflections supported here

I will see about implementing that change so everything can live in harmony.

Dave 

On Nov 12, 2019, at 9:04 AM, David Rector <[hidden email]> wrote:

Fair enough.  

As I say in my readme, I recommend those who need full reflection facilities — up to their class templates, down to the individual IfStmts etc. in their function defs — to use my implementation and simply isolate dependencies on future standards into their own helper functions.  You can do everything now!  No need to complain to or pressure Andrew (who has done some extraordinary work by the way) about what isn’t yet reflected.

*Eventually* the standard will support reflecting everything, years hence, whatever anyone says now.  The demands will just never stop until everything is reflected.  Start writing your algorithms now, switch your helper functions to whatever standards they decide later.

Regarding parallel naming I think we’re talking about different things — what we’re doing now IS a parallel naming structure.  meta::is_constexpr(reflinfo) is a parallel of FunctionDecl::isConstexpr() etc.  A full parallel structure of every property is what we’re looking at building.  If everyone’s looked hard at that eventuality and are okay with maintaining that, then I suppose I have no cause to complain — maybe it’s not as hard as I imagine.



On Nov 12, 2019, at 8:01 AM, Andrew Sutton <[hidden email]> wrote:


I argue the standards should be based on the clang AST.  Clean it up first of course — and yes that alone would be extraordinarily difficult.  But it’s also very difficult to build a gigantic parallel naming system for reflection properties.  And then to maintain it long term…

Best to look hard at what lies ahead, and bite the bullet now.  Clean up the AST, define the standard via it.  (And c’mon we all know those AST nodes have a lot of flab to go with all their muscle; clang would benefit too after we all finish grumbling about the changes.)  Other compilers can do a parallel naming system or adjust their own ASTs to match clang’s naming system — doesn’t need to add any work for them.

This is a non-starter for standardization. There are significant and generally incompatible differences in the internal representations of C++ among the different vendors and no will to standardize or even informally adopt a common representation. Even if there were interest in doing so, the engineering cost would be prohibitive, as it would essentially require gutting and rewriting huge swaths of the compiler (and in GCC's case, possibly the middle-end). Parallel naming is also unlikely because of memory requirements.

You have the same problem with compiled module interfaces, which serialize the internal representation.

Andrew



_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: Reflection/metaparsing implem

Tom Stellard via cfe-dev
In reply to this post by Tom Stellard via cfe-dev


On Tue, Nov 12, 2019 at 6:04 AM David Rector <[hidden email]> wrote:
Fair enough.  

As I say in my readme, I recommend those who need full reflection facilities — up to their class templates, down to the individual IfStmts etc. in their function defs — to use my implementation and simply isolate dependencies on future standards into their own helper functions.  You can do everything now!  No need to complain to or pressure Andrew (who has done some extraordinary work by the way) about what isn’t yet reflected.

*Eventually* the standard will support reflecting everything, years hence, whatever anyone says now.  The demands will just never stop until everything is reflected.

Not all demands must be satisfied. Other languages (I'm looking at Java) have reflection without expression reflection, for example, so far as I know.
 
 Start writing your algorithms now, switch your helper functions to whatever standards they decide later.

Regarding parallel naming I think we’re talking about different things — what we’re doing now IS a parallel naming structure.  meta::is_constexpr(reflinfo) is a parallel of FunctionDecl::isConstexpr() etc.  A full parallel structure of every property is what we’re looking at building.  If everyone’s looked hard at that eventuality and are okay with maintaining that, then I suppose I have no cause to complain — maybe it’s not as hard as I imagine.



On Nov 12, 2019, at 8:01 AM, Andrew Sutton <[hidden email]> wrote:


I argue the standards should be based on the clang AST.  Clean it up first of course — and yes that alone would be extraordinarily difficult.  But it’s also very difficult to build a gigantic parallel naming system for reflection properties.  And then to maintain it long term…

Best to look hard at what lies ahead, and bite the bullet now.  Clean up the AST, define the standard via it.  (And c’mon we all know those AST nodes have a lot of flab to go with all their muscle; clang would benefit too after we all finish grumbling about the changes.)  Other compilers can do a parallel naming system or adjust their own ASTs to match clang’s naming system — doesn’t need to add any work for them.

This is a non-starter for standardization. There are significant and generally incompatible differences in the internal representations of C++ among the different vendors and no will to standardize or even informally adopt a common representation. Even if there were interest in doing so, the engineering cost would be prohibitive, as it would essentially require gutting and rewriting huge swaths of the compiler (and in GCC's case, possibly the middle-end). Parallel naming is also unlikely because of memory requirements.

You have the same problem with compiled module interfaces, which serialize the internal representation.

Andrew


_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: Reflection/metaparsing implem

Tom Stellard via cfe-dev

I don’t want to weigh in for/against any particular Clang proposal, but FWIW regarding this:

 

> Not all demands must be satisfied.

 

True, but in this case…

 

> Other languages (I'm looking at Java) have reflection without expression reflection

 

… IMO we already know that we will need statement and expression reflection, to enable uses like autodiff. For example, in the ML field I’ve encountered C++ developers who would prefer to use C++ because of its AOT compilation performance, but who of have switched to other languages such as Julia primarily because of compile-time autodiff (and more generally for the broader class of AI/ML transformations that includes autodiff) where today in C++ doing it by hand-coding the derivative function greatly slows down the experimentation phase of an ML project… which otherwise would actually be faster with AOT C++.

 

At least one of the proof-of-concept C++ reflection implementations (Circle) already support statement/expression reflection and autodiff.

 

For statements and expressions, my impression is that reflection is pretty straightforward to specify (here’s another meta::info API) and the primary thing to watch for is that the current proposals for injection (generated code, generative programming) will extend well to statements and expressions, again using autodiff as a litmus test example of what we want to be able to express.

 

Herb

 

 

From: cfe-dev <[hidden email]> On Behalf Of David Blaikie via cfe-dev
Sent: Tuesday, November 12, 2019 11:42 AM
To: David Rector <[hidden email]>
Cc: Clang Dev <[hidden email]>
Subject: Re: [cfe-dev] Reflection/metaparsing implem

 

 

 

On Tue, Nov 12, 2019 at 6:04 AM David Rector <[hidden email]> wrote:

Fair enough.  

 

As I say in my readme, I recommend those who need full reflection facilities — up to their class templates, down to the individual IfStmts etc. in their function defs — to use my implementation and simply isolate dependencies on future standards into their own helper functions.  You can do everything now!  No need to complain to or pressure Andrew (who has done some extraordinary work by the way) about what isn’t yet reflected.

 

*Eventually* the standard will support reflecting everything, years hence, whatever anyone says now.  The demands will just never stop until everything is reflected.

 

Not all demands must be satisfied. Other languages (I'm looking at Java) have reflection without expression reflection, for example, so far as I know.

 

 Start writing your algorithms now, switch your helper functions to whatever standards they decide later.

 

Regarding parallel naming I think we’re talking about different things — what we’re doing now IS a parallel naming structure.  meta::is_constexpr(reflinfo) is a parallel of FunctionDecl::isConstexpr() etc.  A full parallel structure of every property is what we’re looking at building.  If everyone’s looked hard at that eventuality and are okay with maintaining that, then I suppose I have no cause to complain — maybe it’s not as hard as I imagine.

 

 



On Nov 12, 2019, at 8:01 AM, Andrew Sutton <[hidden email]> wrote:

 

 

I argue the standards should be based on the clang AST.  Clean it up first of course — and yes that alone would be extraordinarily difficult.  But it’s also very difficult to build a gigantic parallel naming system for reflection properties.  And then to maintain it long term…

 

Best to look hard at what lies ahead, and bite the bullet now.  Clean up the AST, define the standard via it.  (And c’mon we all know those AST nodes have a lot of flab to go with all their muscle; clang would benefit too after we all finish grumbling about the changes.)  Other compilers can do a parallel naming system or adjust their own ASTs to match clang’s naming system — doesn’t need to add any work for them.

 

This is a non-starter for standardization. There are significant and generally incompatible differences in the internal representations of C++ among the different vendors and no will to standardize or even informally adopt a common representation. Even if there were interest in doing so, the engineering cost would be prohibitive, as it would essentially require gutting and rewriting huge swaths of the compiler (and in GCC's case, possibly the middle-end). Parallel naming is also unlikely because of memory requirements.

 

You have the same problem with compiled module interfaces, which serialize the internal representation.

 

Andrew

 


_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: Reflection/metaparsing implem

Tom Stellard via cfe-dev


On Wed, Nov 13, 2019 at 12:42 PM Herb Sutter <[hidden email]> wrote:

I don’t want to weigh in for/against any particular Clang proposal, but FWIW regarding this:

 

> Not all demands must be satisfied.

 

True, but in this case…

 

> Other languages (I'm looking at Java) have reflection without expression reflection

 

… IMO we already know that we will need statement and expression reflection, to enable uses like autodiff. For example, in the ML field I’ve encountered C++ developers who would prefer to use C++ because of its AOT compilation performance, but who of have switched to other languages such as Julia primarily because of compile-time autodiff (and more generally for the broader class of AI/ML transformations that includes autodiff) where today in C++ doing it by hand-coding the derivative function greatly slows down the experimentation phase of an ML project… which otherwise would actually be faster with AOT C++.

 

At least one of the proof-of-concept C++ reflection implementations (Circle) already support statement/expression reflection and autodiff.

 

For statements and expressions, my impression is that reflection is pretty straightforward to specify (here’s another meta::info API) and the primary thing to watch for is that the current proposals for injection (generated code, generative programming) will extend well to statements and expressions, again using autodiff as a litmus test example of what we want to be able to express.


Ah, fair enough - thanks for the context! Yeah, I've not looked in any great detail at what's being proposed for standardization, or what its goals are - but intended to make the point that there are useful points in the reflection surface area that aren't "full AST fidelity". I've no doubt Richard Smith & other Clang folks who are (more than I) involved with the committee do a far better job than I at advocating for the nuances of implementation complexity/flexibily/future-proofing & working to ensure not too much of the implementation details get baked into a language feature while balancing the needs of users.

(compile time symbolic differentiation does sound like fun, to be sure - didn't know that was on the table/part of the motivation here)

- Dave
 

 

Herb

 

 

From: cfe-dev <[hidden email]> On Behalf Of David Blaikie via cfe-dev
Sent: Tuesday, November 12, 2019 11:42 AM
To: David Rector <[hidden email]>
Cc: Clang Dev <[hidden email]>
Subject: Re: [cfe-dev] Reflection/metaparsing implem

 

 

 

On Tue, Nov 12, 2019 at 6:04 AM David Rector <[hidden email]> wrote:

Fair enough.  

 

As I say in my readme, I recommend those who need full reflection facilities — up to their class templates, down to the individual IfStmts etc. in their function defs — to use my implementation and simply isolate dependencies on future standards into their own helper functions.  You can do everything now!  No need to complain to or pressure Andrew (who has done some extraordinary work by the way) about what isn’t yet reflected.

 

*Eventually* the standard will support reflecting everything, years hence, whatever anyone says now.  The demands will just never stop until everything is reflected.

 

Not all demands must be satisfied. Other languages (I'm looking at Java) have reflection without expression reflection, for example, so far as I know.

 

 Start writing your algorithms now, switch your helper functions to whatever standards they decide later.

 

Regarding parallel naming I think we’re talking about different things — what we’re doing now IS a parallel naming structure.  meta::is_constexpr(reflinfo) is a parallel of FunctionDecl::isConstexpr() etc.  A full parallel structure of every property is what we’re looking at building.  If everyone’s looked hard at that eventuality and are okay with maintaining that, then I suppose I have no cause to complain — maybe it’s not as hard as I imagine.

 

 



On Nov 12, 2019, at 8:01 AM, Andrew Sutton <[hidden email]> wrote:

 

 

I argue the standards should be based on the clang AST.  Clean it up first of course — and yes that alone would be extraordinarily difficult.  But it’s also very difficult to build a gigantic parallel naming system for reflection properties.  And then to maintain it long term…

 

Best to look hard at what lies ahead, and bite the bullet now.  Clean up the AST, define the standard via it.  (And c’mon we all know those AST nodes have a lot of flab to go with all their muscle; clang would benefit too after we all finish grumbling about the changes.)  Other compilers can do a parallel naming system or adjust their own ASTs to match clang’s naming system — doesn’t need to add any work for them.

 

This is a non-starter for standardization. There are significant and generally incompatible differences in the internal representations of C++ among the different vendors and no will to standardize or even informally adopt a common representation. Even if there were interest in doing so, the engineering cost would be prohibitive, as it would essentially require gutting and rewriting huge swaths of the compiler (and in GCC's case, possibly the middle-end). Parallel naming is also unlikely because of memory requirements.

 

You have the same problem with compiled module interfaces, which serialize the internal representation.

 

Andrew

 


_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: Reflection/metaparsing implem

Tom Stellard via cfe-dev
In reply to this post by Tom Stellard via cfe-dev
Great example.  One nice thing about clang-specific reflection, distinct from the standards effort, is that there are already a lot of clang tools out there that do what we want…and making them constexpr metafunctions is mostly a matter of copying and pasting, since the syntax is the same.  E.g. I see at least one autodiff clang tool out there. 

Substitute auto/constexpr auto for most of the variable types, use constexpr containers (I’ve implem’d vector, set, and map) in place of any heap allocations, use __queue_metaparse("…") instead of outputstream << "…", and…sure to be other constexpr-related issues, but we’d get them figured out.  (Probably need more constexpr const char *-altering functions, a la Andrew’s excellent __concatenate(…) built in function, since std::string would be out.)

The ultimate litmus test might be whether a metafunction can be written to turn any clang tool into a metafunction…

Dave

On Nov 13, 2019, at 3:42 PM, Herb Sutter <[hidden email]> wrote:

I don’t want to weigh in for/against any particular Clang proposal, but FWIW regarding this:
 
> Not all demands must be satisfied.
 
True, but in this case…
 
> Other languages (I'm looking at Java) have reflection without expression reflection
 
… IMO we already know that we will need statement and expression reflection, to enable uses like autodiff. For example, in the ML field I’ve encountered C++ developers who would prefer to use C++ because of its AOT compilation performance, but who of have switched to other languages such as Julia primarily because of compile-time autodiff (and more generally for the broader class of AI/ML transformations that includes autodiff) where today in C++ doing it by hand-coding the derivative function greatly slows down the experimentation phase of an ML project… which otherwise would actually be faster with AOT C++.
 
At least one of the proof-of-concept C++ reflection implementations (Circle) already support statement/expression reflection and autodiff.
 
For statements and expressions, my impression is that reflection is pretty straightforward to specify (here’s another meta::info API) and the primary thing to watch for is that the current proposals for injection (generated code, generative programming) will extend well to statements and expressions, again using autodiff as a litmus test example of what we want to be able to express.
 
Herb
 
 
From: cfe-dev <[hidden email]> On Behalf Of David Blaikie via cfe-dev
Sent: Tuesday, November 12, 2019 11:42 AM
To: David Rector <[hidden email]>
Cc: Clang Dev <[hidden email]>
Subject: Re: [cfe-dev] Reflection/metaparsing implem
 
 
 
On Tue, Nov 12, 2019 at 6:04 AM David Rector <[hidden email]> wrote:
Fair enough.  
 
As I say in my readme, I recommend those who need full reflection facilities — up to their class templates, down to the individual IfStmts etc. in their function defs — to use my implementation and simply isolate dependencies on future standards into their own helper functions.  You can do everything now!  No need to complain to or pressure Andrew (who has done some extraordinary work by the way) about what isn’t yet reflected.
 
*Eventually* the standard will support reflecting everything, years hence, whatever anyone says now.  The demands will just never stop until everything is reflected.
 
Not all demands must be satisfied. Other languages (I'm looking at Java) have reflection without expression reflection, for example, so far as I know.
 
 Start writing your algorithms now, switch your helper functions to whatever standards they decide later.
 
Regarding parallel naming I think we’re talking about different things — what we’re doing now IS a parallel naming structure.  meta::is_constexpr(reflinfo) is a parallel of FunctionDecl::isConstexpr() etc.  A full parallel structure of every property is what we’re looking at building.  If everyone’s looked hard at that eventuality and are okay with maintaining that, then I suppose I have no cause to complain — maybe it’s not as hard as I imagine.
 
 


On Nov 12, 2019, at 8:01 AM, Andrew Sutton <[hidden email]> wrote:
 
 
I argue the standards should be based on the clang AST.  Clean it up first of course — and yes that alone would be extraordinarily difficult.  But it’s also very difficult to build a gigantic parallel naming system for reflection properties.  And then to maintain it long term…
 
Best to look hard at what lies ahead, and bite the bullet now.  Clean up the AST, define the standard via it.  (And c’mon we all know those AST nodes have a lot of flab to go with all their muscle; clang would benefit too after we all finish grumbling about the changes.)  Other compilers can do a parallel naming system or adjust their own ASTs to match clang’s naming system — doesn’t need to add any work for them.
 
This is a non-starter for standardization. There are significant and generally incompatible differences in the internal representations of C++ among the different vendors and no will to standardize or even informally adopt a common representation. Even if there were interest in doing so, the engineering cost would be prohibitive, as it would essentially require gutting and rewriting huge swaths of the compiler (and in GCC's case, possibly the middle-end). Parallel naming is also unlikely because of memory requirements.
 
You have the same problem with compiled module interfaces, which serialize the internal representation.
 
Andrew


_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev