_LIBCPP_INLINE_VISIBILITY and std::string::length

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

_LIBCPP_INLINE_VISIBILITY and std::string::length

Alexey Kudinkin
Hello!

I'm just curious about what are the reasons to forcing inlining of that sort of functions like std::string::length? Precluding them to be included into libc++ binary breaks linkage of the following snippet of code 

int main(int ARGC, char *ARGV[])() {

  std::vector<std::string>  pray = { "I", "will", "not", "aim", "for", "the", "head" };
  std::vector<size_t>       lengths;

  std::transform(pray.begin(), pray.end(), std::back_inserter(lengths), std::mem_fn(&std::string::size));

  return 0;

}

with the error:

Undefined symbols for architecture x86_64:
  "std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >::length() const", referenced from:
      _main in assignment-F0ysIg.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

Best regards,
Alexey Kudinkin


_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: _LIBCPP_INLINE_VISIBILITY and std::string::length

Howard Hinnant
On Mar 8, 2013, at 2:09 PM, Alexey Kudinkin <[hidden email]> wrote:

> Hello!
>
> I'm just curious about what are the reasons to forcing inlining of that sort of functions like std::string::length? Precluding them to be included into libc++ binary breaks linkage of the following snippet of code
>
>> int main(int ARGC, char *ARGV[])() {
>>
>>   std::vector<std::string>  pray = { "I", "will", "not", "aim", "for", "the", "head" };
>>   std::vector<size_t>       lengths;
>>
>>   std::transform(pray.begin(), pray.end(), std::back_inserter(lengths), std::mem_fn(&std::string::size));
>>
>>   return 0;
>>
>> }
>
> with the error:
>
>> Undefined symbols for architecture x86_64:
>>   "std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >::length() const", referenced from:
>>       _main in assignment-F0ysIg.o
>> ld: symbol(s) not found for architecture x86_64
>> clang: error: linker command failed with exit code 1 (use -v to see invocation)

I believe this is due to a poor interaction in the compiler between extern templates and __attribute__ ((__always_inline__)).  If std::string is not declared extern template, then the compiler will outline a size() member and you won't get this link error.

In my opinion, clang should not assume that extern templates have definitions for inlined members, especially those marked always_inline, and the fact that it does is a clang bug resulting in the link error you see.

The rationale for the use of always_inline in libc++ is to control the ABI of libc++.  In the past I have watched compilers use different heuristics from release to release on making the inline/outline decision.  This can cause code to be silently added to and removed from a dylib.  With the use of always_inline, I am telling the compiler to never add that code to the libc++.dylib binary.

Each of the macros defined and used by libc++ can be overridden.

_LIBCPP_INLINE_VISIBILITY controls how an inlined function is attributed and defaults to:

#ifndef _LIBCPP_INLINE_VISIBILITY
#define _LIBCPP_INLINE_VISIBILITY __attribute__ ((__visibility__("hidden"), __always_inline__))
#endif

You can turn this off with:

-D_LIBCPP_INLINE_VISIBILITY=""

And extern templates are done with:

#ifndef _LIBCPP_EXTERN_TEMPLATE
#define _LIBCPP_EXTERN_TEMPLATE(...) extern template __VA_ARGS__;
#endif

This latter one is more difficult to "turn off".  The incantation is:

-D'_LIBCPP_EXTERN_TEMPLATE(...)='

Using either (or both) of these workarounds will silence your link error.  But a bug report against clang might be a better long term solution.

Howard

_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: _LIBCPP_INLINE_VISIBILITY and std::string::length

Howard Hinnant
On Mar 8, 2013, at 3:18 PM, Howard Hinnant <[hidden email]> wrote:

> On Mar 8, 2013, at 2:09 PM, Alexey Kudinkin <[hidden email]> wrote:
>
>> Hello!
>>
>> I'm just curious about what are the reasons to forcing inlining of that sort of functions like std::string::length? Precluding them to be included into libc++ binary breaks linkage of the following snippet of code
>>
>>> int main(int ARGC, char *ARGV[])() {
>>>
>>>  std::vector<std::string>  pray = { "I", "will", "not", "aim", "for", "the", "head" };
>>>  std::vector<size_t>       lengths;
>>>
>>>  std::transform(pray.begin(), pray.end(), std::back_inserter(lengths), std::mem_fn(&std::string::size));
>>>
>>>  return 0;
>>>
>>> }
>>
>> with the error:
>>
>>> Undefined symbols for architecture x86_64:
>>>  "std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >::length() const", referenced from:
>>>      _main in assignment-F0ysIg.o
>>> ld: symbol(s) not found for architecture x86_64
>>> clang: error: linker command failed with exit code 1 (use -v to see invocation)
>
> I believe this is due to a poor interaction in the compiler between extern templates and __attribute__ ((__always_inline__)).  If std::string is not declared extern template, then the compiler will outline a size() member and you won't get this link error.
>
> In my opinion, clang should not assume that extern templates have definitions for inlined members, especially those marked always_inline, and the fact that it does is a clang bug resulting in the link error you see.
>
> The rationale for the use of always_inline in libc++ is to control the ABI of libc++.  In the past I have watched compilers use different heuristics from release to release on making the inline/outline decision.  This can cause code to be silently added to and removed from a dylib.  With the use of always_inline, I am telling the compiler to never add that code to the libc++.dylib binary.
>
> Each of the macros defined and used by libc++ can be overridden.
>
> _LIBCPP_INLINE_VISIBILITY controls how an inlined function is attributed and defaults to:
>
> #ifndef _LIBCPP_INLINE_VISIBILITY
> #define _LIBCPP_INLINE_VISIBILITY __attribute__ ((__visibility__("hidden"), __always_inline__))
> #endif
>
> You can turn this off with:
>
> -D_LIBCPP_INLINE_VISIBILITY=""
>
> And extern templates are done with:
>
> #ifndef _LIBCPP_EXTERN_TEMPLATE
> #define _LIBCPP_EXTERN_TEMPLATE(...) extern template __VA_ARGS__;
> #endif
>
> This latter one is more difficult to "turn off".  The incantation is:
>
> -D'_LIBCPP_EXTERN_TEMPLATE(...)='
>
> Using either (or both) of these workarounds will silence your link error.  But a bug report against clang might be a better long term solution.

Correction, only the second work around works.  clang assumes extern templates have inline member definitions whether they are marked always_inline or not.

Howard

_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: _LIBCPP_INLINE_VISIBILITY and std::string::length

John McCall
In reply to this post by Howard Hinnant
On Mar 8, 2013, at 12:18 PM, Howard Hinnant <[hidden email]> wrote:

> On Mar 8, 2013, at 2:09 PM, Alexey Kudinkin <[hidden email]> wrote:
>> Hello!
>>
>> I'm just curious about what are the reasons to forcing inlining of that sort of functions like std::string::length? Precluding them to be included into libc++ binary breaks linkage of the following snippet of code
>>
>>> int main(int ARGC, char *ARGV[])() {
>>>
>>>  std::vector<std::string>  pray = { "I", "will", "not", "aim", "for", "the", "head" };
>>>  std::vector<size_t>       lengths;
>>>
>>>  std::transform(pray.begin(), pray.end(), std::back_inserter(lengths), std::mem_fn(&std::string::size));

For what it's worth, IIRC this code is not required to work, because you are not generally allowed to take the address of functions from the standard library.

> I believe this is due to a poor interaction in the compiler between extern templates and __attribute__ ((__always_inline__)).  If std::string is not declared extern template, then the compiler will outline a size() member and you won't get this link error.

always_inline doesn't stop the dylib from exporting that function symbol.  It's the hidden visibility that does that.

> In my opinion, clang should not assume that extern templates have definitions for inlined members, especially those marked always_inline, and the fact that it does is a clang bug resulting in the link error you see.

Every member of a class template is implicitly inline.  We are not going to completely break explicit instantiation declarations.

We do already treat always_inline as a special case, and I'm not sure why it's not applying here.  That does seem like a bug.

> The rationale for the use of always_inline in libc++ is to control the ABI of libc++.  In the past I have watched compilers use different heuristics from release to release on making the inline/outline decision.  This can cause code to be silently added to and removed from a dylib.  With the use of always_inline, I am telling the compiler to never add that code to the libc++.dylib binary.

This is an incredibly brute-force way of accomplishing this goal, and it causes major problems downstream, e.g. with debugging programs that use libc++.

Have you considered only explicitly instantiating the functions you actually want to show up in the library instead of explicitly instantiating the entire class?

John.
_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: _LIBCPP_INLINE_VISIBILITY and std::string::length

Howard Hinnant

On Mar 8, 2013, at 4:05 PM, John McCall <[hidden email]> wrote:

> On Mar 8, 2013, at 12:18 PM, Howard Hinnant <[hidden email]> wrote:
>> On Mar 8, 2013, at 2:09 PM, Alexey Kudinkin <[hidden email]> wrote:
>>> Hello!
>>>
>>> I'm just curious about what are the reasons to forcing inlining of that sort of functions like std::string::length? Precluding them to be included into libc++ binary breaks linkage of the following snippet of code
>>>
>>>> int main(int ARGC, char *ARGV[])() {
>>>>
>>>> std::vector<std::string>  pray = { "I", "will", "not", "aim", "for", "the", "head" };
>>>> std::vector<size_t>       lengths;
>>>>
>>>> std::transform(pray.begin(), pray.end(), std::back_inserter(lengths), std::mem_fn(&std::string::size));
>
> For what it's worth, IIRC this code is not required to work, because you are not generally allowed to take the address of functions from the standard library.

Actually I think this was more true in C++03 than in C++11.  The specification says the vendor can add default arguments and/or overloads to a non-virtual member function.  So you can't assume the member function's precise signature.  Now with variadic templates it is a lot easier to take the address without assuming the precise signature of the member function.

>
>> The rationale for the use of always_inline in libc++ is to control the ABI of libc++.  In the past I have watched compilers use different heuristics from release to release on making the inline/outline decision.  This can cause code to be silently added to and removed from a dylib.  With the use of always_inline, I am telling the compiler to never add that code to the libc++.dylib binary.
>
> This is an incredibly brute-force way of accomplishing this goal, and it causes major problems downstream, e.g. with debugging programs that use libc++.
>
> Have you considered only explicitly instantiating the functions you actually want to show up in the library instead of explicitly instantiating the entire class?

No, I hadn't considered that.  I'll give that some thought.  It sounds tedious but it may be worth it.

Thanks,
Howard

_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: _LIBCPP_INLINE_VISIBILITY and std::string::length

Alexey Kudinkin
In reply to this post by John McCall
Sorry, for the not so prompt response.
 
On Mar 8, 2013, at 12:18 PM, Howard Hinnant <[hidden email]> wrote:
> On Mar 8, 2013, at 2:09 PM, Alexey Kudinkin <[hidden email]> wrote:
>> Hello!
>>
>> I'm just curious about what are the reasons to forcing inlining of that sort of functions like std::string::length? Precluding them to be included into libc++ binary breaks linkage of the following snippet of code
>>
>>> int main(int ARGC, char *ARGV[])() {
>>>
>>>  std::vector<std::string>  pray = { "I", "will", "not", "aim", "for", "the", "head" };
>>>  std::vector<size_t>       lengths;
>>>
>>>  std::transform(pray.begin(), pray.end(), std::back_inserter(lengths), std::mem_fn(&std::string::size));
For what it's worth, IIRC this code is not required to work, because you are not generally allowed to take the address of functions from the standard library.

However, i believe that since it didn't result in compilation error it is perfectly clang bug: it seems like it's easily reproducible with extern templates and __attribute__ (( __always_inline__ )) without involving STL, and hence rendering pointers/references to the members pretty useless.


On Sat, Mar 9, 2013 at 1:05 AM, John McCall <[hidden email]> wrote:
On Mar 8, 2013, at 12:18 PM, Howard Hinnant <[hidden email]> wrote:
> On Mar 8, 2013, at 2:09 PM, Alexey Kudinkin <[hidden email]> wrote:
>> Hello!
>>
>> I'm just curious about what are the reasons to forcing inlining of that sort of functions like std::string::length? Precluding them to be included into libc++ binary breaks linkage of the following snippet of code
>>
>>> int main(int ARGC, char *ARGV[])() {
>>>
>>>  std::vector<std::string>  pray = { "I", "will", "not", "aim", "for", "the", "head" };
>>>  std::vector<size_t>       lengths;
>>>
>>>  std::transform(pray.begin(), pray.end(), std::back_inserter(lengths), std::mem_fn(&std::string::size));

For what it's worth, IIRC this code is not required to work, because you are not generally allowed to take the address of functions from the standard library.

> I believe this is due to a poor interaction in the compiler between extern templates and __attribute__ ((__always_inline__)).  If std::string is not declared extern template, then the compiler will outline a size() member and you won't get this link error.

always_inline doesn't stop the dylib from exporting that function symbol.  It's the hidden visibility that does that.

> In my opinion, clang should not assume that extern templates have definitions for inlined members, especially those marked always_inline, and the fact that it does is a clang bug resulting in the link error you see.

Every member of a class template is implicitly inline.  We are not going to completely break explicit instantiation declarations.

We do already treat always_inline as a special case, and I'm not sure why it's not applying here.  That does seem like a bug.

> The rationale for the use of always_inline in libc++ is to control the ABI of libc++.  In the past I have watched compilers use different heuristics from release to release on making the inline/outline decision.  This can cause code to be silently added to and removed from a dylib.  With the use of always_inline, I am telling the compiler to never add that code to the libc++.dylib binary.

This is an incredibly brute-force way of accomplishing this goal, and it causes major problems downstream, e.g. with debugging programs that use libc++.

Have you considered only explicitly instantiating the functions you actually want to show up in the library instead of explicitly instantiating the entire class?

John.



--
Best regards,
Alexey Kudinkin.

_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev