libclang c API will not find c++ system headers

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

libclang c API will not find c++ system headers

Kristóf Umann via cfe-dev
Hi

I wanted to sort out a bug/limitation in a color coding plugin for neovim (not maintained by me), and tried to set up a vanilla c++ project to find out how libclang works. This is what I've done:

1) Download prebuilt binaries of clang+llvm-8.0.0 for macOS. I later downloaded and tried 7.0.0 as well
2) Create command line tool project in Xcode
3) Link to libclang.dylib under Linked Frameworks and Libraries
4) Add /path/to/clang+llvm/lib under Library search paths
5) Add /path/to/clang+llvm/include under Header search paths
6) Add /path/to/clang+llvm/lib under runpath search paths
7) Create new c++ target with only the auto-generated code to test parsing
8) Build the new target from command line, and copy the clang invocation from the build log
9) Run regex replace on the clang invocation to be sure I have an exact copy of the command line args during build of the test target, as an array of c strings (const char *)
10) Call clang_parseTranslationUnit2 with those parameters, with and without argv[0] (the executable path).

This is my main implementation:

int main(int argc, const char * argv[]) {
    CXIndex idx = clang_createIndex(0,1);
    CXTranslationUnit tu;

    CXErrorCode errorCode;

    if( !( errorCode = clang_parseTranslationUnit2( idx,
                                    NULL,
                                    options,
                                    opt_count,
                                    NULL,
                                    0,
                                    0,
                                    &tu) ) )

        std::cout << "Success\n";
    else
        std::cout << "error: " << errorCode << std::endl;

    clang_disposeTranslationUnit(tu);
    clang_disposeIndex(idx);
    return 0;
}

When I try it on a c file, with system header includes, it produces the following output:
Success

With a c++ file, it produces the following output:
/Users/pbholmen/Projects/ColorCodingTest/Vanilla/main.cpp:9:10: fatal error: 'iostream' file not found
Success

This is the file it tries to parse:
#include <iostream>

int main(int argc, const char * argv[]) {
     // insert code here...
     std::cout << "Hello, World!\n";
     return 0;
}

I have tried to invoke the clang binary inside the downloaded binaries folder, from the command line, compiling the same file with the exact same options as given to
clang_parseTranslationUnit2, and it compiles without issues.

I can post the list of command line options input to clang_parseTranslationUnit2 as well, but the message is getting long and I'd feel I'm spamming you with tons of Xcode-generated
options, but if you ask for it I'll post it as well. For argv[0] I tried both with the path to the Xcode-provided clang, the path to the dowloaded clang, and omitting the path. I tried to provide
the source file both as an argument to clang_parseTranslationUnit2, and embedded in my argv. Same result either way. I have not deleted any files or folders inside the downloaded prebuilt
binaries folder or relocated libclang.dylib.

Per
_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: libclang c API will not find c++ system headers

Kristóf Umann via cfe-dev
On 2019-08-21 16:54, Per Bull Holmen via cfe-dev wrote:

> Hi
>
> I wanted to sort out a bug/limitation in a color coding plugin for neovim (not maintained by me), and tried to set up a vanilla c++ project to find out how libclang works. This is what I've done:
>
> 1) Download prebuilt binaries of clang+llvm-8.0.0 for macOS. I later downloaded and tried 7.0.0 as well
> 2) Create command line tool project in Xcode
> 3) Link to libclang.dylib under Linked Frameworks and Libraries
> 4) Add /path/to/clang+llvm/lib under Library search paths
> 5) Add /path/to/clang+llvm/include under Header search paths
> 6) Add /path/to/clang+llvm/lib under runpath search paths
> 7) Create new c++ target with only the auto-generated code to test parsing
> 8) Build the new target from command line, and copy the clang invocation from the build log
> 9) Run regex replace on the clang invocation to be sure I have an exact copy of the command line args during build of the test target, as an array of c strings (const char *)
> 10) Call clang_parseTranslationUnit2 with those parameters, with and without argv[0] (the executable path).
>
> This is my main implementation:
>
> int main(int argc, const char * argv[]) {
>      CXIndex idx = clang_createIndex(0,1);
>      CXTranslationUnit tu;
>
>      CXErrorCode errorCode;
>
>      if( !( errorCode = clang_parseTranslationUnit2( idx,
>                                      NULL,
>                                      options,
>                                      opt_count,
>                                      NULL,
>                                      0,
>                                      0,
>                                      &tu) ) )
>
>          std::cout << "Success\n";
>      else
>          std::cout << "error: " << errorCode << std::endl;
>
>      clang_disposeTranslationUnit(tu);
>      clang_disposeIndex(idx);
>      return 0;
> }
>
> When I try it on a c file, with system header includes, it produces the following output:
> Success
>
> With a c++ file, it produces the following output:
> /Users/pbholmen/Projects/ColorCodingTest/Vanilla/main.cpp:9:10: fatal error: 'iostream' file not found
> Success
>
> This is the file it tries to parse:
> #include <iostream>
>
> int main(int argc, const char * argv[]) {
>       // insert code here...
>       std::cout << "Hello, World!\n";
>       return 0;
> }
>
> I have tried to invoke the clang binary inside the downloaded binaries folder, from the command line, compiling the same file with the exact same options as given to
> clang_parseTranslationUnit2, and it compiles without issues.
>
> I can post the list of command line options input to clang_parseTranslationUnit2 as well, but the message is getting long and I'd feel I'm spamming you with tons of Xcode-generated
> options, but if you ask for it I'll post it as well. For argv[0] I tried both with the path to the Xcode-provided clang, the path to the dowloaded clang, and omitting the path. I tried to provide
> the source file both as an argument to clang_parseTranslationUnit2, and embedded in my argv. Same result either way. I have not deleted any files or folders inside the downloaded prebuilt
> binaries folder or relocated libclang.dylib.

The per-compiled binaries available at http://releases.llvm.org are
configured differently compared to the binaries shipped by Apple. You
need to either manually (either on the command line or directly in your
application) add these paths or run an extra installation step to
install the headers in /usr/include where both binaries are looking. If
you run Clang with the "-v" flag and compare the one shipped by Apple
and the one provided at http://releases.llvm.org you will see the
difference.

--
/Jacob Carlborg

_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: libclang c API will not find c++ system headers

Kristóf Umann via cfe-dev

On Wednesday, August 21, 2019, 08:40:26 PM GMT+2, Jacob Carlborg via
cfe-dev <[hidden email]> wrote:

> The per-compiled binaries available at http://releases.llvm.org are 
> configured differently compared to the binaries shipped by Apple. You 
> need to either manually (either on the command line or directly in your 
> application) add these paths or run an extra installation step to 
> install the headers in /usr/include where both binaries are looking. If 
> you run Clang with the "-v" flag and compare the one shipped by Apple 
> and the one provided at http://releases.llvm.org you will see the 
> difference.

Thank you. I've tried running with the -v option now, and there is some
difference. This might indicate that third-party plugin and tool
developers should link to the Apple-provided libclang on macOS.

However, I'm not sure it's needed, as I did make it work with the
precompiled binaries at http://releases.llvm.org/download.html simply by
calling clang_parseTranslationUnit2FullArgv, and providing the path to
the compiler executable in argv[0]. No extra steps necessary, no adding
paths, and no change of compiler options beyond those supplied by Xcode
(from the xcodebuild log).

I see some third-party developers of libclang-based plugins and tools
struggle with this, so I tried to do the same thing with a more complex
example and some Apple-specific system header includes that I know
vim plugin devs have struggled with, more specifically umbrella headers
in Apple Frameworks, and again it works smoothly right out of the box.
No errors reported, and I can traverse the AST with no problems.

I've seen plugin devs do all sorts of hacks to work around issues that
could have been solved this way, so I'd like to share a list of
what I did to make it work, in the faint hope that other third party
devs who run into the same problem read it:

1) Make sure you load the libclang you expect, and not some arbitrary
libclang the user might have in their system. Remember that you might
link to one version at compile time, yet your tool or plugin loads a
different version at runtime.

2) Do not move libclang out of the clang+llvm distribution folder, or
remove files that libclang expects to find there. It seems libclang
expects to find certain files relative to the lib location.

3) Use clang_parseTranslationUnit2FullArgv, and provide the path to the
compiler executable in argv[0]. Interestingly, it made no difference in
my case whether I used the one in the pre-compiled binaries folder
downloaded at http://releases.llvm.org/download.html, or the one in
Xcode's toolchain.

I suggest tools and plugins don't mangle with the command line options,
like adding paths or remove -isysroot etc. unless they have tried
everything above (and more), and it still doesn't work. Doing so creates
all kinds of future compatibility issues, and breaks the invariant that
users of your tool should always be able to depend on: If it works on
the command line, with the same clang version, and exactly the same
compiler options, then it should also work with your tool or plug-in.

I don't develop or maintain a cross-platform tool myself, so I can't
guarantee that this invariant always holds for libclang itself,
but I've seen cases where devs of such tools think they have to find
workarounds when it could have been solved by the above steps, and
haven't myself seen cases where it doesn't hold.

Per

_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: libclang c API will not find c++ system headers

Kristóf Umann via cfe-dev
On 2019-08-22 23:46, Per Bull Holmen via cfe-dev wrote:

> Thank you. I've tried running with the -v option now, and there is some
> difference. This might indicate that third-party plugin and tool
> developers should link to the Apple-provided libclang on macOS.
>
> However, I'm not sure it's needed, as I did make it work with the
> precompiled binaries at http://releases.llvm.org/download.html simply by
> calling clang_parseTranslationUnit2FullArgv, and providing the path to
> the compiler executable in argv[0]. No extra steps necessary, no adding
> paths, and no change of compiler options beyond those supplied by Xcode
> (from the xcodebuild log).

Hmm, passing the path to the compiler executable in argv[0] instead of
the tool sounds interesting. I haven't thought about that before. But I
don't want my tool to require to have the compiler installed. On macOS
it might not matter, but on Linux and Windows it's less likely that
Clang is installed.

> I see some third-party developers of libclang-based plugins and tools
> struggle with this, so I tried to do the same thing with a more complex
> example and some Apple-specific system header includes that I know
> vim plugin devs have struggled with, more specifically umbrella headers
> in Apple Frameworks, and again it works smoothly right out of the box.
> No errors reported, and I can traverse the AST with no problems.
>
> I've seen plugin devs do all sorts of hacks to work around issues that
> could have been solved this way, so I'd like to share a list of
> what I did to make it work, in the faint hope that other third party
> devs who run into the same problem read it:
>
> 1) Make sure you load the libclang you expect, and not some arbitrary
> libclang the user might have in their system. Remember that you might
> link to one version at compile time, yet your tool or plugin loads a
> different version at runtime.

With my tool, I statically link libclang. I've got too many bug reports
from users that have used an unsupported version of libclang.

> 2) Do not move libclang out of the clang+llvm distribution folder, or
> remove files that libclang expects to find there. It seems libclang
> expects to find certain files relative to the lib location.
>
> 3) Use clang_parseTranslationUnit2FullArgv, and provide the path to the
> compiler executable in argv[0]. Interestingly, it made no difference in
> my case whether I used the one in the pre-compiled binaries folder
> downloaded at http://releases.llvm.org/download.html, or the one in
> Xcode's toolchain.
>
> I suggest tools and plugins don't mangle with the command line options,
> like adding paths or remove -isysroot etc. unless they have tried
> everything above (and more), and it still doesn't work. Doing so creates
> all kinds of future compatibility issues, and breaks the invariant that
> users of your tool should always be able to depend on: If it works on
> the command line, with the same clang version, and exactly the same
> compiler options, then it should also work with your tool or plug-in.
>
> I don't develop or maintain a cross-platform tool myself, so I can't
> guarantee that this invariant always holds for libclang itself,
> but I've seen cases where devs of such tools think they have to find
> workarounds when it could have been solved by the above steps, and
> haven't myself seen cases where it doesn't hold.

I have the SDK installed in the /usr directory as well, so both the
Apple provided compiler and the on at provided by llvm.org will find the
SDK.

--
/Jacob Carlborg

_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev