clang_parseTranslationUnit2(), "true", and libclang's header search path

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

clang_parseTranslationUnit2(), "true", and libclang's header search path

Sumner, Brian via cfe-dev
I am writing a program which parses other programs written in C. When
I run my program, I find that it fails to find some key system headers
such as stdbool.h. When "true" is present in the parsed program, the
AST appears to be incomplete. When I replace "true" with "1" in the
parsed program, the AST is complete.

I have written a minimal example which follows at the end of this email.

The "good" input program is:

1  #include <stdbool.h>
2  #include <stdio.h>
3
4  int
5  main(int argc, char *argv[])
6  {
7          while (1) {
8                  printf("Hello, world!\n");
9          }
10 }

For this, the attached program produces:

[...]
LINE 5
LINE 5
LINE 5
LINE 6
LINE 7
LINE 7
LINE 7
LINE 8
LINE 8
LINE 8
LINE 8
LINE 8
LINE 8
DONE

The "bad" input program is:

1  #include <stdbool.h>
2  #include <stdio.h>
3
4  int
5  main(int argc, char *argv[])
6  {
7          while (true) {
8                  printf("Hello, world!\n");
9          }
10  }

For this, the attached program produces:

[...]
LINE 5
LINE 5
LINE 5
LINE 6
DONE

All of this appears to follow from my program failing to find stdbool.h. I
used strace to discover that libclang seems to search

        /usr/local/include/,
        /usr/include/, and
        /include/

but not

        /usr/lib64/clang/3.9.1/include/.

The last directory is where clang installed its headers, so I am
surprised that libclang does not look there.

Is the expectation that I discover /usr/lib64/clang/3.9.1/include/
and pass -I/usr/lib64/clang/3.9.1/include/ to my program?

Why does libclang not search clang's header directory by default?


Here is the program:

#include <clang-c/Index.h>
#include <stdio.h>
#include <stdlib.h>

const char *program;
const char *filename;

void
usage(const char *program)
{
        fprintf(stderr, "usage: %s FILE.C\n", program);
        exit(EXIT_FAILURE);
}

enum CXChildVisitResult
visit_children_cb(CXCursor cursor,
                  CXCursor parent,
                  CXClientData client_data)
{
        CXFile file;
        unsigned int line, column, offset;
        unsigned int level = *(unsigned int *) client_data;
        unsigned int next = level + 1;

        clang_getSpellingLocation(clang_getCursorLocation(cursor),
                                 &file,
                                 &line,
                                 &column,
                                &offset);
        printf("LINE %d\n", line);

        clang_visitChildren(cursor, visit_children_cb, &next);

        return CXChildVisit_Continue;
}

int
main(int argc, const char * const* argv)
{
        unsigned int level = 0;
        CXIndex index;
        enum CXErrorCode error;
        CXTranslationUnit tu;
        CXCursor cursor;

        if (argc != 2) {
                usage(program);
        }

        program  = argv[0];
        filename = argv[1];

        index = clang_createIndex(1, 1);

        error = clang_parseTranslationUnit2(index,
                                            NULL,
                                            argv,
                                            argc,
                                            NULL,
                                            0,
                                            0,
                                           &tu);
        if (CXError_Success != error) {
                fprintf(stderr, "Failure parsing %s\n", filename);
                exit(EXIT_FAILURE);
        }

        cursor = clang_getTranslationUnitCursor(tu);
        clang_visitChildren(cursor, visit_children_cb, &level);

        clang_disposeTranslationUnit(tu);
        clang_disposeIndex(index);

        printf("DONE\n");

        exit(EXIT_SUCCESS);
}

--
Mike

:wq
_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: clang_parseTranslationUnit2(), "true", and libclang's header search path

Sumner, Brian via cfe-dev
On 2017-06-03 13:48, W. Michael Petullo via cfe-dev wrote:
> I am writing a program which parses other programs written in C. When
> I run my program, I find that it fails to find some key system headers
> such as stdbool.h.

IIRC, stdbool.h is a Clang internal header file. They are searched for
relative to the binary.

In my tool I solved this by embedding some of the internal header files
into the executable [1][2]. Then pass them to libclang as virtual files
using CXUnsavedFile [3][4][5].

[1] https://github.com/jacob-carlborg/dstep/blob/master/clang/Compiler.d#L26

[2]
https://github.com/jacob-carlborg/dstep/blob/master/clang/Compiler.d#L37-L38

[3] https://github.com/jacob-carlborg/dstep/blob/master/clang/Compiler.d#L54

[4]
https://github.com/jacob-carlborg/dstep/blob/master/clang/Compiler.d#L53-L54

[5]
https://github.com/jacob-carlborg/dstep/blob/master/dstep/driver/Application.d#L166

--
/Jacob Carlborg

_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: clang_parseTranslationUnit2(), "true", and libclang's header search path

Sumner, Brian via cfe-dev
In reply to this post by Sumner, Brian via cfe-dev


On 06/03/2017 01:48 PM, W. Michael Petullo via cfe-dev wrote:

> I am writing a program which parses other programs written in C. When
> I run my program, I find that it fails to find some key system headers
> such as stdbool.h. When "true" is present in the parsed program, the
> AST appears to be incomplete. When I replace "true" with "1" in the
> parsed program, the AST is complete.
>
> I have written a minimal example which follows at the end of this email.
>
> The "good" input program is:
>
> 1  #include <stdbool.h>
> 2  #include <stdio.h>
> 3
> 4  int
> 5  main(int argc, char *argv[])
> 6  {
> 7          while (1) {
> 8                  printf("Hello, world!\n");
> 9          }
> 10 }
>
> For this, the attached program produces:
>
> [...]
> LINE 5
> LINE 5
> LINE 5
> LINE 6
> LINE 7
> LINE 7
> LINE 7
> LINE 8
> LINE 8
> LINE 8
> LINE 8
> LINE 8
> LINE 8
> DONE
>
> The "bad" input program is:
>
> 1  #include <stdbool.h>
> 2  #include <stdio.h>
> 3
> 4  int
> 5  main(int argc, char *argv[])
> 6  {
> 7          while (true) {
> 8                  printf("Hello, world!\n");
> 9          }
> 10  }
>
> For this, the attached program produces:
>
> [...]
> LINE 5
> LINE 5
> LINE 5
> LINE 6
> DONE
>
> All of this appears to follow from my program failing to find stdbool.h. I
> used strace to discover that libclang seems to search
>
> /usr/local/include/,
> /usr/include/, and
> /include/
>
> but not
>
> /usr/lib64/clang/3.9.1/include/.
>
> The last directory is where clang installed its headers, so I am
> surprised that libclang does not look there.
>
> Is the expectation that I discover /usr/lib64/clang/3.9.1/include/
> and pass -I/usr/lib64/clang/3.9.1/include/ to my program?
>
> Why does libclang not search clang's header directory by default?

FWIW I've seen this problem before too. The rust libclang bindings
"solve" this just trying to run |clang -E -x <lang> -v|, and parsing the
output[1], which is not only somewhat overkill, but also a fairly hacky
solution. It works well in practice though, AFAICT (I don't maintain
that library, but I use it).

I think it doesn't seem unreasonable, and if including them by default
would be breaking (which I guess it would), it may make sense to
retrieve the default include paths as a CXStringSet or something of the
sort.

That being said, I'm not sure how difficult to implement it would be, or
even if it's feasible without a clang binary around?

 -- Emilio

[1]: https://github.com/KyleMayes/clang-sys/blob/master/src/support.rs#L182
_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: clang_parseTranslationUnit2(), "true", and libclang's header search path

Sumner, Brian via cfe-dev
[...]

>> All of this appears to follow from my program failing to find stdbool.h. I
>> used strace to discover that libclang seems to search
>>
>> /usr/local/include/,
>> /usr/include/, and
>> /include/
>>
>> but not
>>
>> /usr/lib64/clang/3.9.1/include/.
>>
>> The last directory is where clang installed its headers, so I am
>> surprised that libclang does not look there.
>>
>> Is the expectation that I discover /usr/lib64/clang/3.9.1/include/
>> and pass -I/usr/lib64/clang/3.9.1/include/ to my program?

[From Jacob Carlborg:]
> In my tool I solved this by embedding some of the internal header files into
> the executable. Then pass them to libclang as virtual files using
> CXUnsavedFile
 
[From Emilio Cobos Álvarez:]
> FWIW I've seen this problem before too. The rust libclang bindings
> "solve" this just trying to run |clang -E -x <lang> -v|, and parsing the
> output, which is not only somewhat overkill, but also a fairly hacky
> solution. It works well in practice though, AFAICT (I don't maintain
> that library, but I use it).

Both of these solutions seem a bit kludgy. Because the former hard-codes
a header in the binary which uses clang, a system upgrade might result in
an inconsistency between the binary and the filesystem copy of the header.
The latter seems kludgy too for the reason Emilio described.

Another idea is to find, e.g., /usr/lib64/clang/3.9.1/include/ at
./configure time and set a string in the program which uses libclang. But
this too is asking for an inconsistency: The path might change if the
clang installation is updated without reconfiguring and rebuilding
the program.

>> Why does libclang not search clang's header directory by default?

[From Emilio Cobos Álvarez:]
> I think it doesn't seem unreasonable, and if including them by default
> would be breaking (which I guess it would), it may make sense to
> retrieve the default include paths as a CXStringSet or something of the
> sort.
>
> That being said, I'm not sure how difficult to implement it would be, or
> even if it's feasible without a clang binary around?

I am surprised that this is not already the case since the libraries
ship with clang itself. Does anyone else have any comments as to why,
e.g., /usr/lib64/clang/3.9.1/include/ is not already known to libclang?
At the very least clang could install a configuration file which contains
the location of these internal headers.

--
Mike

:wq
_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Loading...