How to use libtooling to parse multiple files at once? and succesfully find stddef.h?

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

How to use libtooling to parse multiple files at once? and succesfully find stddef.h?

rayjcwu
This post was updated on .
I'm writing a tool to parse C family source code basically follow these tutorials
[1] http://clang.llvm.org/docs/LibTooling.html#libtooling-builtin-includes
[2] http://kevinaboos.blogspot.com/2013/07/clang-tutorial-part-ii-libtooling.html

I try to parse opencv for now since it uses CMake allowing me to get a correct compile_commands.json for free.

My first problme is how do I specify to parse thousands of files at once? At first I thought giving libtooling the path of compile_commands.json then it will do the rest. However, if I use run  "$ mybinary -p . -- " and use CommonOptionsParser to process argc, argv, it will complain "Not enough positional command line arguments specified!", so I still have to specified all files I want to parse.

If I use CompilationDatabase::autoDetectFromDirectory to load compile_commands.json and use getAllFiles() to pass to ClangTool to parse all files, it could parse but it can't find headers like stdarg.h, stddef.h and such. In FAQ it said use clang with -### option will see all options necessary for this issue.

However, since I'm using CompilationDatabase::autoDetectFromDirectory to start parsing, where should these options go? If I pass these options to CommonOptionsParser, how to do I start parsing (since it always complains about missing positional arguments). Besides, I want my tool only need to pass the path containing compile_commands.json as argument, how could my program get options from "clang -### -c _file_to_compile_"?

==== Update ====

Read CommonOptionsParser.cpp source code. It uses FixedCompilationDatabase to guess CompilationDatabase by arguments after --, then pass arguments before -- for custom (only -p in CommonOptionParser) options. In my case compile_commands.json is required, so I could skip using CommonOptionsParser.

Therefore my problem reduce to how to pass those options from "clang -###" to LibTooling when I have a compile_commands.json? Should I invoke shell command for every file I want to parse?
Reply | Threaded
Open this post in threaded view
|

Re: How to use libtooling to parse multiple files at once? and succesfully find stddef.h?

rayjcwu
Edwin Vane replied me says compilation database should be self contained. First I need to make sure the compile_commands.json is generated with using clang and I could use clang to build opencv.

I set these environment variables

export CC=/home/jcwu/repos/llvm-release/Release/bin/clang
export CXX=/home/jcwu/repos/llvm-release/Release/bin/clang++
export C_INCLUDE_PATH=/usr/local/include:/home/jcwu/repos/llvm-release/Release/lib/clang/3.4/include:/usr/include/x86_64-linux-gnu:/usr/include  # these are from clang -v -c files.cpp
export CPLUS_INCLUDE_PATH=/usr/local/include:/home/jcwu/repos/llvm-release/Release/lib/clang/3.4/include:/usr/include/x86_64-linux-gnu:/usr/include

then regenerate compile_commands.json, it could find stddef.h but new issue comes up

[ 31%] Building CXX object modules/ts/CMakeFiles/opencv_ts.dir/src/ts.cpp.o
In file included from /home/jcwu/repos/opencv/modules/ts/src/ts.cpp:116:
/usr/include/setjmp.h:60:12: error: conflicting types for '__sigsetjmp'
extern int __sigsetjmp (struct __jmp_buf_tag __env[1], int __savemask) __THROWNL;
           ^
/usr/include/pthread.h:727:12: note: previous declaration is here
extern int __sigsetjmp (struct __jmp_buf_tag *__env, int __savemask) __THROW;
           ^
1 error generated.
make[2]: *** [modules/ts/CMakeFiles/opencv_ts.dir/src/ts.cpp.o] Error 1
make[1]: *** [modules/ts/CMakeFiles/opencv_ts.dir/all] Error 2
make: *** [all] Error 2

I can't use clang to build opencv due to a type conflict or two system header files.
Havne't figured out how to solve this.
Reply | Threaded
Open this post in threaded view
|

Re: How to use libtooling to parse multiple files at once? and succesfully find stddef.h?

Manuel Klimek
On Wed, Oct 30, 2013 at 4:54 AM, rayjcwu <[hidden email]> wrote:
Edwin Vane replied me says compilation database should be self contained.
First I need to make sure the compile_commands.json is generated with using
clang and I could use clang to build opencv.

I set these environment variables

export CC=/home/jcwu/repos/llvm-release/Release/bin/clang
export CXX=/home/jcwu/repos/llvm-release/Release/bin/clang++
export
C_INCLUDE_PATH=/usr/local/include:/home/jcwu/repos/llvm-release/Release/lib/clang/3.4/include:/usr/include/x86_64-linux-gnu:/usr/include
# these are from clang -v -c files.cpp
export
CPLUS_INCLUDE_PATH=/usr/local/include:/home/jcwu/repos/llvm-release/Release/lib/clang/3.4/include:/usr/include/x86_64-linux-gnu:/usr/include

then regenerate compile_commands.json, it could find stddef.h but new issue
comes up

[ 31%] Building CXX object modules/ts/CMakeFiles/opencv_ts.dir/src/ts.cpp.o
In file included from /home/jcwu/repos/opencv/modules/ts/src/ts.cpp:116:
/usr/include/setjmp.h:60:12: error: conflicting types for '__sigsetjmp'
extern int __sigsetjmp (struct __jmp_buf_tag __env[1], int __savemask)
__THROWNL;
           ^
/usr/include/pthread.h:727:12: note: previous declaration is here
extern int __sigsetjmp (struct __jmp_buf_tag *__env, int __savemask)
__THROW;
           ^
1 error generated.
make[2]: *** [modules/ts/CMakeFiles/opencv_ts.dir/src/ts.cpp.o] Error 1
make[1]: *** [modules/ts/CMakeFiles/opencv_ts.dir/all] Error 2
make: *** [all] Error 2

I can't use clang to build opencv due to a type conflict or two system
header files.
Havne't figured out how to solve this.

Well, if you cannot use clang to build your code, you cannot use libtooling to parse it (or am I misunderstanding something?)

Cheers,
/Manuel
 



--
View this message in context: http://clang-developers.42468.n3.nabble.com/How-to-use-libtooling-to-parse-multiple-files-at-once-and-succesfully-find-stddef-h-tp4035389p4035422.html
Sent from the Clang Developers mailing list archive at Nabble.com.
_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev


_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: How to use libtooling to parse multiple files at once? and succesfully find stddef.h?

rayjcwu
I change my target to gnu tools. Taking wget for example, I could build it by clang and I use Bear to generate compile_commands.json.
When I use my code, which does nothing meanful except visiting, to visit all ast nodes in source code, I will get a lot of header files not found error.
But I could use sourceweb, which is listed on exteral clang projects, to parse wget with the same .json file without any trouble. Even if I change my llvm/clang to version 3.2 the error still exists.

Is there any quick possible solution for that before I tracing sourceweb source code?

======
1 error generated.
Error while processing /home/jcwu/repos/wget-1.14/src/init.c.
glthread/threadlib.c:19:10: fatal error: 'config.h' file not found
#include <config.h>
         ^
1 error generated.
Error while processing /home/jcwu/repos/wget-1.14/lib/glthread/threadlib.c.
In file included from ftp-basic.c:32:
./wget.h:40:10: fatal error: 'config.h' file not found
#include "config.h"
         ^
1 error generated.
Error while processing /home/jcwu/repos/wget-1.14/src/ftp-basic.c.
futimens.c:19:10: fatal error: 'config.h' file not found
#include <config.h>
         ^
1 error generated.
Error while processing /home/jcwu/repos/wget-1.14/lib/futimens.c.
stripslash.c:19:10: fatal error: 'config.h' file not found
#include <config.h>
         ^
1 error generated.
Error while processing /home/jcwu/repos/wget-1.14/lib/stripslash.c.
dup-safer-flag.c:21:10: fatal error: 'config.h' file not found
#include <config.h>
         ^
1 error generated.
Error while processing /home/jcwu/repos/wget-1.14/lib/dup-safer-flag.c.
In file included from html-parse.c:92:
./wget.h:40:10: fatal error: 'config.h' file not found
#include "config.h"
         ^
1 error generated.
Error while processing /home/jcwu/repos/wget-1.14/src/html-parse.c.
In file included from http.c:32:
./wget.h:40:10: fatal error: 'config.h' file not found
#include "config.h"
         ^
1 error generated.
Error while processing /home/jcwu/repos/wget-1.14/src/http.c.
In file included from recur.c:32:
./wget.h:40:10: fatal error: 'config.h' file not found
#include "config.h"
         ^
1 error generated.

Reply | Threaded
Open this post in threaded view
|

Re: How to use libtooling to parse multiple files at once? and succesfully find stddef.h?

Lukas Vogel
This is because clang has some own headers.

Well sourceweb finds the headers because it uses the Clang Path to find the include directory (I guess).

You can manually specify the include command (with the -I flag) for each Source-File in the compile_commands.json file.
In linux the path to the files is something like /usr/lib/clang/3.3/include or /usr/lib/llvm-3.4/include/clang/


Hope that helps,

Cheers


2013/11/4 rayjcwu <[hidden email]>
I change my target to gnu tools. Taking wget for example, I could build it by
clang and I use  Bear <https://github.com/rizsotto/Bear>   to generate
compile_commands.json.
When I use  my code
<https://github.com/rayjcwu/clang.libtooling.annotator/blob/master/main.cpp>
, which does nothing meanful except visiting, to visit all ast nodes in
source code, I will get a lot of header files not found error.
But I could use  sourceweb <https://github.com/rprichard/sourceweb>  , which
is listed on exteral clang projects, to parse wget with the same .json file
without any trouble. Even if I change my llvm/clang to version 3.2 the error
still exists.

Is there any quick possible solution for that before I tracing sourceweb
source code?

======
1 error generated.
Error while processing /home/jcwu/repos/wget-1.14/src/init.c.
glthread/threadlib.c:19:10: fatal error: 'config.h' file not found
#include <config.h>
         ^
1 error generated.
Error while processing /home/jcwu/repos/wget-1.14/lib/glthread/threadlib.c.
In file included from ftp-basic.c:32:
./wget.h:40:10: fatal error: 'config.h' file not found
#include "config.h"
         ^
1 error generated.
Error while processing /home/jcwu/repos/wget-1.14/src/ftp-basic.c.
futimens.c:19:10: fatal error: 'config.h' file not found
#include <config.h>
         ^
1 error generated.
Error while processing /home/jcwu/repos/wget-1.14/lib/futimens.c.
stripslash.c:19:10: fatal error: 'config.h' file not found
#include <config.h>
         ^
1 error generated.
Error while processing /home/jcwu/repos/wget-1.14/lib/stripslash.c.
dup-safer-flag.c:21:10: fatal error: 'config.h' file not found
#include <config.h>
         ^
1 error generated.
Error while processing /home/jcwu/repos/wget-1.14/lib/dup-safer-flag.c.
In file included from html-parse.c:92:
./wget.h:40:10: fatal error: 'config.h' file not found
#include "config.h"
         ^
1 error generated.
Error while processing /home/jcwu/repos/wget-1.14/src/html-parse.c.
In file included from http.c:32:
./wget.h:40:10: fatal error: 'config.h' file not found
#include "config.h"
         ^
1 error generated.
Error while processing /home/jcwu/repos/wget-1.14/src/http.c.
In file included from recur.c:32:
./wget.h:40:10: fatal error: 'config.h' file not found
#include "config.h"
         ^
1 error generated.





--
View this message in context: http://clang-developers.42468.n3.nabble.com/How-to-use-libtooling-to-parse-multiple-files-at-once-and-succesfully-find-stddef-h-tp4035389p4035587.html
Sent from the Clang Developers mailing list archive at Nabble.com.
_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev


_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: How to use libtooling to parse multiple files at once? and succesfully find stddef.h?

Manuel Klimek
On Mon, Nov 4, 2013 at 1:11 PM, Lukas Vogel <[hidden email]> wrote:
This is because clang has some own headers.

Well sourceweb finds the headers because it uses the Clang Path to find the include directory (I guess).

You can manually specify the include command (with the -I flag) for each Source-File in the compile_commands.json file.
In linux the path to the files is something like /usr/lib/clang/3.3/include or /usr/lib/llvm-3.4/include/clang/


I don't think the builtin headers are a problem here - config.h seems to be a header in the project that's not found. 

Hope that helps,

Cheers


2013/11/4 rayjcwu <[hidden email]>
I change my target to gnu tools. Taking wget for example, I could build it by
clang and I use  Bear <https://github.com/rizsotto/Bear>   to generate
compile_commands.json.
When I use  my code
<https://github.com/rayjcwu/clang.libtooling.annotator/blob/master/main.cpp>
, which does nothing meanful except visiting, to visit all ast nodes in
source code, I will get a lot of header files not found error.
But I could use  sourceweb <https://github.com/rprichard/sourceweb>  , which
is listed on exteral clang projects, to parse wget with the same .json file
without any trouble. Even if I change my llvm/clang to version 3.2 the error
still exists.

Is there any quick possible solution for that before I tracing sourceweb
source code?

======
1 error generated.
Error while processing /home/jcwu/repos/wget-1.14/src/init.c.
glthread/threadlib.c:19:10: fatal error: 'config.h' file not found
#include <config.h>
         ^
1 error generated.
Error while processing /home/jcwu/repos/wget-1.14/lib/glthread/threadlib.c.
In file included from ftp-basic.c:32:
./wget.h:40:10: fatal error: 'config.h' file not found
#include "config.h"
         ^
1 error generated.
Error while processing /home/jcwu/repos/wget-1.14/src/ftp-basic.c.
futimens.c:19:10: fatal error: 'config.h' file not found
#include <config.h>
         ^
1 error generated.
Error while processing /home/jcwu/repos/wget-1.14/lib/futimens.c.
stripslash.c:19:10: fatal error: 'config.h' file not found
#include <config.h>
         ^
1 error generated.
Error while processing /home/jcwu/repos/wget-1.14/lib/stripslash.c.
dup-safer-flag.c:21:10: fatal error: 'config.h' file not found
#include <config.h>
         ^
1 error generated.
Error while processing /home/jcwu/repos/wget-1.14/lib/dup-safer-flag.c.
In file included from html-parse.c:92:
./wget.h:40:10: fatal error: 'config.h' file not found
#include "config.h"
         ^
1 error generated.
Error while processing /home/jcwu/repos/wget-1.14/src/html-parse.c.
In file included from http.c:32:
./wget.h:40:10: fatal error: 'config.h' file not found
#include "config.h"
         ^
1 error generated.
Error while processing /home/jcwu/repos/wget-1.14/src/http.c.
In file included from recur.c:32:
./wget.h:40:10: fatal error: 'config.h' file not found
#include "config.h"
         ^
1 error generated.





--
View this message in context: http://clang-developers.42468.n3.nabble.com/How-to-use-libtooling-to-parse-multiple-files-at-once-and-succesfully-find-stddef-h-tp4035389p4035587.html
Sent from the Clang Developers mailing list archive at Nabble.com.
_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev


_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev



_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: How to use libtooling to parse multiple files at once? and succesfully find stddef.h?

Manuel Klimek
In reply to this post by rayjcwu
I'd run your normal build in verbose mode, and compare the command lines you get to what you find in compile_commands.json.


On Mon, Nov 4, 2013 at 3:48 AM, rayjcwu <[hidden email]> wrote:
I change my target to gnu tools. Taking wget for example, I could build it by
clang and I use  Bear <https://github.com/rizsotto/Bear>   to generate
compile_commands.json.
When I use  my code
<https://github.com/rayjcwu/clang.libtooling.annotator/blob/master/main.cpp>
, which does nothing meanful except visiting, to visit all ast nodes in
source code, I will get a lot of header files not found error.
But I could use  sourceweb <https://github.com/rprichard/sourceweb>  , which
is listed on exteral clang projects, to parse wget with the same .json file
without any trouble. Even if I change my llvm/clang to version 3.2 the error
still exists.

Is there any quick possible solution for that before I tracing sourceweb
source code?

======
1 error generated.
Error while processing /home/jcwu/repos/wget-1.14/src/init.c.
glthread/threadlib.c:19:10: fatal error: 'config.h' file not found
#include <config.h>
         ^
1 error generated.
Error while processing /home/jcwu/repos/wget-1.14/lib/glthread/threadlib.c.
In file included from ftp-basic.c:32:
./wget.h:40:10: fatal error: 'config.h' file not found
#include "config.h"
         ^
1 error generated.
Error while processing /home/jcwu/repos/wget-1.14/src/ftp-basic.c.
futimens.c:19:10: fatal error: 'config.h' file not found
#include <config.h>
         ^
1 error generated.
Error while processing /home/jcwu/repos/wget-1.14/lib/futimens.c.
stripslash.c:19:10: fatal error: 'config.h' file not found
#include <config.h>
         ^
1 error generated.
Error while processing /home/jcwu/repos/wget-1.14/lib/stripslash.c.
dup-safer-flag.c:21:10: fatal error: 'config.h' file not found
#include <config.h>
         ^
1 error generated.
Error while processing /home/jcwu/repos/wget-1.14/lib/dup-safer-flag.c.
In file included from html-parse.c:92:
./wget.h:40:10: fatal error: 'config.h' file not found
#include "config.h"
         ^
1 error generated.
Error while processing /home/jcwu/repos/wget-1.14/src/html-parse.c.
In file included from http.c:32:
./wget.h:40:10: fatal error: 'config.h' file not found
#include "config.h"
         ^
1 error generated.
Error while processing /home/jcwu/repos/wget-1.14/src/http.c.
In file included from recur.c:32:
./wget.h:40:10: fatal error: 'config.h' file not found
#include "config.h"
         ^
1 error generated.





--
View this message in context: http://clang-developers.42468.n3.nabble.com/How-to-use-libtooling-to-parse-multiple-files-at-once-and-succesfully-find-stddef-h-tp4035389p4035587.html
Sent from the Clang Developers mailing list archive at Nabble.com.
_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev


_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: How to use libtooling to parse multiple files at once? and succesfully find stddef.h?

rayjcwu
In reply to this post by Lukas Vogel
I set the environment variables C_INCLUDE_PATH, CXX_INCLUDE_PATH to /usr/local/include:/home/myaccount/repos/llvm/Release/lib/clang/3.4/include:/usr/include/x86_64-linux-gnu:/usr/include.
I'm not sure what's the difference between giving -I for each command in compile_commands.json and only setting these two variables, but it solves the "stddef.h not found" (while using libtooling), so I guess setting these two variables is enough.

There are two kinds of config.h, one is <config.h> and the other is "config.h". The build-in headers solution be irrelevant to "config.h" error.

I'm comparing output generate by -v with compile_commands.json base on Manuel Klimek's suggestion. Now sure what I'll find...



Reply | Threaded
Open this post in threaded view
|

Re: How to use libtooling to parse multiple files at once? and succesfully find stddef.h?

rayjcwu
In reply to this post by Manuel Klimek
Poking here and there but I still can't figure out why.
I take away the environment variables C_INCLUDE_PATH. Take tempname.c for example, if I use verbose mode in make file, I have

=====
depbase=`echo tempname.o | sed 's|[^/]*$|.deps/&|;s|\.o$||'`;\
        /home/jcwu/repos/llvm-release/Release/bin/clang -DHAVE_CONFIG_H -I. -I../src     -O2 -Wall -MT tempname.o -MD -MP -MF $depbase.Tpo -c -o tempname.o tempname.c &&\
        mv -f $depbase.Tpo $depbase.Po
=====

Which is almost the same with the command in compile_commands.json

====
/home/jcwu/repos/llvm-release/Release/bin/clang -DHAVE_CONFIG_H -I. -I../src -O2 -Wall -MT tempname.o -MD -MP -MF .deps/tempname.Tpo -c -o tempname.o tempname.c
====

Original compile_commands.json has a lot of stddef.h not found error. I add paths generated by -v option, so for this output

====
ignoring nonexistent directory "/include"
#include "..." search starts here:
#include <...> search starts here:
 .
 ../src
* /usr/local/include            
* /home/jcwu/repos/llvm-release/Release/lib/clang/3.4/include
* /usr/include/x86_64-linux-gnu
* /usr/include
End of search list.
====

I put those paths marked with * with -I option in compile_commands.json. There are still ot of errors, take this error message for example

===
Processing: /home/jcwu/repos/wget-1.14/lib/tempname.c.
tempname.c:21:11: fatal error: cannot open file './config.h': No such file or directory
# include <config.h>
          ^
1 error generated.
Error while processing /home/jcwu/repos/wget-1.14/lib/tempname.c.
===

The modified section for this file in compile_commands.json is

===
  {
    "directory": "/home/jcwu/repos/wget-1.14/lib",
    "command": "/home/jcwu/repos/llvm-release/Release/bin/clang -DHAVE_CONFIG_H -I. -I../src -I/usr/local/include -I/home/jcwu/repos/llvm-release/Release/lib/clang/3.4/include -I/usr/include/x86_64-linux-gnu -I/usr/include  -O2 -Wall -MT tempname.o -MD -MP -MF .deps/tempname.Tpo -c -o tempname.o tempname.c",
    "file": "/home/jcwu/repos/wget-1.14/lib/tempname.c"
  },

===

I could run that command without any trouble. adding -v option, I have

===
clang version 3.4 (trunk 192426)
Target: x86_64-unknown-linux-gnu
Thread model: posix
Found candidate GCC installation: /usr/lib/gcc/i686-linux-gnu/4.6
Found candidate GCC installation: /usr/lib/gcc/i686-linux-gnu/4.6.3
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/4.6
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/4.6.3
Selected GCC installation: /usr/lib/gcc/x86_64-linux-gnu/4.6
 "/home/jcwu/repos/llvm-release/Release/bin/clang" -cc1 -triple x86_64-unknown-linux-gnu -emit-obj -disable-free -disable-llvm-verifier -main-file-name tempname.c -mrelocation-model static -fmath-errno -masm-verbose -mconstructor-aliases -munwind-tables -target-cpu x86-64 -target-linker-version 2.22 -momit-leaf-frame-pointer -v -coverage-file /home/jcwu/repos/wget-1.14/lib/tempname.o -resource-dir /home/jcwu/repos/llvm-release/Release/bin/../lib/clang/3.4 -dependency-file .deps/tempname.Tpo -sys-header-deps -MP -MT tempname.o -D HAVE_CONFIG_H -I . -I ../src -I /usr/local/include -I /home/jcwu/repos/llvm-release/Release/lib/clang/3.4/include -I /usr/include/x86_64-linux-gnu -I /usr/include -cxx-isystem /usr/local/include -cxx-isystem /home/jcwu/repos/llvm-release/Release/lib/clang/3.4/include -cxx-isystem /usr/include/x86_64-linux-gnu -cxx-isystem /usr/include -cxx-isystem /usr/src/linux-headers-3.5.0-42-generic/include/config/i2o/ -internal-isystem /usr/local/include -internal-isystem /home/jcwu/repos/llvm-release/Release/bin/../lib/clang/3.4/include -internal-externc-isystem /usr/include/x86_64-linux-gnu -internal-externc-isystem /include -internal-externc-isystem /usr/include -O2 -Wall -fdebug-compilation-dir /home/jcwu/repos/wget-1.14/lib -ferror-limit 19 -fmessage-length 205 -mstackrealign -fobjc-runtime=gcc -fdiagnostics-show-option -fcolor-diagnostics -vectorize-loops -vectorize-slp -o tempname.o -x c tempname.c
clang -cc1 version 3.4 based upon LLVM 3.4svn default target x86_64-unknown-linux-gnu
ignoring nonexistent directory "/include"
ignoring duplicate directory "/usr/local/include"
  as it is a non-system directory that duplicates a system directory
ignoring duplicate directory "/home/jcwu/repos/llvm-release/Release/lib/clang/3.4/include"
  as it is a non-system directory that duplicates a system directory
ignoring duplicate directory "/usr/include/x86_64-linux-gnu"
  as it is a non-system directory that duplicates a system directory
ignoring duplicate directory "/usr/include"
  as it is a non-system directory that duplicates a system directory
#include "..." search starts here:
#include <...> search starts here:
 .
 ../src
 /usr/local/include
 /home/jcwu/repos/llvm-release/Release/lib/clang/3.4/include
 /usr/include/x86_64-linux-gnu
 /usr/include
End of search list.
===

searching for a while and I could find out the <config.h> is actually ../src/config.h.
However, it is already specified in -I../src, I don't know why clang libtooling cannot find it.
Since it's complaing "cannot open file './config.h'", I also tried to remove -I., but obviously it's not the case.
If I change the command to the one generated by -v option, the error messages become

===
error: unknown argument: '-cc1'
error: unknown argument: '-triple'
error: unknown argument: '-main-file-name'
error: unknown argument: '-mrelocation-model'
error: unknown argument: '-masm-verbose'
error: unknown argument: '-mconstructor-aliases'
error: unknown argument: '-munwind-tables'
error: unknown argument: '-target-cpu'
error: unknown argument: '-target-linker-version'
error: unknown argument: '-coverage-file'
error: unknown argument: '-sys-header-deps'
error: unknown argument: '-fdebug-compilation-dir'
error: unknown argument: '-ferror-limit'
error: unknown argument: '-vectorize-slp'
clang version 3.4 (trunk 192426)
Target: x86_64-unknown-linux-gnu
Thread model: posix
Found candidate GCC installation: /usr/lib/gcc/i686-linux-gnu/4.6
Found candidate GCC installation: /usr/lib/gcc/i686-linux-gnu/4.6.3
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/4.6
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/4.6.3
Selected GCC installation: /usr/lib/gcc/x86_64-linux-gnu/4.6
warning: x86_64-unknown-linux-gnu: 'linker' input unused
warning: static: 'linker' input unused
warning: x86-64: 'linker' input unused
warning: 2.22: 'linker' input unused
warning: /home/jcwu/repos/wget-1.14/lib/tempname.o: 'linker' input unused
warning: /usr/local/include: 'linker' input unused
warning: /home/jcwu/repos/llvm-release/Release/bin/../lib/clang/3.4/include: 'linker' input unused
warning: /usr/include/x86_64-linux-gnu: 'linker' input unused
warning: /include: 'linker' input unused
warning: /usr/include: 'linker' input unused
warning: /home/jcwu/repos/wget-1.14/lib: 'linker' input unused
warning: 19: 'linker' input unused
warning: 205: 'linker' input unused
error: unable to handle compilation, expected exactly one compiler job in ' "/home/jcwu/repos/clang.libtooling.annotator/main" "-cc1" "-triple" "x86_64-unknown-linux-gnu" "-fsyntax-only" "-disable-free" "-disable-llvm-verifier" "-main-file-name" "tempname.c" "-mrelocation-model" "static" "-fmath-errno" "-masm-verbose" "-mconstructor-aliases" "-munwind-tables" "-target-cpu" "x86-64" "-target-linker-version" "2.22" "-momit-leaf-frame-pointer" "-v" "-resource-dir" "/home/jcwu/repos/llvm-release/Release/bin/../lib/clang/3.4" "-MP" "-MT" "tempname.o" "-D" "HAVE_CONFIG_H" "-I" "." "-I" "../src" "-I" "/usr/local/include" "-I" "/home/jcwu/repos/llvm-release/Release/lib/clang/3.4/include" "-I" "/usr/include/x86_64-linux-gnu" "-I" "/usr/include" "-internal-isystem" "/usr/local/include" "-internal-isystem" "/home/jcwu/repos/llvm-release/Release/bin/../lib/clang/3.4/include" "-internal-externc-isystem" "/usr/include/x86_64-linux-gnu" "-internal-externc-isystem" "/include" "-internal-externc-isystem" "/usr/include" "-O2" "-Wall" "-fdebug-compilation-dir" "/home/jcwu/repos/wget-1.14/lib" "-ferror-limit" "19" "-fmessage-length" "205" "-backend-option" "-force-align-stack" "-mstackrealign" "-fobjc-runtime=gcc" "-fdiagnostics-show-option" "-fcolor-diagnostics" "-vectorize-loops" "-vectorize-slp" "-x" "c" "tempname.c";  "/home/jcwu/repos/clang.libtooling.annotator/main" "-cc1" "-triple" "x86_64-unknown-linux-gnu" "-fsyntax-only" "-disable-free" "-disable-llvm-verifier" "-main-file-name" "tempname.c" "-mrelocation-model" "static" "-fmath-errno" "-masm-verbose" "-mconstructor-aliases" "-munwind-tables" "-target-cpu" "x86-64" "-target-linker-version" "2.22" "-momit-leaf-frame-pointer" "-v" "-resource-dir" "/home/jcwu/repos/llvm-release/Release/bin/../lib/clang/3.4" "-MP" "-MT" "tempname.o" "-D" "HAVE_CONFIG_H" "-I" "." "-I" "../src" "-I" "/usr/local/include" "-I" "/home/jcwu/repos/llvm-release/Release/lib/clang/3.4/include" "-I" "/usr/include/x86_64-linux-gnu" "-I" "/usr/include" "-internal-isystem" "/usr/local/include" "-internal-isystem" "/home/jcwu/repos/llvm-release/Release/bin/../lib/clang/3.4/include" "-internal-externc-isystem" "/usr/include/x86_64-linux-gnu" "-internal-externc-isystem" "/include" "-internal-externc-isystem" "/usr/include" "-O2" "-Wall" "-fdebug-compilation-dir" "/home/jcwu/repos/wget-1.14/lib" "-ferror-limit" "19" "-fmessage-length" "205" "-backend-option" "-force-align-stack" "-mstackrealign" "-fobjc-runtime=gcc" "-fdiagnostics-show-option" "-fcolor-diagnostics" "-vectorize-loops" "-vectorize-slp" "-x" "c" "tempname.c"; '
Error while processing /home/jcwu/repos/wget-1.14/lib/tempname.c.
===

Maybe it's not the correct way to use it in compile_commands.json
Reply | Threaded
Open this post in threaded view
|

Re: How to use libtooling to parse multiple files at once? and succesfully find stddef.h?

rayjcwu
In reply to this post by rayjcwu
in short:  If I change compile_commands.json, leave only those files causing errors while processing, my program could successfully parse all files listed in compile_commands.json without any error.

I compared compiling arguments for clang drivers from make (verbose mode) and compile_commands.json but didn't find anything. If I run ClangTool to run the FrontendAction, it will have a lot of error messages and core dump in the end. Error messages are complaing about can't find some headers within this project. But if I run those commands in compile_commands.json in shell, everything works fine.

I'm using llvm/clang r194059 on ubuntu 12.04 trying to parse apache 2.4.7. Generate compile_commands.json by https://github.com/rizsotto/Bear. Error messages are like

==============================================================================
Error while processing /home/jcwu/repos/httpd-2.4.7/support/htcacheclean.c.
atomic/unix/builtins.c:71:53: warning: passing 'const void *' to parameter of type 'volatile void *' discards qualifiers [-Wincompatible-pointer-types-discards-qualifiers]
    return (void*) __sync_val_compare_and_swap(mem, cmp, with);
                                                    ^~~
1 warning generated.
atomic/unix/builtins.c:71:53: warning: passing 'const void *' to parameter of type 'volatile void *' discards qualifiers [-Wincompatible-pointer-types-discards-qualifiers]
    return (void*) __sync_val_compare_and_swap(mem, cmp, with);
                                                    ^~~
1 warning generated.
repos.c:35:10: fatal error: cannot open file './mod_dav.h': No such file or directory
#include "mod_dav.h"
         ^
1 error generated.
Error while processing /home/jcwu/repos/httpd-2.4.7/modules/dav/fs/repos.c.
repos.c:35:10: fatal error: cannot open file './mod_dav.h': No such file or directory
#include "mod_dav.h"
         ^
1 error generated.
Error while processing /home/jcwu/repos/httpd-2.4.7/modules/dav/fs/repos.c.
lock.c:32:10: fatal error: cannot open file './mod_dav.h': No such file or directory
#include "mod_dav.h"
         ^
1 error generated.
Error while processing /home/jcwu/repos/httpd-2.4.7/modules/dav/fs/lock.c.
lock.c:32:10: fatal error: cannot open file './mod_dav.h': No such file or directory
#include "mod_dav.h"
         ^
1 error generated.
Error while processing /home/jcwu/repos/httpd-2.4.7/modules/dav/fs/lock.c.
clang-annotator: /home/jcwu/repos/llvm-r194059-src/lib/Support/MemoryBuffer.cpp:57: void llvm::MemoryBuffer::init(const char*, const char*, bool): Assertion `(!RequiresNullTerminator || BufEnd[0] == 0) && "Buffer is not null terminated!"' failed.
Aborted (core dumped)
==============================================================================

But the visitor didn't do anything meaningful. Even if I use an empty visitor without override any functions, the error still exist.

The strangest thing is, if I change compile_commands.json, leave only those files causing errors while processing them, my program could successfully parse all files listed in compile_commands.json without any error.

My code is at https://github.com/rayjcwu/clang-annotator/blob/master/parser.cpp and the pseudo code is roughtly like the following: (I turn on exception supports because I need to use a database library using exceptions)

// A dummy visitor doing nothing, omitted

void ExampleASTConsumer::HandleTranslationUnit(ASTContext &Context) {
  visitor->TraverseDecl(Context.getTranslationUnitDecl());
}

ASTConsumer *ExampleFrontendAction::CreateASTConsumer(CompilerInstance &CI, StringRef file) {
  return new ExampleASTConsumer(&CI); // pass CI pointer to ASTConsumer
}

class ClangParser {
private:
    CompilationDatabase *CD;
public:
   ClangParser(string srcPath) {
      string errorMsg = "";
      CD = CompilationDatabase::autoDetectFromDirectory (srcPath.c_str(), errorMsg);
      if (errorMsg != "") {
         cerr << errorMsg << endl;
         throw invalid_argument(errorMsg);
      }

   void run() {
      ClangTool Tool(*CD, CD->getAllFiles());
      Tool.run(newFrontendActionFactory<ExampleFrontendAction>());
   }
}