issue with libclang and Python bindings

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

issue with libclang and Python bindings

David Chisnall via cfe-dev

Hello,

 

I’m using libclang and its Python bindings to build a binding generator to a large C++ library using pybind11. All is going well but I am experiencing a strange issue and I’m hoping someone on this list has an idea. Unfortunately, I’m not able to reproduce the issue on a small scale so I’ll just describe it as best I can and if anyone has any thoughts please share.

 

I have a single include file (“all_includes.h”) that includes all the headers of the C++ library. This file is parsed by libclang and I use the results to write pybind11 code via the libclang Python bindings. I don’t get any translation errors or warnings. In some cases, certain types (usually parameter declarations) get resolved to “int” instead of the actual type like “std::list<TypeA>”.

 

If my “all_inlcudes.h” has only the header file where the issue occurs, the types are resolved correctly. It’s only when I include all my headers (or a large number of them) does this strange behavior start to appear. I can’t figure out a root reason or pattern for it happening, the only thing I’ve noticed is it seems to happen with things in a namespace liked “std::list<>” or “TypeA::value”.

 

One other thing, the C++ library makes heavy use of class templates. In libclang, these class declarations have no children (i.e., constructors, methods, etc.). In order to generate the binding code I have to retrieve the cursor to the class template, write its code then replace the template parameters with a string replace method. Is there a reason these class declarations from class templates don’t resolve the children of the class template with the template parameters already in place? It would make the binding generation process much easier and robust.

 

For either of these issues, my next step is to rewrite my binding generation tool in C++ and use the clang libraries like LibTooling. They seem to provide more functionality anyway, but the Python version is so close I’d thought it’d be worth asking before I go down that path.

 

Thanks,

Trevor

 


_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: issue with libclang and Python bindings

David Chisnall via cfe-dev

On Sun, Nov 5, 2017 at 2:39 PM, Trevor Laughlin via cfe-dev <[hidden email]> wrote:

Hello,

 

I’m using libclang and its Python bindings to build a binding generator to a large C++ library using pybind11. All is going well but I am experiencing a strange issue and I’m hoping someone on this list has an idea. Unfortunately, I’m not able to reproduce the issue on a small scale so I’ll just describe it as best I can and if anyone has any thoughts please share.

 

I have a single include file (“all_includes.h”) that includes all the headers of the C++ library. This file is parsed by libclang and I use the results to write pybind11 code via the libclang Python bindings. I don’t get any translation errors or warnings. In some cases, certain types (usually parameter declarations) get resolved to “int” instead of the actual type like “std::list<TypeA>”.

 


What kind of behavior do you get when you use a command-line clang or other C++ compiler with a source file that includes "all_includes.h"?

I don't know much about libclang's operation but here's a wild guess: are you getting implicit int parameters for undeclared functions?  If so it would seem that libclang might think this is C source and not C++.  Going out on a big limb: maybe this is also why you have trouble with types using the scope resolution operator?  Are you giving a header file to libclang as if it were a translation unit for compilation?  If that's the case then does the behavior change if you call it "all_includes.hpp" or "all_includes.H"?  Have you tried explicitly indicating the language w/" -x "?   See https://clang.llvm.org/docs/ClangCommandLineReference.html#introduction towards the end.
   

If my “all_inlcudes.h” has only the header file where the issue occurs, the types are resolved correctly. It’s only when I include all my headers (or a large number of them) does this strange behavior start to appear. I can’t figure out a root reason or pattern for it happening, the only thing I’ve noticed is it seems to happen with things in a namespace liked “std::list<>” or “TypeA::value”.

 


Is it possible that these are from preprocessor macros and they're getting omitted when some critical definitions are absent/present?

 

One other thing, the C++ library makes heavy use of class templates. In libclang, these class declarations have no children (i.e., constructors, methods, etc.). In order to generate the binding code I have to retrieve the cursor to the class template, write its code then replace the template parameters with a string replace method. Is there a reason these class declarations from class templates don’t resolve the children of the class template with the template parameters already in place? It would make the binding generation process much easier and robust.

 

For either of these issues, my next step is to rewrite my binding generation tool in C++ and use the clang libraries like LibTooling. They seem to provide more functionality anyway, but the Python version is so close I’d thought it’d be worth asking before I go down that path.

 



I would try to bisect the differences between your libclang usage and some known-good compiler configuration.  Are the definitions the same?  "clang++ -dM -E -x c++ /dev/null > cmdline_defs"  -- Is the preprocessed output the same?  "clang++ -E all_includes.h > all_includes.ii"



_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev