index library

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

index library

Andrew Sutton
I spent yesterday morning hacking out a Python wrapper for the CIndex library (using ctypes). It seemed like a quick and easy way to pull data from C/C++ files... It's very much a first attempt. Probably not great, but it seems to work so far. I can make the code available if anybody's interested. Anyhow, I ran into a couple of problems that I couldn't quite resolve on my own.

When I tried to compile a simple program with some templates (clang -emit-ast), clang fails with an 'Unexepcted DeclType', which I've taken to mean that PCHWriter (and probably PCHReader) don't support the necessary visitors for TemplateDecl or Decl's with associated TemplateDecls.

I was going to take a shot at encoding templates into PCH's when I realized that I have absolutely no idea what is actually supposed to go into a PCH or AST file. In fact, it didn't like there was a lot support for C++ Decls in the PCHWriter/Reader. I was just curious what the plans are for these AST objects. Are there any? What needs to be supported?

Andrew Sutton
[hidden email]

_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: index library

John McCall

On Jan 5, 2010, at 5:51 PM, Andrew Sutton wrote:

> I spent yesterday morning hacking out a Python wrapper for the CIndex library (using ctypes). It seemed like a quick and easy way to pull data from C/C++ files... It's very much a first attempt. Probably not great, but it seems to work so far. I can make the code available if anybody's interested. Anyhow, I ran into a couple of problems that I couldn't quite resolve on my own.
>
> When I tried to compile a simple program with some templates (clang -emit-ast), clang fails with an 'Unexepcted DeclType', which I've taken to mean that PCHWriter (and probably PCHReader) don't support the necessary visitors for TemplateDecl or Decl's with associated TemplateDecls.
>
> I was going to take a shot at encoding templates into PCH's when I realized that I have absolutely no idea what is actually supposed to go into a PCH or AST file. In fact, it didn't like there was a lot support for C++ Decls in the PCHWriter/Reader. I was just curious what the plans are for these AST objects. Are there any? What needs to be supported?

You're absolutely right that there is minimal support for C++ AST objects in the PCH routines right now.  It's a known problem which blocks a lot of interesting feature work, and we'll definitely need to fix it sooner rather than later;  unfortunately, at the moment most of us are occupied on other tasks, like getting clang to self-host.  If you're interested in working on it, that would be fantastic.  Most C++ nodes have settled down enough for PCH support to be usefully added in parallel.

The PCH record for an AST node stores everything that you'd see in the corresponding class.  Most of the work is mechanical:  PCHWriter adds entries to a record, and PCHReader pulls those entries back out in the same order.  If we could do this with reflective meta-programming, we would. :)

I believe that we are not currently interested in maintaining PCH forward/backward compatibility.

John.
_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: index library

Andrew Sutton

Please continue sending these to cfe-dev rather than only me.  I will definitely look at them in either case, but it's good to be open.

Sorry about that. I wasn't paying attention to the "Reply-To". I'm responding to the list and can re-post the patches later.
 
Looking at the PCH patch, your general approach seems good, but I'm not sure why you've introduced PCHReader/Writer::VisitFunctionBase rather than just calling VisitFunctionDecl from all the subclass methods.  Is there some reason I'm not recalling that the param references have to come last in the record?

I wasn't sure if ordering of records mattered or not. I was trying to preserve the same basic ordering of records for Method and kin, and not set the Code member twice. In the end, it probably won't matter too much.

I'll experiment a little and roll back that change if it's not a big deal.

+void PCHDeclWriter::visitFunctionTemplateDecl(FunctionTemplateDecl *D) {
+  VisitTemplateDecl(D);
+  FunctionDecl* Func = D->getTemplatedDecl();
+  if (CXXConstructorDecl* Ctor = dyn_cast<CXXConstructorDecl>(Func)) {
+    VisitCXXConstructorDecl(Ctor);
+  } else {
+    VisitFunctionDecl(Func);
+  }
+  Code = pch::DECL_FUNCTION_TEMPLATE;
+}

The easier and faster way to do this is to call Visit(D->getTemplatedDecl()).

Good point :) I actually forgot that I'd written it that way.
 
Eventually it'll need to support anything you can get in C, ObjC, C++, or ObjC++.  That is, it will need to support everything that clang can produce.
Everything which is a DeclContext needs a cursor, but not necessarily its own cursor type.  CXXRecordDecls need their own cursor type because of the base clauses;  CXXConstructorDecls need their own cursor type because of the initializers;  everything else should be able to use Function.  But I'm not totally sure;  I've done relatively little work in here.

I'll have to spend some time digesting that information.

Andrew Sutton
[hidden email]

_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: index library

John McCall
On Jan 14, 2010, at 11:05 AM, Andrew Sutton wrote:
Looking at the PCH patch, your general approach seems good, but I'm not sure why you've introduced PCHReader/Writer::VisitFunctionBase rather than just calling VisitFunctionDecl from all the subclass methods.  Is there some reason I'm not recalling that the param references have to come last in the record?

I wasn't sure if ordering of records mattered or not. I was trying to preserve the same basic ordering of records for Method and kin, and not set the Code member twice. In the end, it probably won't matter too much.

If you mean ordering of data within a record, it's better to have more redundancy between record kinds, because we do support compact representations.  That probably won't matter much here.  Worry about having clean code first;  we don't really understand where the significant overheads are in PCH processing, but they're almost certainly not in redundantly setting the Code member. :)

John.
_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: index library

Daniel Dunbar
In reply to this post by Andrew Sutton
On Tue, Jan 5, 2010 at 5:51 PM, Andrew Sutton <[hidden email]> wrote:
> I spent yesterday morning hacking out a Python wrapper for the CIndex
> library (using ctypes). It seemed like a quick and easy way to pull data
> from C/C++ files... It's very much a first attempt. Probably not great, but
> it seems to work so far. I can make the code available if anybody's
> interested. Anyhow, I ran into a couple of problems that I couldn't quite
> resolve on my own.

If you feel like its good enough to be useful to others, please submit
it. I would love for us to have Python bindings support in-tree.

 - Daniel

> When I tried to compile a simple program with some templates (clang
> -emit-ast), clang fails with an 'Unexepcted DeclType', which I've taken to
> mean that PCHWriter (and probably PCHReader) don't support the necessary
> visitors for TemplateDecl or Decl's with associated TemplateDecls.
>
> I was going to take a shot at encoding templates into PCH's when I realized
> that I have absolutely no idea what is actually supposed to go into a PCH or
> AST file. In fact, it didn't like there was a lot support for C++ Decls in
> the PCHWriter/Reader. I was just curious what the plans are for these AST
> objects. Are there any? What needs to be supported?
>
> Andrew Sutton
> [hidden email]
>
> _______________________________________________
> cfe-dev mailing list
> [hidden email]
> http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
>
>
_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: index library

Andrew Sutton

If you feel like its good enough to be useful to others, please submit
it. I would love for us to have Python bindings support in-tree.

 It's not :) Not yet, anyways.

So far I've started by wrapping libCIndex using Python's ctypes because its probably the most direct route. Unfortunately, there are some issues translating back and forth between the void*'s and Python objects that make the writing visitors a little unnatural. If I can work around that in Python, I'll have something sooner. If not, I'll probably have to write a C library that does this.

More later, I'm sure.

Andrew Sutton
[hidden email]

_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev