ClangD

classic Classic list List threaded Threaded
17 messages Options
Reply | Threaded
Open this post in threaded view
|

ClangD

Roman Popov via cfe-dev
Hi fellow clang devs,

we wanted to let you know that we're (finally) starting up work on ClangD, which you might know from email threads such as [1] (June 2012!).

In the meantime, YCM had done a good enough job at packaging up a libclang interface to our favorite editors, and other priorities (like modules) have eaten up a lot of folks priority lunches.

What has changed?
1. YCM is starting to develop more and more into a language multiplexer, with other languages (Go, Typescript, etc) providing their own servers to talk to
2. MS has created a language server protocol [2], which already has both a bunch of client and server implementations
3. Debugging through python into libclang crashers is a pain and eating our support resources
4. While libclang is a good abstraction if you want to have something run in your process with close coupling, a standard protocol like the language server protocol seems like a better way to allow fast iterations on the server implementation without the need to keep backward-compatibility hacks through a restrictive C API.

One of the cool things about the MS language server protocol is that it seems to not actually do any networking, which means that we do not need to introduce any new dependencies into clang-tools-extra, which is where we want to start developing this.

If you have any thoughts / concerns let me know; otherwise look forward to code reviews on initial ClangD dropping by :D

Cheers,
/Manuel



_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: ClangD

Roman Popov via cfe-dev
Good news, good luck!

2017-01-25 14:11 GMT+01:00 Manuel Klimek via cfe-dev <[hidden email]>:
Hi fellow clang devs,

we wanted to let you know that we're (finally) starting up work on ClangD, which you might know from email threads such as [1] (June 2012!).

In the meantime, YCM had done a good enough job at packaging up a libclang interface to our favorite editors, and other priorities (like modules) have eaten up a lot of folks priority lunches.

What has changed?
1. YCM is starting to develop more and more into a language multiplexer, with other languages (Go, Typescript, etc) providing their own servers to talk to
2. MS has created a language server protocol [2], which already has both a bunch of client and server implementations
3. Debugging through python into libclang crashers is a pain and eating our support resources
4. While libclang is a good abstraction if you want to have something run in your process with close coupling, a standard protocol like the language server protocol seems like a better way to allow fast iterations on the server implementation without the need to keep backward-compatibility hacks through a restrictive C API.

One of the cool things about the MS language server protocol is that it seems to not actually do any networking, which means that we do not need to introduce any new dependencies into clang-tools-extra, which is where we want to start developing this.

If you have any thoughts / concerns let me know; otherwise look forward to code reviews on initial ClangD dropping by :D

Cheers,
/Manuel



_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev



_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: ClangD

Roman Popov via cfe-dev
In reply to this post by Roman Popov via cfe-dev
Thanks Manuel, I'm quite excited to hear that about ClangD!

I have a couple of questions:
Do you think ClangD would be able to replace libclang in the future? Would it be possible to deprecate libclang for IDE use after ClangD catches up to it, or should we keep libclang as it is even after that point?

On 25 January 2017 at 13:11, Manuel Klimek via cfe-dev <[hidden email]> wrote:
Hi fellow clang devs,

we wanted to let you know that we're (finally) starting up work on ClangD, which you might know from email threads such as [1] (June 2012!).

In the meantime, YCM had done a good enough job at packaging up a libclang interface to our favorite editors, and other priorities (like modules) have eaten up a lot of folks priority lunches.

What has changed?
1. YCM is starting to develop more and more into a language multiplexer, with other languages (Go, Typescript, etc) providing their own servers to talk to
2. MS has created a language server protocol [2], which already has both a bunch of client and server implementations
3. Debugging through python into libclang crashers is a pain and eating our support resources
4. While libclang is a good abstraction if you want to have something run in your process with close coupling, a standard protocol like the language server protocol seems like a better way to allow fast iterations on the server implementation without the need to keep backward-compatibility hacks through a restrictive C API.

One of the cool things about the MS language server protocol is that it seems to not actually do any networking, which means that we do not need to introduce any new dependencies into clang-tools-extra, which is where we want to start developing this.

If you have any thoughts / concerns let me know; otherwise look forward to code reviews on initial ClangD dropping by :D

Cheers,
/Manuel



_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev



_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: ClangD

Roman Popov via cfe-dev
On Wed, Jan 25, 2017 at 3:08 PM Alex L <[hidden email]> wrote:
Thanks Manuel, I'm quite excited to hear that about ClangD!

I have a couple of questions:
Do you think ClangD would be able to replace libclang in the future? Would it be possible to deprecate libclang for IDE use after ClangD catches up to it, or should we keep libclang as it is even after that point?
 
In the foreseeable future, I don't see libclang going away. Not everybody is able to switch to a new development workflow easily, and there is a lot of investment in the current libclang based workflows.
 
On 25 January 2017 at 13:11, Manuel Klimek via cfe-dev <[hidden email]> wrote:
Hi fellow clang devs,

we wanted to let you know that we're (finally) starting up work on ClangD, which you might know from email threads such as [1] (June 2012!).

In the meantime, YCM had done a good enough job at packaging up a libclang interface to our favorite editors, and other priorities (like modules) have eaten up a lot of folks priority lunches.

What has changed?
1. YCM is starting to develop more and more into a language multiplexer, with other languages (Go, Typescript, etc) providing their own servers to talk to
2. MS has created a language server protocol [2], which already has both a bunch of client and server implementations
3. Debugging through python into libclang crashers is a pain and eating our support resources
4. While libclang is a good abstraction if you want to have something run in your process with close coupling, a standard protocol like the language server protocol seems like a better way to allow fast iterations on the server implementation without the need to keep backward-compatibility hacks through a restrictive C API.

One of the cool things about the MS language server protocol is that it seems to not actually do any networking, which means that we do not need to introduce any new dependencies into clang-tools-extra, which is where we want to start developing this.

If you have any thoughts / concerns let me know; otherwise look forward to code reviews on initial ClangD dropping by :D

Cheers,
/Manuel



_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev


_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: ClangD

Roman Popov via cfe-dev
In reply to this post by Roman Popov via cfe-dev
Hi,

I think this is an excellent idea which should be pursued as far as possible as it will enable us to seamlessly integrate multiple clang-based services into the development environment. Possibly it will enable us to integrate non-clang-based kind of services as well. I might say that I've developed a (prototype) solution which somewhat resembles this idea (implementation goes up to extent of my current needs & time resources). See [1] for a short overview but in general it is a generic, message-queue based, client-server architecture which acts as a thin proxy layer between the client requests and concrete service implementation (i.e. syntax-highlighting, clang-format, etc.). Server transparently makes sure that each service is run in its own context (thread/process) and provides full-control of services (startup-(all), shutdown-(all), run, etc.): see [2] for server implementation. This way a service developer can completely put focus on service implementation details itself without caring about other details. See [3] for base service implementation and [4] & [5] for examples of concrete service implementations (clang-format service & syntax-highlight service).  

What would be a way to get involved with ClangD?

Cheers,
Adi



On 25 January 2017 at 14:11, Manuel Klimek via cfe-dev <[hidden email]> wrote:
Hi fellow clang devs,

we wanted to let you know that we're (finally) starting up work on ClangD, which you might know from email threads such as [1] (June 2012!).

In the meantime, YCM had done a good enough job at packaging up a libclang interface to our favorite editors, and other priorities (like modules) have eaten up a lot of folks priority lunches.

What has changed?
1. YCM is starting to develop more and more into a language multiplexer, with other languages (Go, Typescript, etc) providing their own servers to talk to
2. MS has created a language server protocol [2], which already has both a bunch of client and server implementations
3. Debugging through python into libclang crashers is a pain and eating our support resources
4. While libclang is a good abstraction if you want to have something run in your process with close coupling, a standard protocol like the language server protocol seems like a better way to allow fast iterations on the server implementation without the need to keep backward-compatibility hacks through a restrictive C API.

One of the cool things about the MS language server protocol is that it seems to not actually do any networking, which means that we do not need to introduce any new dependencies into clang-tools-extra, which is where we want to start developing this.

If you have any thoughts / concerns let me know; otherwise look forward to code reviews on initial ClangD dropping by :D

Cheers,
/Manuel



_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev



_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: ClangD

Roman Popov via cfe-dev
On Wed, Jan 25, 2017 at 3:34 PM Jusufadis Bakamovic <[hidden email]> wrote:
Hi,

I think this is an excellent idea which should be pursued as far as possible as it will enable us to seamlessly integrate multiple clang-based services into the development environment. Possibly it will enable us to integrate non-clang-based kind of services as well. I might say that I've developed a (prototype) solution which somewhat resembles this idea (implementation goes up to extent of my current needs & time resources). See [1] for a short overview but in general it is a generic, message-queue based, client-server architecture which acts as a thin proxy layer between the client requests and concrete service implementation (i.e. syntax-highlighting, clang-format, etc.). Server transparently makes sure that each service is run in its own context (thread/process) and provides full-control of services (startup-(all), shutdown-(all), run, etc.): see [2] for server implementation. This way a service developer can completely put focus on service implementation details itself without caring about other details. See [3] for base service implementation and [4] & [5] for examples of concrete service implementations (clang-format service & syntax-highlight service).  

What would be a way to get involved with ClangD?

Lurk on the cfe-commits mailing list for clangd based code reviews to come in (Benjamin is working on getting an initial prototype up), and contribute there.

Once we have more of the basic infrastructure in place, we'll accept patches like we do for the rest of clang and have a bug list.
 

On 25 January 2017 at 14:11, Manuel Klimek via cfe-dev <[hidden email]> wrote:
Hi fellow clang devs,

we wanted to let you know that we're (finally) starting up work on ClangD, which you might know from email threads such as [1] (June 2012!).

In the meantime, YCM had done a good enough job at packaging up a libclang interface to our favorite editors, and other priorities (like modules) have eaten up a lot of folks priority lunches.

What has changed?
1. YCM is starting to develop more and more into a language multiplexer, with other languages (Go, Typescript, etc) providing their own servers to talk to
2. MS has created a language server protocol [2], which already has both a bunch of client and server implementations
3. Debugging through python into libclang crashers is a pain and eating our support resources
4. While libclang is a good abstraction if you want to have something run in your process with close coupling, a standard protocol like the language server protocol seems like a better way to allow fast iterations on the server implementation without the need to keep backward-compatibility hacks through a restrictive C API.

One of the cool things about the MS language server protocol is that it seems to not actually do any networking, which means that we do not need to introduce any new dependencies into clang-tools-extra, which is where we want to start developing this.

If you have any thoughts / concerns let me know; otherwise look forward to code reviews on initial ClangD dropping by :D

Cheers,
/Manuel



_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev


_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: ClangD

Roman Popov via cfe-dev
From the YCM side: I think this is a fantastic idea! We were thinking about pulling out the C++ completion out of ycmd and into a separate server that ycmd would talk to (much like it works for several other languages already) but hearing that clang devs are taking this on saves us a bunch of work!

And it's great to hear it will follow the common completion API MS is pushing; we'll be supporting that API on the back-end of ycmd some time in the future (and clangd might accelerate that). We're looking into possibly supporting it on the front-end as well.

On Wed, Jan 25, 2017 at 6:38 AM, Manuel Klimek <[hidden email]> wrote:
On Wed, Jan 25, 2017 at 3:34 PM Jusufadis Bakamovic <[hidden email]> wrote:
Hi,

I think this is an excellent idea which should be pursued as far as possible as it will enable us to seamlessly integrate multiple clang-based services into the development environment. Possibly it will enable us to integrate non-clang-based kind of services as well. I might say that I've developed a (prototype) solution which somewhat resembles this idea (implementation goes up to extent of my current needs & time resources). See [1] for a short overview but in general it is a generic, message-queue based, client-server architecture which acts as a thin proxy layer between the client requests and concrete service implementation (i.e. syntax-highlighting, clang-format, etc.). Server transparently makes sure that each service is run in its own context (thread/process) and provides full-control of services (startup-(all), shutdown-(all), run, etc.): see [2] for server implementation. This way a service developer can completely put focus on service implementation details itself without caring about other details. See [3] for base service implementation and [4] & [5] for examples of concrete service implementations (clang-format service & syntax-highlight service).  

What would be a way to get involved with ClangD?

Lurk on the cfe-commits mailing list for clangd based code reviews to come in (Benjamin is working on getting an initial prototype up), and contribute there.

Once we have more of the basic infrastructure in place, we'll accept patches like we do for the rest of clang and have a bug list.
 

On 25 January 2017 at 14:11, Manuel Klimek via cfe-dev <[hidden email]> wrote:
Hi fellow clang devs,

we wanted to let you know that we're (finally) starting up work on ClangD, which you might know from email threads such as [1] (June 2012!).

In the meantime, YCM had done a good enough job at packaging up a libclang interface to our favorite editors, and other priorities (like modules) have eaten up a lot of folks priority lunches.

What has changed?
1. YCM is starting to develop more and more into a language multiplexer, with other languages (Go, Typescript, etc) providing their own servers to talk to
2. MS has created a language server protocol [2], which already has both a bunch of client and server implementations
3. Debugging through python into libclang crashers is a pain and eating our support resources
4. While libclang is a good abstraction if you want to have something run in your process with close coupling, a standard protocol like the language server protocol seems like a better way to allow fast iterations on the server implementation without the need to keep backward-compatibility hacks through a restrictive C API.

One of the cool things about the MS language server protocol is that it seems to not actually do any networking, which means that we do not need to introduce any new dependencies into clang-tools-extra, which is where we want to start developing this.

If you have any thoughts / concerns let me know; otherwise look forward to code reviews on initial ClangD dropping by :D

Cheers,
/Manuel



_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev



_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: ClangD

Roman Popov via cfe-dev
In reply to this post by Roman Popov via cfe-dev
On 01/25/2017 02:11 PM, Manuel Klimek via cfe-dev wrote:

> Hi fellow clang devs,
>
> we wanted to let you know that we're (finally) starting up work on
> ClangD, which you might know from email threads such as [1] (June 2012!).
>
> In the meantime, YCM had done a good enough job at packaging up a
> libclang interface to our favorite editors, and other priorities (like
> modules) have eaten up a lot of folks priority lunches.
>
> What has changed?
> 1. YCM is starting to develop more and more into a language multiplexer,
> with other languages (Go, Typescript, etc) providing their own servers
> to talk to
> 2. MS has created a language server protocol [2], which already has both
> a bunch of client and server implementations
> 3. Debugging through python into libclang crashers is a pain and eating
> our support resources
> 4. While libclang is a good abstraction if you want to have something
> run in your process with close coupling, a standard protocol like the
> language server protocol seems like a better way to allow fast
> iterations on the server implementation without the need to keep
> backward-compatibility hacks through a restrictive C API.
>
> One of the cool things about the MS language server protocol is that it
> seems to not actually do any networking, which means that we do not need
> to introduce any new dependencies into clang-tools-extra, which is where
> we want to start developing this.
>
> If you have any thoughts / concerns let me know; otherwise look forward
> to code reviews on initial ClangD dropping by :D

[Huch, my first association with "ClangD" was support for the D language]

Hi!

We from the Qt Creator IDE team are also interested in this. :)

Some notes:

1. You've indirectly referenced the initial design document at [1]. Will
this get an update or is it already updated somewhere else? An updated
document clarifying the scope and goals would be probably useful to all
interested parties, also in regard to contribution and how to align own
current development that uses libclang/libtooling/clang tools.

E.g. Qt Creator also already launches a separate process
("clangbackend") that makes use of libclang to provide the basic
functionality like updated diagnostics, highlighting information and
results for completion requests.


2. How is clangd related to "clang-refactor" [2]?

If it's not related at all, what's the state of clang-refactor?


3. I take this opportunity to share our biggest problems with libclang
from the Qt Creator perspective since clangd users will probably face
them also (as far as I understand, clangd will be based upon libclang -
please correct me if I'm wrong). I share this in hope that they will be
addressed or taken into consideration during the development of clangd
for the benefit of all interested parties now that the development of
clangd is announced. Please keep in mind that the actual problems might
be in other components that libclang makes use of.

The problems we are facing with libclang, in this order, are:

3.1 Windows-specific issues

The whole performance of parse/reparse in general is worser than on
Mac/Linux (probably also because of slow IO on Windows). At the end of
the day, this makes it less usable on Windows for real-world
projects/files. There are also other Windows-specific issues.

My gut feeling is that there aren't that many libclang users on Windows.

@Manual: To which extend do you or your team use and develop on Windows?
Are any Windows developers working on clangd?

Concrete issues are:

  * Preamble is also generated for the second reparse on Windows
    https://llvm.org/bugs/show_bug.cgi?id=28535

  * Problems with FileManager caches leading to leaking file descriptors
    https://reviews.llvm.org/D27810

  * Very slow reparsing for big header and even slower with
    preamble-generation enabled (might be related to the one above)
    https://llvm.org/bugs/show_bug.cgi?id=31353

  * Windows: Crash with absolute path with forward slashes
    https://llvm.org/bugs/show_bug.cgi?id=28381

3.2 Synchronous/blocking API
Low-latency completion results at all times are impossible with a single
CXTranslationUnit that is just being parsed or reparsed. This gets worse
the longer the parse/reparse takes (hello Windows).

One can mitigate this by triggering the parse/reparse after a
significant timeout of user-typing-inactivity, but this leads to a
sluggish experience regarding new diagnostics and highlighting
informations (you don't want to wait more than 2s for new diagnostics to
see if your code is valid now).

Another approach is to use two CXTranslationUnits per file, so at least
there is always a non-blocking CXTranslationUnits ready for completion
requests. Downside is doubled memory consumption and potentially
threading issues (@Manual: do they still exist or are they already
resolved?).

3.3 libclang requires unsaved files to exist on disk.

If the unsaved file does not exist a preamble generation will happen for
every reparse which simply takes to long.

For the concrete use case, refer to

        https://llvm.org/bugs/show_bug.cgi?id=31112

This one might be also a problem if you do version control checkouts and
files suddenly vanish.



Best regards,
    Nikolai



[1] https://github.com/chandlerc/llvm-designs/blob/master/ClangService.rst
[2] http://lists.llvm.org/pipermail/cfe-dev/2016-September/050641.html

_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: ClangD

Roman Popov via cfe-dev
In reply to this post by Roman Popov via cfe-dev
On 01/25/2017 03:13 PM, Manuel Klimek via cfe-dev wrote:

> On Wed, Jan 25, 2017 at 3:08 PM Alex L <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     Thanks Manuel, I'm quite excited to hear that about ClangD!
>
>     I have a couple of questions:
>     Do you think ClangD would be able to replace libclang in the future?
>     Would it be possible to deprecate libclang for IDE use after ClangD
>     catches up to it, or should we keep libclang as it is even after
>     that point?
>
>
> In the foreseeable future, I don't see libclang going away. Not
> everybody is able to switch to a new development workflow easily, and
> there is a lot of investment in the current libclang based workflows.

Just for clarity, please elaborate on "lot of investment in the current
libclang based workflows".

>     On 25 January 2017 at 13:11, Manuel Klimek via cfe-dev
>     <[hidden email] <mailto:[hidden email]>> wrote:
>
>         Hi fellow clang devs,
>
>         we wanted to let you know that we're (finally) starting up work
>         on ClangD, which you might know from email threads such as [1]
>         (June 2012!).
>
>         In the meantime, YCM had done a good enough job at packaging up
>         a libclang interface to our favorite editors, and other
>         priorities (like modules) have eaten up a lot of folks priority
>         lunches.
>
>         What has changed?
>         1. YCM is starting to develop more and more into a language
>         multiplexer, with other languages (Go, Typescript, etc)
>         providing their own servers to talk to
>         2. MS has created a language server protocol [2], which already
>         has both a bunch of client and server implementations
>         3. Debugging through python into libclang crashers is a pain and
>         eating our support resources
>         4. While libclang is a good abstraction if you want to have
>         something run in your process with close coupling, a standard
>         protocol like the language server protocol seems like a better
>         way to allow fast iterations on the server implementation
>         without the need to keep backward-compatibility hacks through a
>         restrictive C API.
>
>         One of the cool things about the MS language server protocol is
>         that it seems to not actually do any networking, which means
>         that we do not need to introduce any new dependencies into
>         clang-tools-extra, which is where we want to start developing this.
>
>         If you have any thoughts / concerns let me know; otherwise look
>         forward to code reviews on initial ClangD dropping by :D
>
>         Cheers,
>         /Manuel
>
>         [1] http://lists.llvm.org/pipermail/cfe-dev/2012-June/022028.html
>         [2] https://github.com/Microsoft/language-server-protocol
>
>
>         _______________________________________________
>         cfe-dev mailing list
>         [hidden email] <mailto:[hidden email]>
>         http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>
>
>
> _______________________________________________
> cfe-dev mailing list
> [hidden email]
> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>

_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: ClangD

Roman Popov via cfe-dev
In reply to this post by Roman Popov via cfe-dev
On Thu, Jan 26, 2017 at 3:33 PM Nikolai Kosjar <[hidden email]> wrote:
On 01/25/2017 02:11 PM, Manuel Klimek via cfe-dev wrote:
> Hi fellow clang devs,
>
> we wanted to let you know that we're (finally) starting up work on
> ClangD, which you might know from email threads such as [1] (June 2012!).
>
> In the meantime, YCM had done a good enough job at packaging up a
> libclang interface to our favorite editors, and other priorities (like
> modules) have eaten up a lot of folks priority lunches.
>
> What has changed?
> 1. YCM is starting to develop more and more into a language multiplexer,
> with other languages (Go, Typescript, etc) providing their own servers
> to talk to
> 2. MS has created a language server protocol [2], which already has both
> a bunch of client and server implementations
> 3. Debugging through python into libclang crashers is a pain and eating
> our support resources
> 4. While libclang is a good abstraction if you want to have something
> run in your process with close coupling, a standard protocol like the
> language server protocol seems like a better way to allow fast
> iterations on the server implementation without the need to keep
> backward-compatibility hacks through a restrictive C API.
>
> One of the cool things about the MS language server protocol is that it
> seems to not actually do any networking, which means that we do not need
> to introduce any new dependencies into clang-tools-extra, which is where
> we want to start developing this.
>
> If you have any thoughts / concerns let me know; otherwise look forward
> to code reviews on initial ClangD dropping by :D

[Huch, my first association with "ClangD" was support for the D language]

Hi!

We from the Qt Creator IDE team are also interested in this. :)

Some notes:

1. You've indirectly referenced the initial design document at [1]. Will
this get an update or is it already updated somewhere else? An updated
document clarifying the scope and goals would be probably useful to all
interested parties, also in regard to contribution and how to align own
current development that uses libclang/libtooling/clang tools.

So far the scope is replacing the C++ parts of YCM (that is, code completion, code browsing, "fast" syntax checking and fixits)
 
E.g. Qt Creator also already launches a separate process
("clangbackend") that makes use of libclang to provide the basic
functionality like updated diagnostics, highlighting information and
results for completion requests.


2. How is clangd related to "clang-refactor" [2]?

If it's not related at all, what's the state of clang-refactor?

clang-refactor is still in the design stage. I think at some point we'll probably want to integrate clang-refactor, but we first want a prototype and see how well that goes.
 

3. I take this opportunity to share our biggest problems with libclang
from the Qt Creator perspective since clangd users will probably face
them also (as far as I understand, clangd will be based upon libclang -
please correct me if I'm wrong). I share this in hope that they will be
addressed or taken into consideration during the development of clangd
for the benefit of all interested parties now that the development of
clangd is announced. Please keep in mind that the actual problems might
be in other components that libclang makes use of.

The problems we are facing with libclang, in this order, are:

3.1 Windows-specific issues

The whole performance of parse/reparse in general is worser than on
Mac/Linux (probably also because of slow IO on Windows). At the end of
the day, this makes it less usable on Windows for real-world
projects/files. There are also other Windows-specific issues.

My gut feeling is that there aren't that many libclang users on Windows.

@Manual: To which extend do you or your team use and develop on Windows?

We are not Windows developers :) We hope that by integrating with VS Code / MS language protocol we can get some interested parties on Windows, but so far we rely on help from Windows wizards.
 
Are any Windows developers working on clangd?

Well, it'll be an open source project ;)
 
Concrete issues are:

  * Preamble is also generated for the second reparse on Windows
    https://llvm.org/bugs/show_bug.cgi?id=28535

  * Problems with FileManager caches leading to leaking file descriptors
    https://reviews.llvm.org/D27810

  * Very slow reparsing for big header and even slower with
    preamble-generation enabled (might be related to the one above)
    https://llvm.org/bugs/show_bug.cgi?id=31353

  * Windows: Crash with absolute path with forward slashes
    https://llvm.org/bugs/show_bug.cgi?id=28381

3.2 Synchronous/blocking API
Low-latency completion results at all times are impossible with a single
CXTranslationUnit that is just being parsed or reparsed. This gets worse
the longer the parse/reparse takes (hello Windows).

One can mitigate this by triggering the parse/reparse after a
significant timeout of user-typing-inactivity, but this leads to a
sluggish experience regarding new diagnostics and highlighting
informations (you don't want to wait more than 2s for new diagnostics to
see if your code is valid now).

Another approach is to use two CXTranslationUnits per file, so at least
there is always a non-blocking CXTranslationUnits ready for completion
requests. Downside is doubled memory consumption and potentially
threading issues (@Manual: do they still exist or are they already
resolved?).

I work on a very large codebase and don't have problems with code completion. I think all YCM integrations I know take care to not block the UI for completion requests (or in general).
 
3.3 libclang requires unsaved files to exist on disk.

If the unsaved file does not exist a preamble generation will happen for
every reparse which simply takes to long.

I would be surprised if that was true - I often work on unsaved files without seeing these problems.
 

For the concrete use case, refer to

        https://llvm.org/bugs/show_bug.cgi?id=31112

This one might be also a problem if you do version control checkouts and
files suddenly vanish.



Best regards,
    Nikolai



[1] https://github.com/chandlerc/llvm-designs/blob/master/ClangService.rst
[2] http://lists.llvm.org/pipermail/cfe-dev/2016-September/050641.html

>
> Cheers,
> /Manuel
>
> [1] http://lists.llvm.org/pipermail/cfe-dev/2012-June/022028.html
> [2] https://github.com/Microsoft/language-server-protocol
>
>
>
> _______________________________________________
> cfe-dev mailing list
> [hidden email]
> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev


_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: ClangD

Roman Popov via cfe-dev
In reply to this post by Roman Popov via cfe-dev
On Thu, Jan 26, 2017 at 3:33 PM Nikolai Kosjar <[hidden email]> wrote:
On 01/25/2017 03:13 PM, Manuel Klimek via cfe-dev wrote:
> On Wed, Jan 25, 2017 at 3:08 PM Alex L <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     Thanks Manuel, I'm quite excited to hear that about ClangD!
>
>     I have a couple of questions:
>     Do you think ClangD would be able to replace libclang in the future?
>     Would it be possible to deprecate libclang for IDE use after ClangD
>     catches up to it, or should we keep libclang as it is even after
>     that point?
>
>
> In the foreseeable future, I don't see libclang going away. Not
> everybody is able to switch to a new development workflow easily, and
> there is a lot of investment in the current libclang based workflows.

Just for clarity, please elaborate on "lot of investment in the current
libclang based workflows".

YCM exists, and people use it. Xcode exists and people use it. There is a python integration based on libclang. Probably more users :)
 

>     On 25 January 2017 at 13:11, Manuel Klimek via cfe-dev
>     <[hidden email] <mailto:[hidden email]>> wrote:
>
>         Hi fellow clang devs,
>
>         we wanted to let you know that we're (finally) starting up work
>         on ClangD, which you might know from email threads such as [1]
>         (June 2012!).
>
>         In the meantime, YCM had done a good enough job at packaging up
>         a libclang interface to our favorite editors, and other
>         priorities (like modules) have eaten up a lot of folks priority
>         lunches.
>
>         What has changed?
>         1. YCM is starting to develop more and more into a language
>         multiplexer, with other languages (Go, Typescript, etc)
>         providing their own servers to talk to
>         2. MS has created a language server protocol [2], which already
>         has both a bunch of client and server implementations
>         3. Debugging through python into libclang crashers is a pain and
>         eating our support resources
>         4. While libclang is a good abstraction if you want to have
>         something run in your process with close coupling, a standard
>         protocol like the language server protocol seems like a better
>         way to allow fast iterations on the server implementation
>         without the need to keep backward-compatibility hacks through a
>         restrictive C API.
>
>         One of the cool things about the MS language server protocol is
>         that it seems to not actually do any networking, which means
>         that we do not need to introduce any new dependencies into
>         clang-tools-extra, which is where we want to start developing this.
>
>         If you have any thoughts / concerns let me know; otherwise look
>         forward to code reviews on initial ClangD dropping by :D
>
>         Cheers,
>         /Manuel
>
>         [1] http://lists.llvm.org/pipermail/cfe-dev/2012-June/022028.html
>         [2] https://github.com/Microsoft/language-server-protocol
>
>
>         _______________________________________________
>         cfe-dev mailing list
>         [hidden email] <mailto:[hidden email]>
>         http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>
>
>
> _______________________________________________
> cfe-dev mailing list
> [hidden email]
> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>


_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: ClangD

Roman Popov via cfe-dev
In reply to this post by Roman Popov via cfe-dev
Hi Manuel,

> If you have any thoughts / concerns let me know; otherwise look forward to
> code reviews on initial ClangD dropping by :D

One minor point, which I could not spot on any of the replies: is it
worth looking at RTags[1]? Its a libclang-based project with very good
emacs integration. I've been using it daily on a large-ish code base
and its pretty stable. Adding the MS protocol to it would probably be
easier than starting from scratch. (I am in no way affiliated with the
project, other than as a user).

Also, the name ClangD seems to evoke D support for Clang :-)

My 0.02 Angolan Kwanzas.

Cheers
--
Marco Craveiro

MD, Domain Driven Consulting
about: http://about.me/marcocraveiro
blog: http://mcraveiro.blogspot.co.uk
twitter: https://twitter.com/MarcoCraveiro

That the Ideas are themselves manifestations (of the Idea-Idea) and
that the Idea-Idea is a-kind-of Manifestation-Idea—which is a-kind-of
itself, so that the system is completely self-describing— would have
been appreciated by Plato as an extremely practical joke [Plato]. --
Alan Key

[1] https://github.com/Andersbakken/rtags/
_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: ClangD

Roman Popov via cfe-dev


On Mon, Feb 6, 2017 at 5:40 PM Marco Craveiro <[hidden email]> wrote:
Hi Manuel,

> If you have any thoughts / concerns let me know; otherwise look forward to
> code reviews on initial ClangD dropping by :D

One minor point, which I could not spot on any of the replies: is it
worth looking at RTags[1]? Its a libclang-based project with very good
emacs integration. I've been using it daily on a large-ish code base
and its pretty stable. Adding the MS protocol to it would probably be
easier than starting from scratch. (I am in no way affiliated with the
project, other than as a user).

YCM is a libclang based editor integration that many people use - we specifically want to get away from libclang (for various reasons).
For what it's worth, I also don't think adding the editor protocol to a different project will be easier than starting from scratch.
 

Also, the name ClangD seems to evoke D support for Clang :-)

My 0.02 Angolan Kwanzas.

Cheers
--
Marco Craveiro

MD, Domain Driven Consulting
about: http://about.me/marcocraveiro
blog: http://mcraveiro.blogspot.co.uk
twitter: https://twitter.com/MarcoCraveiro

That the Ideas are themselves manifestations (of the Idea-Idea) and
that the Idea-Idea is a-kind-of Manifestation-Idea—which is a-kind-of
itself, so that the system is completely self-describing— would have
been appreciated by Plato as an extremely practical joke [Plato]. --
Alan Key

[1] https://github.com/Andersbakken/rtags/

_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: ClangD

Roman Popov via cfe-dev
<rtags>
> YCM is a libclang based editor integration that many people use - we
> specifically want to get away from libclang (for various reasons).
> For what it's worth, I also don't think adding the editor protocol to a
> different project will be easier than starting from scratch.

Ah, I see - missed that part. Well, eagerly awaiting for CLangD then. :-)

Cheers
--
Marco Craveiro

MD, Domain Driven Consulting
about: http://about.me/marcocraveiro
blog: http://mcraveiro.blogspot.co.uk
twitter: https://twitter.com/MarcoCraveiro

That the Ideas are themselves manifestations (of the Idea-Idea) and
that the Idea-Idea is a-kind-of Manifestation-Idea—which is a-kind-of
itself, so that the system is completely self-describing— would have
been appreciated by Plato as an extremely practical joke [Plato]. --
Alan Key
_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: ClangD

Roman Popov via cfe-dev
In reply to this post by Roman Popov via cfe-dev
On Mon, Feb 6, 2017 at 5:40 PM, Marco Craveiro via cfe-dev
<[hidden email]> wrote:

> Hi Manuel,
>
>> If you have any thoughts / concerns let me know; otherwise look forward to
>> code reviews on initial ClangD dropping by :D
>
> One minor point, which I could not spot on any of the replies: is it
> worth looking at RTags[1]? Its a libclang-based project with very good
> emacs integration. I've been using it daily on a large-ish code base
> and its pretty stable. Adding the MS protocol to it would probably be
> easier than starting from scratch. (I am in no way affiliated with the
> project, other than as a user).
>
> Also, the name ClangD seems to evoke D support for Clang :-)

I've heard this from multiple people, so the official spelling should
be clangd or Clangd in the traditional unix way ;)

>
> My 0.02 Angolan Kwanzas.
>
> Cheers
> --
> Marco Craveiro
>
> MD, Domain Driven Consulting
> about: http://about.me/marcocraveiro
> blog: http://mcraveiro.blogspot.co.uk
> twitter: https://twitter.com/MarcoCraveiro
>
> That the Ideas are themselves manifestations (of the Idea-Idea) and
> that the Idea-Idea is a-kind-of Manifestation-Idea—which is a-kind-of
> itself, so that the system is completely self-describing— would have
> been appreciated by Plato as an extremely practical joke [Plato]. --
> Alan Key
>
> [1] https://github.com/Andersbakken/rtags/
> _______________________________________________
> cfe-dev mailing list
> [hidden email]
> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: ClangD

Roman Popov via cfe-dev

On Feb 6, 2017, at 11:49 AM, Benjamin Kramer via cfe-dev <[hidden email]> wrote:

On Mon, Feb 6, 2017 at 5:40 PM, Marco Craveiro via cfe-dev
<[hidden email]> wrote:
Hi Manuel,

If you have any thoughts / concerns let me know; otherwise look forward to
code reviews on initial ClangD dropping by :D

One minor point, which I could not spot on any of the replies: is it
worth looking at RTags[1]? Its a libclang-based project with very good
emacs integration. I've been using it daily on a large-ish code base
and its pretty stable. Adding the MS protocol to it would probably be
easier than starting from scratch. (I am in no way affiliated with the
project, other than as a user).

Also, the name ClangD seems to evoke D support for Clang :-)

I've heard this from multiple people, so the official spelling should
be clangd or Clangd in the traditional unix way ;)

Trying to prevent future naming collision issue: clangd seems too generic to me and likely to collide with a general purpose clang `compilation server`, unless you plan to have a single `clangd` that can service multiple type of `service` (indexing, compiling, linking, …).

— 
Mehdi


_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: ClangD

Roman Popov via cfe-dev
On Tue, Feb 7, 2017 at 9:18 AM Mehdi Amini via cfe-dev <[hidden email]> wrote:
On Feb 6, 2017, at 11:49 AM, Benjamin Kramer via cfe-dev <[hidden email]> wrote:

On Mon, Feb 6, 2017 at 5:40 PM, Marco Craveiro via cfe-dev
<[hidden email]> wrote:
Hi Manuel,

If you have any thoughts / concerns let me know; otherwise look forward to
code reviews on initial ClangD dropping by :D

One minor point, which I could not spot on any of the replies: is it
worth looking at RTags[1]? Its a libclang-based project with very good
emacs integration. I've been using it daily on a large-ish code base
and its pretty stable. Adding the MS protocol to it would probably be
easier than starting from scratch. (I am in no way affiliated with the
project, other than as a user).

Also, the name ClangD seems to evoke D support for Clang :-)

I've heard this from multiple people, so the official spelling should
be clangd or Clangd in the traditional unix way ;)

Trying to prevent future naming collision issue: clangd seems too generic to me and likely to collide with a general purpose clang `compilation server`, unless you plan to have a single `clangd` that can service multiple type of `service` (indexing, compiling, linking, …).

Compiling and linking is within scope. It is not yet clear in what form.
 

— 
Mehdi

_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev

_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev