Distcc implementation details

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Distcc implementation details

Mike Miller
Hi cfe-dev,

I'm planning to (attempt to) add in native distcc support to clang(per Chris Lattner's suggestion). The feature would work like this:

1. The user on the "master" would do something like CC="clang --distributed dist.conf" make -j 100(where 100 is the number of slave nodes)

2. Each clang process spawned by 'make' would attempt to connect to a socket(local UNIX socket, that is) with a path representative of the configuration file's path. I was thinking maybe /tmp/clang/hash_of_path_of_dist.conf. The other end of the socket would be a listening clang server, which would have a central FileManager(and thus a central cache). This would save big on syscalls and disk accesses(Chris Lattner showed this in his LLVM 2.0 tech talk at Google) - around 3.3x preprocessing speedup! The server would be started up on demand. If it wasn't possible to setup the socket, the fork()ed server will connect to the socket, and if the connection succeeds, will exit, because the socket has already been setup.

3. If the socket connection succeeds, clang would send over the command sent by the user to the process, which would preprocess the source, and send it out to a "slave" node along with all compiler options(is it worth performing basic compression on the preprocessed source beforehand?). The slave node would compile the source down to object code, and send it back to the "master" node(along with diagnostics). After this, the "central" process running on the host would send a "DONE" command back to the calling clang process via the UNIX socket, along with diagnostics generated during the build.

4. If the socket connection didn't succeed, the server is started up on demand. clang would fork() off a server which would run in a loop that looks like this:

connect_to_slaves();
while(1){
accept(); //non-blocking call.
 //alternate between accepting local requests and network requests from slaves
if(localRequest){
recieve_command_line_args();
preprocess_source();
send_preprocessed_source_and_command_line_args_to_slave(); //Round robin?
}
else if(slaveResponse){
get_obj_data();
get_diagnostics();
save_obj_data_to_disk();
send_diagnostics_to_calling_clang_process();
}
timeout(); //If certain time period(30s?) elapsed since last request, 
  // AND no requests received
  // AND no files left to process from slaves
//exit() server.
//Otherwise, loop back.
}


After the fork(), the parent clang process would attempt to reconnect to the server.

On the slave nodes, there would be multiple listening 'clangd' processes(presumably one per core, but this could be configurable) which would all be listening starting at some port(maybe 63000 for core 1, 63001 for core 2, etc.). The slave nodes would do everything from preprocessing to codegen - this means for the first version LTO wouldn't be supported.

Each slave node would have a server thread which accepted requests and put them on a (lockless?) queue, and a "consumption" thread would keep pulling requests off the queue and processing them.

In terms of actual implementation, I'm very new to the clang codebase, and have very little familiarity with it. After poking around a bit, it seems like a good place for my new code to be called from would be "Compilation *Driver::BuildCompilation(int argc, const char **argv)" in Driver.cpp. Does this seem like a good place to "inject" distcc code?

Before I go off and (attempt to) implement this, I'd like to ask for any feedback about the implementation I've proposed. Am I doing something the "wrong way"? Would you do anything differently? Please, let me know any concerns you have - I'd hate to implement this and find out I did it the "wrong way" and have to re-implement all over again!

Thanks!
Mike

_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: Distcc implementation details

Chris Lattner
On Mar 14, 2010, at 12:03 AM, Mike Miller wrote:
> Hi cfe-dev,
>
> I'm planning to (attempt to) add in native distcc support to clang(per Chris Lattner's suggestion). The feature would work like this:

This sounds great to me!

> 3. If the socket connection succeeds, clang would send over the command sent by the user to the process, which would preprocess the source, and send it out to a "slave" node along with all compiler options(is it worth performing basic compression on the preprocessed source beforehand?).

It's hard to say and will probably depend on the configuration.  I'd start with not doing it and then experiment with adding it later.

> After the fork(), the parent clang process would attempt to reconnect to the server.

The forked server should beware of the potenial race condition when multiple clang's all start up at the same time, all notice there is no server process, and all fork themselves.  Only one should win.

> On the slave nodes, there would be multiple listening 'clangd' processes(presumably one per core, but this could be configurable) which would all be listening starting at some port(maybe 63000 for core 1, 63001 for core 2, etc.). The slave nodes would do everything from preprocessing to codegen - this means for the first version LTO wouldn't be supported.

Right, eventually it could just be one multithreaded server once clang and llvm are thread safe enough.

> In terms of actual implementation, I'm very new to the clang codebase, and have very little familiarity with it. After poking around a bit, it seems like a good place for my new code to be called from would be "Compilation *Driver::BuildCompilation(int argc, const char **argv)" in Driver.cpp. Does this seem like a good place to "inject" distcc code?

Daniel is the right one to answer this,

> Before I go off and (attempt to) implement this, I'd like to ask for any feedback about the implementation I've proposed. Am I doing something the "wrong way"? Would you do anything differently? Please, let me know any concerns you have - I'd hate to implement this and find out I did it the "wrong way" and have to re-implement all over again!

Sounds great to me!

-Chris
_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev