Compile Compact C Compiler from Clang on Windows 10. For Use with Lua

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Compile Compact C Compiler from Clang on Windows 10. For Use with Lua

Hans Wennborg via cfe-dev
Hi, 

I've started investigating the clang source code. I've built LLVM and Clang from source in Windows 10 using Visual Studio (I'll try again with Ninja at some point). I'd like to only build the C tools if possible? I found documentation on cmake options for building clang (e.g. building C++ or building Objective-c) but I've lost track of it if anyone could point me in the right direction?

Anyway, my objective is to create a very compact C compiler so I want to strip out anything unnecessary for building small dlls from existing code. My use case is this:

A user downloads and installs my package that contains a pre-compiled Lua interpreter and the Lua package manager called Luarocks, as well as the C toolchain. They can then drop to the command line and build and install a lua module known as a "rock". For example, if I wanted to download, build and install the stopwatch module chronos:

luarocks install chronos

The build system is built into Luarocks (and I may also include gmake). The user would never debug or analyze the C code, they would only use compiled the C modules in Lua.

So, I'm wondering if I can remove the static analyzer and other components? Technically I don't even need debug information. 

During my meandering through the documentation I read that I can compile clang using dlls to conserve space at the cost of efficiency, which may be a trade off I'm willing to make. I'm wondering if the performance is *terrible* or just not as efficient as one would like? If we are talking about a few seconds difference to build a small library then I think that's worth while.

I also read in the documentation something along the lines of "Use the Visual Studio solution generated from CMake to read the Clang source code". I tried that but the solution files created by CMake don't seem to link to the source code? In fact, I've never had CMake generated VS solutions link to source code so I'm not sure what was actually being expressed? Can VS be used to browse the project or should I try to import it into Eclipse or CLion? I typically use Geany but most of my C work in Windows is limited in scope.

Any and all input is welcome. 

Regards, 
Russ

_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: Compile Compact C Compiler from Clang on Windows 10. For Use with Lua

Hans Wennborg via cfe-dev
I apologize for not reading and answering in more detail, but generally speaking, Clang is a big compiler, not a small compiler. Clang has grown many features, and there is no support for compiling them out individually. For example, while you can compile out LLVM backend targets that you aren't using, it's not possible to do the same in Clang. Clang can always generate IR for all targets it supports, which includes all the intrinsic builtins, which are ever-growing and non-negligible in size. Similarly, one cannot disable ObjC, C++, MS C++ ABI, CUDA, ARC, or OpenMP support. Sometimes I wish this were possible, but it would result in far too many build configurations for us to test.

I think there is a separate aspect to what you are asking, which is, even if you take all of clang as it is, it is still not quite enough to compile and link C code into a DLL. You typically need SDK headers (windows.h) and a standard C library (MSVCRT or mingw). LLVM doesn't provide this, and you have to get it from Microsoft or mingw.

I would say that what you want to do is definitely possible, but it is not easy. Sorry that wasn't more helpful, but I hope it answers your question.

On Sat, Jan 18, 2020 at 9:35 PM Russell Haley via cfe-dev <[hidden email]> wrote:
Hi, 

I've started investigating the clang source code. I've built LLVM and Clang from source in Windows 10 using Visual Studio (I'll try again with Ninja at some point). I'd like to only build the C tools if possible? I found documentation on cmake options for building clang (e.g. building C++ or building Objective-c) but I've lost track of it if anyone could point me in the right direction?

Anyway, my objective is to create a very compact C compiler so I want to strip out anything unnecessary for building small dlls from existing code. My use case is this:

A user downloads and installs my package that contains a pre-compiled Lua interpreter and the Lua package manager called Luarocks, as well as the C toolchain. They can then drop to the command line and build and install a lua module known as a "rock". For example, if I wanted to download, build and install the stopwatch module chronos:

luarocks install chronos

The build system is built into Luarocks (and I may also include gmake). The user would never debug or analyze the C code, they would only use compiled the C modules in Lua.

So, I'm wondering if I can remove the static analyzer and other components? Technically I don't even need debug information. 

During my meandering through the documentation I read that I can compile clang using dlls to conserve space at the cost of efficiency, which may be a trade off I'm willing to make. I'm wondering if the performance is *terrible* or just not as efficient as one would like? If we are talking about a few seconds difference to build a small library then I think that's worth while.

I also read in the documentation something along the lines of "Use the Visual Studio solution generated from CMake to read the Clang source code". I tried that but the solution files created by CMake don't seem to link to the source code? In fact, I've never had CMake generated VS solutions link to source code so I'm not sure what was actually being expressed? Can VS be used to browse the project or should I try to import it into Eclipse or CLion? I typically use Geany but most of my C work in Windows is limited in scope.

Any and all input is welcome. 

Regards, 
Russ
_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev

_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: Compile Compact C Compiler from Clang on Windows 10. For Use with Lua

Hans Wennborg via cfe-dev


On Sun, Jan 19, 2020 at 8:57 AM Reid Kleckner <[hidden email]> wrote:
I apologize for not reading and answering in more detail, but generally speaking, Clang is a big compiler, not a small compiler.
No problem. Long rambling emails from the uninformed are best answered in a general sense. :)
 
Clang has grown many features, and there is no support for compiling them out individually. For example, while you can compile out LLVM backend targets that you aren't using, it's not possible to do the same in Clang. Clang can always generate IR for all targets it supports, which includes all the intrinsic builtins, which are ever-growing and non-negligible in size. Similarly, one cannot disable ObjC, C++, MS C++ ABI, CUDA, ARC, or OpenMP support. Sometimes I wish this were possible, but it would result in far too many build configurations for us to test.
Okay, good to know. I just checked a release build and it's much smaller than my debug build (57 Mb vs. 270 Mb). I'll compare to the official binary download. I'm aiming to bring the entire thing in at 200 Mb (including all other tools and libraries) so that might be acceptable.

I think there is a separate aspect to what you are asking, which is, even if you take all of clang as it is, it is still not quite enough to compile and link C code into a DLL. You typically need SDK headers (windows.h) and a standard C library (MSVCRT or mingw). LLVM doesn't provide this, and you have to get it from Microsoft or mingw.
Yes, I'm aware of the need for a standard library. I'm researching if I can re-distribute the header and lib files from the MSVC Build Tools download. A brief reading of their license indicates that I cannot redistribute the *debug* binaries but doesn't say anything about the headers or lib files. A second possible approach as suggested by Zachary Turner would be to compile the mingw library with clang and include it but I don't necessarily want posix support compared to the MS libc (also noted by Mr. Turner). 

Another approach would be similar to what Embarcadero has done with clang and create my own runtime, either from BSD libc or the libc included with the CLang sources. I saw a document describing a means to stub functionality from one libc to another but it was *nix specific. More investigation is necessary.

I would say that what you want to do is definitely possible, but it is not easy. Sorry that wasn't more helpful, but I hope it answers your question.
Answers no, but informs, yes. Thank you for your time!

Russ 

On Sat, Jan 18, 2020 at 9:35 PM Russell Haley via cfe-dev <[hidden email]> wrote:
Hi, 

I've started investigating the clang source code. I've built LLVM and Clang from source in Windows 10 using Visual Studio (I'll try again with Ninja at some point). I'd like to only build the C tools if possible? I found documentation on cmake options for building clang (e.g. building C++ or building Objective-c) but I've lost track of it if anyone could point me in the right direction?

Anyway, my objective is to create a very compact C compiler so I want to strip out anything unnecessary for building small dlls from existing code. My use case is this:

A user downloads and installs my package that contains a pre-compiled Lua interpreter and the Lua package manager called Luarocks, as well as the C toolchain. They can then drop to the command line and build and install a lua module known as a "rock". For example, if I wanted to download, build and install the stopwatch module chronos:

luarocks install chronos

The build system is built into Luarocks (and I may also include gmake). The user would never debug or analyze the C code, they would only use compiled the C modules in Lua.

So, I'm wondering if I can remove the static analyzer and other components? Technically I don't even need debug information. 

During my meandering through the documentation I read that I can compile clang using dlls to conserve space at the cost of efficiency, which may be a trade off I'm willing to make. I'm wondering if the performance is *terrible* or just not as efficient as one would like? If we are talking about a few seconds difference to build a small library then I think that's worth while.

I also read in the documentation something along the lines of "Use the Visual Studio solution generated from CMake to read the Clang source code". I tried that but the solution files created by CMake don't seem to link to the source code? In fact, I've never had CMake generated VS solutions link to source code so I'm not sure what was actually being expressed? Can VS be used to browse the project or should I try to import it into Eclipse or CLion? I typically use Geany but most of my C work in Windows is limited in scope.

Any and all input is welcome. 

Regards, 
Russ
_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev

_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: Compile Compact C Compiler from Clang on Windows 10. For Use with Lua

Hans Wennborg via cfe-dev
On Sun, 19 Jan 2020, Russell Haley via cfe-dev wrote:

>       I think there is a separate aspect to what you are asking, which
>       is, even if you take all of clang as it is, it is still not
>       quite enough to compile and link C code into a DLL. You
>       typically need SDK headers (windows.h) and a standard C library
>       (MSVCRT or mingw). LLVM doesn't provide this, and you have to
>       get it from Microsoft or mingw.
>
> Yes, I'm aware of the need for a standard library. I'm researching if I can
> re-distribute the header and lib files from the MSVC Build Tools download. A
> brief reading of their license indicates that I cannot redistribute the
> *debug* binaries but doesn't say anything about the headers or lib files. A
> second possible approach as suggested by Zachary Turner would be to compile
> the mingw library with clang and include it but I don't necessarily want
> posix support compared to the MS libc (also noted by Mr. Turner). 
As far as I know, those files aren't redistributable. The fact that they
aren't is one of the primary reasons for mingw in itself.

Also do note that mingw doesn't provide posix support; mingw primarily
intends to provide a freely redistributable "sdk" for pure native windows
code, allowing you to build executables that only rely on the CRT DLLs
(the windows internal msvcrt.dll, and recently, the UCRT) that ship with
windows itself (to allow building GPL code and redistributing it).

In addition to the CRT and the windows APIs themselves, mingw does provide
a few extra posix originated APIs (usleep, getopt and a few other similar
ones), but this is mostly an exception.

This is contrary to cygwin, which provides a full posix environment (fork
and all), but requires linking everything to their runtime, and generally
behaves as an isolated environment from the rest of windows.

If what you want is clang + a redistributable sdk, I just so happen to
have that prepackaged for you, at https://github.com/mstorsjo/llvm-mingw.
The packed release is around 150 MB, and the unpacked release weighs in at
around 500 MB at the moment, but if you remove import libraries for three
out of four architectures, LLDB and lesser used tools like llvm-objdump,
llvm-nm etc, you can get it down to slightly above 200 MB.

// Martin

_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: Compile Compact C Compiler from Clang on Windows 10. For Use with Lua

Hans Wennborg via cfe-dev


On Sun, Jan 19, 2020 at 12:27 PM Martin Storsjö <[hidden email]> wrote:
On Sun, 19 Jan 2020, Russell Haley via cfe-dev wrote:

>       I think there is a separate aspect to what you are asking, which
>       is, even if you take all of clang as it is, it is still not
>       quite enough to compile and link C code into a DLL. You
>       typically need SDK headers (windows.h) and a standard C library
>       (MSVCRT or mingw). LLVM doesn't provide this, and you have to
>       get it from Microsoft or mingw.
>
> Yes, I'm aware of the need for a standard library. I'm researching if I can
> re-distribute the header and lib files from the MSVC Build Tools download. A
> brief reading of their license indicates that I cannot redistribute the
> *debug* binaries but doesn't say anything about the headers or lib files. A
> second possible approach as suggested by Zachary Turner would be to compile
> the mingw library with clang and include it but I don't necessarily want
> posix support compared to the MS libc (also noted by Mr. Turner). 

As far as I know, those files aren't redistributable. The fact that they
aren't is one of the primary reasons for mingw in itself.

Also do note that mingw doesn't provide posix support; mingw primarily
intends to provide a freely redistributable "sdk" for pure native windows
code, allowing you to build executables that only rely on the CRT DLLs
(the windows internal msvcrt.dll, and recently, the UCRT) that ship with
windows itself (to allow building GPL code and redistributing it).

In addition to the CRT and the windows APIs themselves, mingw does provide
a few extra posix originated APIs (usleep, getopt and a few other similar
ones), but this is mostly an exception.

This is contrary to cygwin, which provides a full posix environment (fork
and all), but requires linking everything to their runtime, and generally
behaves as an isolated environment from the rest of windows.

If what you want is clang + a redistributable sdk, I just so happen to
have that prepackaged for you, at https://github.com/mstorsjo/llvm-mingw.
The packed release is around 150 MB, and the unpacked release weighs in at
around 500 MB at the moment, but if you remove import libraries for three
out of four architectures, LLDB and lesser used tools like llvm-objdump,
llvm-nm etc, you can get it down to slightly above 200 MB.

// Martin

That sounds exactly like what I'm trying to achieve.

You may have answered a question I've had for some time: "Does mingw require a redistributable"? It sounds like the answer is "whenever linking to non-standard Windows API calls". If that's the case, how does one get that redistributable and/or know when it's required?

Many thanks!
Russell

_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: Compile Compact C Compiler from Clang on Windows 10. For Use with Lua

Hans Wennborg via cfe-dev
Hi,

On Sun, 19 Jan 2020, Russell Haley wrote:

> You may have answered a question I've had for some time: "Does mingw require
> a redistributable"? It sounds like the answer is "whenever linking to
> non-standard Windows API calls". If that's the case, how does one get that
> redistributable and/or know when it's required?

In general, mingw doesn't require extra redistributables.

The full answer is more like this:

Mingw does (as mentioned before) provide a few extra nonstandard APIs like
usleep, getopt and maybe a few others. These are always statically linked
into your executable and doesn't impose any extra redistributable
components. Mingw also statically links a few other bits, primarily a
varied set of reimplementations of math functions, as the default linked
msvcrt.dll doesn't provide all C99 math functions, and not all of the ones
that are provided are fully C99 compliant.

There used to be a separate redistributable dll that had to be linked and
included for some thread specific initialization, but that's not
required/used in modern times, at least not when using the mingw-w64 fork
(which I'm using).

If you build C++ code, you'll end up with a dependence on a separate DLL
for the C++ standard library, but you can either add -static when linking,
or remove libc++.dll.a, to avoid linking it dynamically and instead
statically linking it into each module.

If using a GCC based mingw setup (e.g. like the ones from
https://sourceforge.net/projects/mingw-w64/files/Toolchains%20targetting%20Win64/Personal%20Builds/mingw-builds/8.1.0/),
you can also get libgcc linked dynamically, but you can overcome that in
the same way as above, with either -static or -static-libgcc. My
toolchains use compiler_rt instead of libgcc, and that's always linked
statically.

// Martin

_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev