Out of Memory Building Clang and LLVM

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Out of Memory Building Clang and LLVM

Cameron McInally via cfe-dev
Greetings,

Seems I'm running out of memory on a host system trying to build llvm, tooling and clang. It was configured
as:
cmake -DLLVM_ENABLE_PROJECTS=clang -G "Unix Makefiles" ../llvm

and build with:

make -j4 using gcc as the host compiler, version 8.

The dmesg I'm getting is related to the linker and not sure if the linker has a bug or it's normal
to expect 3GB or more used for linking clang. Doesn't seem to be but not sure.

dmesg output:
[186517.023397] Out of memory: Kill process 2143 (ld) score 286 or sacrifice child
[186517.023403] Killed process 2143 (ld) total-vm:4622188kB, anon-rss:3202188kB, file-rss:4kB, shmem-rss:0kB


Nick
_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: Out of Memory Building Clang and LLVM

Cameron McInally via cfe-dev
On Wed, Jun 19, 2019 at 06:36:33PM -0400, nick via cfe-dev wrote:

> Greetings,
>
> Seems I'm running out of memory on a host system trying to build llvm, tooling and clang. It was configured
> as:
> cmake -DLLVM_ENABLE_PROJECTS=clang -G "Unix Makefiles" ../llvm
>
> and build with:
>
> make -j4 using gcc as the host compiler, version 8.
>
> The dmesg I'm getting is related to the linker and not sure if the linker has a bug or it's normal
> to expect 3GB or more used for linking clang. Doesn't seem to be but not sure.
>
> dmesg output:
> [186517.023397] Out of memory: Kill process 2143 (ld) score 286 or sacrifice child
> [186517.023403] Killed process 2143 (ld) total-vm:4622188kB, anon-rss:3202188kB, file-rss:4kB, shmem-rss:0kB
I can't comment on this particular configuration, but last I
heard it still builds in 3GB.  That said, you may want to include
-DLLVM_PARALLEL_LINK_JOBS=1 in your linker arguments.  At least with
ninja (not sure about make), there tend to be many programs being linked
at the end and on a low-memory system this can lead to out of memory
conditions.

-- Brooks

_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev

signature.asc (465 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Out of Memory Building Clang and LLVM

Cameron McInally via cfe-dev
In reply to this post by Cameron McInally via cfe-dev


On Wed, 19 Jun 2019 at 23:36, nick via cfe-dev <[hidden email]> wrote:
Greetings,

Seems I'm running out of memory on a host system trying to build llvm, tooling and clang. It was configured
as:
cmake -DLLVM_ENABLE_PROJECTS=clang -G "Unix Makefiles" ../llvm

and build with:

make -j4 using gcc as the host compiler, version 8.

The dmesg I'm getting is related to the linker and not sure if the linker has a bug or it's normal
to expect 3GB or more used for linking clang. Doesn't seem to be but not sure.

dmesg output:
[186517.023397] Out of memory: Kill process 2143 (ld) score 286 or sacrifice child
[186517.023403] Killed process 2143 (ld) total-vm:4622188kB, anon-rss:3202188kB, file-rss:4kB, shmem-rss:0kB



HI,

I have a 16 Core/32 Thread system with 32GB RAM, and it can have difficulty building CLANG/LLVM.
The link stage definitely needs to be with make -j1.

That being said, if you force a "Release" version, and not the "Debug" version, much less RAM is needed.

Kind Regards

James



_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: Out of Memory Building Clang and LLVM

Cameron McInally via cfe-dev

Clang definitely hits a pathological case in LD that makes it do terrible things if you use use more than a process or two in DEBUG. 

 

However, if you use “gold” or “LLD” instead, it works fine.  Assuming you have LLD, you can do:

-DLLVM_USE_LINKER=lld or –DLLVM_ENABLE_LLD=ON

 

Otherwise you should have gold available, which you can do with:

-DLLVM_USE_LINKER=gold

 

I can only run about 3-4 links at a time with LD and can do –j30 with no problems with Gold.

 

From: cfe-dev [mailto:[hidden email]] On Behalf Of James Courtier-Dutton via cfe-dev
Sent: Thursday, June 20, 2019 3:27 AM
To: nick <[hidden email]>
Cc: [hidden email] Developers <[hidden email]>
Subject: Re: [cfe-dev] Out of Memory Building Clang and LLVM

 

 

 

On Wed, 19 Jun 2019 at 23:36, nick via cfe-dev <[hidden email]> wrote:

Greetings,

Seems I'm running out of memory on a host system trying to build llvm, tooling and clang. It was configured
as:
cmake -DLLVM_ENABLE_PROJECTS=clang -G "Unix Makefiles" ../llvm

and build with:

make -j4 using gcc as the host compiler, version 8.

The dmesg I'm getting is related to the linker and not sure if the linker has a bug or it's normal
to expect 3GB or more used for linking clang. Doesn't seem to be but not sure.

dmesg output:
[186517.023397] Out of memory: Kill process 2143 (ld) score 286 or sacrifice child
[186517.023403] Killed process 2143 (ld) total-vm:4622188kB, anon-rss:3202188kB, file-rss:4kB, shmem-rss:0kB

 

HI,

 

I have a 16 Core/32 Thread system with 32GB RAM, and it can have difficulty building CLANG/LLVM.

The link stage definitely needs to be with make -j1.

 

That being said, if you force a "Release" version, and not the "Debug" version, much less RAM is needed.

 

Kind Regards

 

James

 

 


_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: Out of Memory Building Clang and LLVM

Cameron McInally via cfe-dev

From https://llvm.org/docs/GettingStarted.html

Just finished a clang build on Xubuntu 19.04 using cpp.

cmake -G Ninja -DLLVM_ENABLE_PROJECTS="clang" ../llvm

Memory usage hit a maximum at the link for bin/clang-9 at 13 gigabytes with ninja -j 1.

Disk usage for the build directory was 61 gigabytes.

My sense is that this kinds of usage is more than what a typical new user would expect. Perhaps the page above might be changed to reflect the expected result and with an easier route noted by Erich below.

On 6/20/19 6:56 AM, Keane, Erich via cfe-dev wrote:

Clang definitely hits a pathological case in LD that makes it do terrible things if you use use more than a process or two in DEBUG. 

 

However, if you use “gold” or “LLD” instead, it works fine.  Assuming you have LLD, you can do:

-DLLVM_USE_LINKER=lld or –DLLVM_ENABLE_LLD=ON

 

Otherwise you should have gold available, which you can do with:

-DLLVM_USE_LINKER=gold

 

I can only run about 3-4 links at a time with LD and can do –j30 with no problems with Gold.

 

From: cfe-dev [[hidden email]] On Behalf Of James Courtier-Dutton via cfe-dev
Sent: Thursday, June 20, 2019 3:27 AM
To: nick [hidden email]
Cc: [hidden email] Developers [hidden email]
Subject: Re: [cfe-dev] Out of Memory Building Clang and LLVM

 

 

 

On Wed, 19 Jun 2019 at 23:36, nick via cfe-dev <[hidden email]> wrote:

Greetings,

Seems I'm running out of memory on a host system trying to build llvm, tooling and clang. It was configured
as:
cmake -DLLVM_ENABLE_PROJECTS=clang -G "Unix Makefiles" ../llvm

and build with:

make -j4 using gcc as the host compiler, version 8.

The dmesg I'm getting is related to the linker and not sure if the linker has a bug or it's normal
to expect 3GB or more used for linking clang. Doesn't seem to be but not sure.

dmesg output:
[186517.023397] Out of memory: Kill process 2143 (ld) score 286 or sacrifice child
[186517.023403] Killed process 2143 (ld) total-vm:4622188kB, anon-rss:3202188kB, file-rss:4kB, shmem-rss:0kB

 

HI,

 

I have a 16 Core/32 Thread system with 32GB RAM, and it can have difficulty building CLANG/LLVM.

The link stage definitely needs to be with make -j1.

 

That being said, if you force a "Release" version, and not the "Debug" version, much less RAM is needed.

 

Kind Regards

 

James

 

 


_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev

_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: Out of Memory Building Clang and LLVM

Cameron McInally via cfe-dev
What OS are you guys compiling on?
 
When I compile LLVM in MinSizeRel mode it doesn't take nearly that much ram, but I'm on OSX with ld64.

_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: Out of Memory Building Clang and LLVM

Cameron McInally via cfe-dev
In reply to this post by Cameron McInally via cfe-dev
13GB to link clang and 60+ GB build directory is consistent with my experience building LLVM+Clang on Linux using the GNU linker.

Using lld will dramatically reduce memory usage during linking and will also radically speed up linking. As will reducing the number of LLVM targets (LLVM_TARGETS_TO_BUILD). The documentation on the GettingStarted page is pretty far out of date, but it is tough to give a reasonable estimate given that there are so many configuration values that could impact the size and system resources required for a build.

Since it was brought up in another email, generally speaking users on OS X with ld64 don't encounter these kinds of issues. That is because ld64's algorithm for linking is radically different from the GNU linker which results in shorter link times and lower memory usage.

-Chris

On Jun 20, 2019, at 1:07 PM, Neil Nelson via cfe-dev <[hidden email]> wrote:

From https://llvm.org/docs/GettingStarted.html

Just finished a clang build on Xubuntu 19.04 using cpp.

cmake -G Ninja -DLLVM_ENABLE_PROJECTS="clang" ../llvm

Memory usage hit a maximum at the link for bin/clang-9 at 13 gigabytes with ninja -j 1.

Disk usage for the build directory was 61 gigabytes.

My sense is that this kinds of usage is more than what a typical new user would expect. Perhaps the page above might be changed to reflect the expected result and with an easier route noted by Erich below.

On 6/20/19 6:56 AM, Keane, Erich via cfe-dev wrote:
Clang definitely hits a pathological case in LD that makes it do terrible things if you use use more than a process or two in DEBUG. 
 
However, if you use “gold” or “LLD” instead, it works fine.  Assuming you have LLD, you can do:
-DLLVM_USE_LINKER=lld or –DLLVM_ENABLE_LLD=ON
 
Otherwise you should have gold available, which you can do with:
-DLLVM_USE_LINKER=gold
 
I can only run about 3-4 links at a time with LD and can do –j30 with no problems with Gold.
From: cfe-dev [[hidden email]] On Behalf Of James Courtier-Dutton via cfe-dev
Sent: Thursday, June 20, 2019 3:27 AM
To: nick [hidden email]
Cc: [hidden email] Developers [hidden email]
Subject: Re: [cfe-dev] Out of Memory Building Clang and LLVM
 
 
 
On Wed, 19 Jun 2019 at 23:36, nick via cfe-dev <[hidden email]> wrote:

Greetings, 

Seems I'm running out of memory on a host system trying to build llvm, tooling and clang. It was configured
as:
cmake -DLLVM_ENABLE_PROJECTS=clang -G "Unix Makefiles" ../llvm

and build with:

make -j4 using gcc as the host compiler, version 8.

The dmesg I'm getting is related to the linker and not sure if the linker has a bug or it's normal
to expect 3GB or more used for linking clang. Doesn't seem to be but not sure.

dmesg output:
[186517.023397] Out of memory: Kill process 2143 (ld) score 286 or sacrifice child
[186517.023403] Killed process 2143 (ld) total-vm:4622188kB, anon-rss:3202188kB, file-rss:4kB, shmem-rss:0kB

 
HI,
 
I have a 16 Core/32 Thread system with 32GB RAM, and it can have difficulty building CLANG/LLVM.
The link stage definitely needs to be with make -j1.
 
That being said, if you force a "Release" version, and not the "Debug" version, much less RAM is needed.
 
Kind Regards
 
James
 
 

_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev


_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: Out of Memory Building Clang and LLVM

Cameron McInally via cfe-dev


On Thu, Jun 20, 2019 at 3:26 PM Chris Bieneman via cfe-dev <[hidden email]> wrote:
13GB to link clang and 60+ GB build directory is consistent with my experience building LLVM+Clang on Linux using the GNU linker.

Using lld will dramatically reduce memory usage during linking and will also radically speed up linking. As will reducing the number of LLVM targets (LLVM_TARGETS_TO_BUILD). The documentation on the GettingStarted page is pretty far out of date, but it is tough to give a reasonable estimate given that there are so many configuration values that could impact the size and system resources required for a build.

Since it was brought up in another email, generally speaking users on OS X with ld64 don't encounter these kinds of issues. That is because ld64's algorithm for linking is radically different from the GNU linker which results in shorter link times and lower memory usage.

I imagine the other reason is the debug info model (those sort of numbers of memory usage and build directory size sound like a Debug build to me) - leaving the debug info in the object files - less work for the linker and smaller linked binaries. You can get /some/ of that on Linux if you enable Split DWARF (LLVM_USE_SPLIT_DWARF).
 

-Chris

On Jun 20, 2019, at 1:07 PM, Neil Nelson via cfe-dev <[hidden email]> wrote:

From https://llvm.org/docs/GettingStarted.html

Just finished a clang build on Xubuntu 19.04 using cpp.

cmake -G Ninja -DLLVM_ENABLE_PROJECTS="clang" ../llvm

Memory usage hit a maximum at the link for bin/clang-9 at 13 gigabytes with ninja -j 1.

Disk usage for the build directory was 61 gigabytes.

My sense is that this kinds of usage is more than what a typical new user would expect. Perhaps the page above might be changed to reflect the expected result and with an easier route noted by Erich below.

On 6/20/19 6:56 AM, Keane, Erich via cfe-dev wrote:
Clang definitely hits a pathological case in LD that makes it do terrible things if you use use more than a process or two in DEBUG. 
 
However, if you use “gold” or “LLD” instead, it works fine.  Assuming you have LLD, you can do:
-DLLVM_USE_LINKER=lld or –DLLVM_ENABLE_LLD=ON
 
Otherwise you should have gold available, which you can do with:
-DLLVM_USE_LINKER=gold
 
I can only run about 3-4 links at a time with LD and can do –j30 with no problems with Gold.
From: cfe-dev [[hidden email]] On Behalf Of James Courtier-Dutton via cfe-dev
Sent: Thursday, June 20, 2019 3:27 AM
To: nick [hidden email]
Cc: [hidden email] Developers [hidden email]
Subject: Re: [cfe-dev] Out of Memory Building Clang and LLVM
 
 
 
On Wed, 19 Jun 2019 at 23:36, nick via cfe-dev <[hidden email]> wrote:

Greetings, 

Seems I'm running out of memory on a host system trying to build llvm, tooling and clang. It was configured
as:
cmake -DLLVM_ENABLE_PROJECTS=clang -G "Unix Makefiles" ../llvm

and build with:

make -j4 using gcc as the host compiler, version 8.

The dmesg I'm getting is related to the linker and not sure if the linker has a bug or it's normal
to expect 3GB or more used for linking clang. Doesn't seem to be but not sure.

dmesg output:
[186517.023397] Out of memory: Kill process 2143 (ld) score 286 or sacrifice child
[186517.023403] Killed process 2143 (ld) total-vm:4622188kB, anon-rss:3202188kB, file-rss:4kB, shmem-rss:0kB

 
HI,
 
I have a 16 Core/32 Thread system with 32GB RAM, and it can have difficulty building CLANG/LLVM.
The link stage definitely needs to be with make -j1.
 
That being said, if you force a "Release" version, and not the "Debug" version, much less RAM is needed.
 
Kind Regards
 
James
 
 

_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev

_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev

_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: Out of Memory Building Clang and LLVM

Cameron McInally via cfe-dev
In reply to this post by Cameron McInally via cfe-dev
That sounds like a Debug build.  FreeBSD still selfhosts including llvm
8 on i386 and armv7 but that's a Release build.  Debug builds haven't
been viable for ages on 32-bit systems.

-- Brooks

On Thu, Jun 20, 2019 at 03:25:57PM -0700, Chris Bieneman via cfe-dev wrote:

> 13GB to link clang and 60+ GB build directory is consistent with my experience building LLVM+Clang on Linux using the GNU linker.
>
> Using lld will dramatically reduce memory usage during linking and will also radically speed up linking. As will reducing the number of LLVM targets (LLVM_TARGETS_TO_BUILD). The documentation on the GettingStarted page is pretty far out of date, but it is tough to give a reasonable estimate given that there are so many configuration values that could impact the size and system resources required for a build.
>
> Since it was brought up in another email, generally speaking users on OS X with ld64 don't encounter these kinds of issues. That is because ld64's algorithm for linking is radically different from the GNU linker which results in shorter link times and lower memory usage.
>
> -Chris
>
> > On Jun 20, 2019, at 1:07 PM, Neil Nelson via cfe-dev <[hidden email]> wrote:
> >
> > From https://llvm.org/docs/GettingStarted.html <https://llvm.org/docs/GettingStarted.html>
> > Just finished a clang build on Xubuntu 19.04 using cpp.
> >
> > cmake -G Ninja -DLLVM_ENABLE_PROJECTS="clang" ../llvm
> >
> > Memory usage hit a maximum at the link for bin/clang-9 at 13 gigabytes with ninja -j 1.
> >
> > Disk usage for the build directory was 61 gigabytes.
> >
> > My sense is that this kinds of usage is more than what a typical new user would expect. Perhaps the page above might be changed to reflect the expected result and with an easier route noted by Erich below.
> >
> > On 6/20/19 6:56 AM, Keane, Erich via cfe-dev wrote:
> >> Clang definitely hits a pathological case in LD that makes it do terrible things if you use use more than a process or two in DEBUG.
> >>  
> >> However, if you use ???gold??? or ???LLD??? instead, it works fine.  Assuming you have LLD, you can do:
> >> -DLLVM_USE_LINKER=lld or ???DLLVM_ENABLE_LLD=ON
> >>  
> >> Otherwise you should have gold available, which you can do with:
> >> -DLLVM_USE_LINKER=gold
> >>  
> >> I can only run about 3-4 links at a time with LD and can do ???j30 with no problems with Gold.
> >> ?? <>
> >>  <>From: cfe-dev [mailto:[hidden email] <mailto:[hidden email]>] On Behalf Of James Courtier-Dutton via cfe-dev
> >> Sent: Thursday, June 20, 2019 3:27 AM
> >> To: nick <[hidden email]> <mailto:[hidden email]>
> >> Cc: [hidden email] <mailto:[hidden email]> Developers <[hidden email]> <mailto:[hidden email]>
> >> Subject: Re: [cfe-dev] Out of Memory Building Clang and LLVM
> >>  
> >>  
> >>  
> >> On Wed, 19 Jun 2019 at 23:36, nick via cfe-dev <[hidden email] <mailto:[hidden email]>> wrote:
> >> Greetings,
> >>
> >> Seems I'm running out of memory on a host system trying to build llvm, tooling and clang. It was configured
> >> as:
> >> cmake -DLLVM_ENABLE_PROJECTS=clang -G "Unix Makefiles" ../llvm
> >>
> >> and build with:
> >>
> >> make -j4 using gcc as the host compiler, version 8.
> >>
> >> The dmesg I'm getting is related to the linker and not sure if the linker has a bug or it's normal
> >> to expect 3GB or more used for linking clang. Doesn't seem to be but not sure.
> >>
> >> dmesg output:
> >> [186517.023397] Out of memory: Kill process 2143 (ld) score 286 or sacrifice child
> >> [186517.023403] Killed process 2143 (ld) total-vm:4622188kB, anon-rss:3202188kB, file-rss:4kB, shmem-rss:0kB
> >>
> >>
> >>  
> >> HI,
> >>  
> >> I have a 16 Core/32 Thread system with 32GB RAM, and it can have difficulty building CLANG/LLVM.
> >> The link stage definitely needs to be with make -j1.
> >>  
> >> That being said, if you force a "Release" version, and not the "Debug" version, much less RAM is needed.
> >>  
> >> Kind Regards
> >>  
> >> James
> >>  
> >>  
> >>
> >>
> >> _______________________________________________
> >> cfe-dev mailing list
> >> [hidden email] <mailto:[hidden email]>
> >> https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev <https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev>
> > _______________________________________________
> > cfe-dev mailing list
> > [hidden email] <mailto:[hidden email]>
> > https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev <https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev>
>

> _______________________________________________
> cfe-dev mailing list
> [hidden email]
> https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev


_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev

signature.asc (465 bytes) Download Attachment