speed up clang build after modifying checker code of static analyzer

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

speed up clang build after modifying checker code of static analyzer

Yvan Roux via cfe-dev
Hi,

I'm writing my first clang static analyzer checker. I wrote the scaffold
code to add the checker into clang. But every time when I change the
checker implementation code, the build process after generating the
checker object file is very slow. It's over 4 minutes in my laptop,
which has the i7-6820HQ four core CPU and 32G memory. The generated
clang binary is over 2G. The affected artifacts are as follows:

---------------------

Scanning dependencies of target clangStaticAnalyzerCheckers
[ 95%] Building CXX object
tools/clang/lib/StaticAnalyzer/Checkers/CMakeFiles/clangStaticAnalyzerCheckers.dir/IntegerOverflowChecker.cpp.o
[ 95%] Linking CXX static library
../../../../../lib/libclangStaticAnalyzerCheckers.a
[ 98%] Built target clangStaticAnalyzerCheckers
[ 98%] Linking CXX executable ../../../../bin/arcmt-test
[ 98%] Linking CXX executable ../../../../bin/clang-check
[ 98%] Linking CXX executable ../../../../bin/clang
[ 98%] Linking CXX shared library ../../../../lib/libclang.so
[ 98%] Built target arcmt-test   <--- this seems to take the most time
[ 98%] Built target libclang
[ 98%] Linking CXX executable ../../../../bin/c-arcmt-test
[ 98%] Linking CXX executable ../../../../bin/c-index-test
[ 98%] Built target c-arcmt-test
[ 98%] Built target clang-check
[ 98%] Built target clang
[100%] Built target c-index-test

real    4m23.014s
user    8m4.032s
sys     0m51.992s

---------------------

How can I speed up the build process so that I don't have to wait for
almost 4.5 minute for a single change in the implementation code? I only
work on the checker of the static analyzer.

--
Thanks,
Lou

_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: speed up clang build after modifying checker code of static analyzer

Yvan Roux via cfe-dev
If the clang binary is 2GB, I would recommend building in Release mode with assertions enabled. Linking binaries without debug information is dramatically faster. This means passing -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_ASSERTIONS=ON to cmake.

If you use debug information, I recommend explicitly building one target, the target you are testing (usually clang), until you feel the need to run the complete clang test suite.

On Tue, Sep 18, 2018 at 5:42 PM Lou Wynn via cfe-dev <[hidden email]> wrote:
Hi,

I'm writing my first clang static analyzer checker. I wrote the scaffold
code to add the checker into clang. But every time when I change the
checker implementation code, the build process after generating the
checker object file is very slow. It's over 4 minutes in my laptop,
which has the i7-6820HQ four core CPU and 32G memory. The generated
clang binary is over 2G. The affected artifacts are as follows:

---------------------

Scanning dependencies of target clangStaticAnalyzerCheckers
[ 95%] Building CXX object
tools/clang/lib/StaticAnalyzer/Checkers/CMakeFiles/clangStaticAnalyzerCheckers.dir/IntegerOverflowChecker.cpp.o
[ 95%] Linking CXX static library
../../../../../lib/libclangStaticAnalyzerCheckers.a
[ 98%] Built target clangStaticAnalyzerCheckers
[ 98%] Linking CXX executable ../../../../bin/arcmt-test
[ 98%] Linking CXX executable ../../../../bin/clang-check
[ 98%] Linking CXX executable ../../../../bin/clang
[ 98%] Linking CXX shared library ../../../../lib/libclang.so
[ 98%] Built target arcmt-test   <--- this seems to take the most time
[ 98%] Built target libclang
[ 98%] Linking CXX executable ../../../../bin/c-arcmt-test
[ 98%] Linking CXX executable ../../../../bin/c-index-test
[ 98%] Built target c-arcmt-test
[ 98%] Built target clang-check
[ 98%] Built target clang
[100%] Built target c-index-test

real    4m23.014s
user    8m4.032s
sys     0m51.992s

---------------------

How can I speed up the build process so that I don't have to wait for
almost 4.5 minute for a single change in the implementation code? I only
work on the checker of the static analyzer.

--
Thanks,
Lou

_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev

_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: speed up clang build after modifying checker code of static analyzer

Yvan Roux via cfe-dev

I do need debug information because tracing code is a great way for me to get familiarized with clang and the static analyzer.

I've managed to run "make -j8 clang" at the top of the build directory. However, the final linking process takes too long now. Is there a way to build a thin version that only runs the static analyzer? I think that I can at least cut the code gen part out of the build process but have no idea how. The only command I run is "clang -cc1 -analyze -analyzer-checker=xxx" It'd be great if I could reduce the final object file to few hundred megabytes.

[ 96%] Built target clangStaticAnalyzerCore
Scanning dependencies of target clangStaticAnalyzerCheckers
[ 96%] Building CXX object tools/clang/lib/StaticAnalyzer/Checkers/CMakeFiles/clangStaticAnalyzerCheckers.dir/IntegerOverflowChecker.cpp.o
[ 96%] Linking CXX static library ../../../../../lib/libclangStaticAnalyzerCheckers.a
[100%] Built target clangStaticAnalyzerCheckers
[100%] Linking CXX executable ../../../../bin/clang  <-- this take most time now
[100%] Built target clang

real    2m22.707s
user    2m10.812s
sys     0m14.936s

Thanks,
Lou
On 09/18/2018 06:25 PM, Reid Kleckner wrote:
If the clang binary is 2GB, I would recommend building in Release mode with assertions enabled. Linking binaries without debug information is dramatically faster. This means passing -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_ASSERTIONS=ON to cmake.

If you use debug information, I recommend explicitly building one target, the target you are testing (usually clang), until you feel the need to run the complete clang test suite.

On Tue, Sep 18, 2018 at 5:42 PM Lou Wynn via cfe-dev <[hidden email]> wrote:
Hi,

I'm writing my first clang static analyzer checker. I wrote the scaffold
code to add the checker into clang. But every time when I change the
checker implementation code, the build process after generating the
checker object file is very slow. It's over 4 minutes in my laptop,
which has the i7-6820HQ four core CPU and 32G memory. The generated
clang binary is over 2G. The affected artifacts are as follows:

---------------------

Scanning dependencies of target clangStaticAnalyzerCheckers
[ 95%] Building CXX object
tools/clang/lib/StaticAnalyzer/Checkers/CMakeFiles/clangStaticAnalyzerCheckers.dir/IntegerOverflowChecker.cpp.o
[ 95%] Linking CXX static library
../../../../../lib/libclangStaticAnalyzerCheckers.a
[ 98%] Built target clangStaticAnalyzerCheckers
[ 98%] Linking CXX executable ../../../../bin/arcmt-test
[ 98%] Linking CXX executable ../../../../bin/clang-check
[ 98%] Linking CXX executable ../../../../bin/clang
[ 98%] Linking CXX shared library ../../../../lib/libclang.so
[ 98%] Built target arcmt-test   <--- this seems to take the most time
[ 98%] Built target libclang
[ 98%] Linking CXX executable ../../../../bin/c-arcmt-test
[ 98%] Linking CXX executable ../../../../bin/c-index-test
[ 98%] Built target c-arcmt-test
[ 98%] Built target clang-check
[ 98%] Built target clang
[100%] Built target c-index-test

real    4m23.014s
user    8m4.032s
sys     0m51.992s

---------------------

How can I speed up the build process so that I don't have to wait for
almost 4.5 minute for a single change in the implementation code? I only
work on the checker of the static analyzer.

--
Thanks,
Lou

_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev


_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: speed up clang build after modifying checker code of static analyzer

Yvan Roux via cfe-dev
Hi!

A couple tips I can give:
  * Compile with clang. Make sure you use a release build for this purpose, optimized with LTO. I usually just download a prebuild binary and go with that. Clang tends to be about 2-3 times faster than gcc.
  * For linking, use LLD, a linker part of the LLVM family. Especially with -j8, it will make a significant difference. Your next best option is gnu gold.

Great to hear about a new checker! I'd encourage you to upload your work in progress early on to phabricator, even if it only contains an empty callback, because an entire working solution (if lengthy enough) can drastically increase the time it takes to review it.

I know because I made that mistake :).

Cheers,
Kristóf Umann


On 19 Sep 2018 04:06, "Lou Wynn via cfe-dev" <[hidden email]> wrote:

I do need debug information because tracing code is a great way for me to get familiarized with clang and the static analyzer.

I've managed to run "make -j8 clang" at the top of the build directory. However, the final linking process takes too long now. Is there a way to build a thin version that only runs the static analyzer? I think that I can at least cut the code gen part out of the build process but have no idea how. The only command I run is "clang -cc1 -analyze -analyzer-checker=xxx" It'd be great if I could reduce the final object file to few hundred megabytes.

[ 96%] Built target clangStaticAnalyzerCore


Scanning dependencies of target clangStaticAnalyzerCheckers
[ 96%] Building CXX object tools/clang/lib/StaticAnalyzer/Checkers/CMakeFiles/clangStaticAnalyzerCheckers.dir/IntegerOverflowChecker.cpp.o
[ 96%] Linking CXX static library ../../../../../lib/libclangStaticAnalyzerCheckers.a
[100%] Built target clangStaticAnalyzerCheckers
[100%] Linking CXX executable ../../../../bin/clang  <-- this take most time now
[100%] Built target clang

real    2m22.707s
user    2m10.812s
sys     0m14.936s

Thanks,
Lou
On 09/18/2018 06:25 PM, Reid Kleckner wrote:
If the clang binary is 2GB, I would recommend building in Release mode with assertions enabled. Linking binaries without debug information is dramatically faster. This means passing -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_ASSERTIONS=ON to cmake.

If you use debug information, I recommend explicitly building one target, the target you are testing (usually clang), until you feel the need to run the complete clang test suite.

On Tue, Sep 18, 2018 at 5:42 PM Lou Wynn via cfe-dev <[hidden email]> wrote:
Hi,

I'm writing my first clang static analyzer checker. I wrote the scaffold
code to add the checker into clang. But every time when I change the
checker implementation code, the build process after generating the
checker object file is very slow. It's over 4 minutes in my laptop,
which has the i7-6820HQ four core CPU and 32G memory. The generated
clang binary is over 2G. The affected artifacts are as follows:

---------------------

Scanning dependencies of target clangStaticAnalyzerCheckers
[ 95%] Building CXX object
tools/clang/lib/StaticAnalyzer/Checkers/CMakeFiles/clangStaticAnalyzerCheckers.dir/IntegerOverflowChecker.cpp.o
[ 95%] Linking CXX static library
../../../../../lib/libclangStaticAnalyzerCheckers.a
[ 98%] Built target clangStaticAnalyzerCheckers
[ 98%] Linking CXX executable ../../../../bin/arcmt-test
[ 98%] Linking CXX executable ../../../../bin/clang-check
[ 98%] Linking CXX executable ../../../../bin/clang
[ 98%] Linking CXX shared library ../../../../lib/libclang.so
[ 98%] Built target arcmt-test   <--- this seems to take the most time
[ 98%] Built target libclang
[ 98%] Linking CXX executable ../../../../bin/c-arcmt-test
[ 98%] Linking CXX executable ../../../../bin/c-index-test
[ 98%] Built target c-arcmt-test
[ 98%] Built target clang-check
[ 98%] Built target clang
[100%] Built target c-index-test

real    4m23.014s
user    8m4.032s
sys     0m51.992s

---------------------

How can I speed up the build process so that I don't have to wait for
almost 4.5 minute for a single change in the implementation code? I only
work on the checker of the static analyzer.

--
Thanks,
Lou

_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev


_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev



_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: speed up clang build after modifying checker code of static analyzer

Yvan Roux via cfe-dev
Consider using Split DWARF (there's a cake option for it) if your debugging use cases can handle it. It'll reduce the size of object files and the final binary, should help out the linker a bit.

On Wed., 19 Sep. 2018, 12:03 am Kristóf Umann via cfe-dev, <[hidden email]> wrote:
Hi!

A couple tips I can give:
  * Compile with clang. Make sure you use a release build for this purpose, optimized with LTO. I usually just download a prebuild binary and go with that. Clang tends to be about 2-3 times faster than gcc.
  * For linking, use LLD, a linker part of the LLVM family. Especially with -j8, it will make a significant difference. Your next best option is gnu gold.

Great to hear about a new checker! I'd encourage you to upload your work in progress early on to phabricator, even if it only contains an empty callback, because an entire working solution (if lengthy enough) can drastically increase the time it takes to review it.

I know because I made that mistake :).

Cheers,
Kristóf Umann


On 19 Sep 2018 04:06, "Lou Wynn via cfe-dev" <[hidden email]> wrote:

I do need debug information because tracing code is a great way for me to get familiarized with clang and the static analyzer.

I've managed to run "make -j8 clang" at the top of the build directory. However, the final linking process takes too long now. Is there a way to build a thin version that only runs the static analyzer? I think that I can at least cut the code gen part out of the build process but have no idea how. The only command I run is "clang -cc1 -analyze -analyzer-checker=xxx" It'd be great if I could reduce the final object file to few hundred megabytes.

[ 96%] Built target clangStaticAnalyzerCore


Scanning dependencies of target clangStaticAnalyzerCheckers
[ 96%] Building CXX object tools/clang/lib/StaticAnalyzer/Checkers/CMakeFiles/clangStaticAnalyzerCheckers.dir/IntegerOverflowChecker.cpp.o
[ 96%] Linking CXX static library ../../../../../lib/libclangStaticAnalyzerCheckers.a
[100%] Built target clangStaticAnalyzerCheckers
[100%] Linking CXX executable ../../../../bin/clang  <-- this take most time now
[100%] Built target clang

real    2m22.707s
user    2m10.812s
sys     0m14.936s

Thanks,
Lou
On 09/18/2018 06:25 PM, Reid Kleckner wrote:
If the clang binary is 2GB, I would recommend building in Release mode with assertions enabled. Linking binaries without debug information is dramatically faster. This means passing -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_ASSERTIONS=ON to cmake.

If you use debug information, I recommend explicitly building one target, the target you are testing (usually clang), until you feel the need to run the complete clang test suite.

On Tue, Sep 18, 2018 at 5:42 PM Lou Wynn via cfe-dev <[hidden email]> wrote:
Hi,

I'm writing my first clang static analyzer checker. I wrote the scaffold
code to add the checker into clang. But every time when I change the
checker implementation code, the build process after generating the
checker object file is very slow. It's over 4 minutes in my laptop,
which has the i7-6820HQ four core CPU and 32G memory. The generated
clang binary is over 2G. The affected artifacts are as follows:

---------------------

Scanning dependencies of target clangStaticAnalyzerCheckers
[ 95%] Building CXX object
tools/clang/lib/StaticAnalyzer/Checkers/CMakeFiles/clangStaticAnalyzerCheckers.dir/IntegerOverflowChecker.cpp.o
[ 95%] Linking CXX static library
../../../../../lib/libclangStaticAnalyzerCheckers.a
[ 98%] Built target clangStaticAnalyzerCheckers
[ 98%] Linking CXX executable ../../../../bin/arcmt-test
[ 98%] Linking CXX executable ../../../../bin/clang-check
[ 98%] Linking CXX executable ../../../../bin/clang
[ 98%] Linking CXX shared library ../../../../lib/libclang.so
[ 98%] Built target arcmt-test   <--- this seems to take the most time
[ 98%] Built target libclang
[ 98%] Linking CXX executable ../../../../bin/c-arcmt-test
[ 98%] Linking CXX executable ../../../../bin/c-index-test
[ 98%] Built target c-arcmt-test
[ 98%] Built target clang-check
[ 98%] Built target clang
[100%] Built target c-index-test

real    4m23.014s
user    8m4.032s
sys     0m51.992s

---------------------

How can I speed up the build process so that I don't have to wait for
almost 4.5 minute for a single change in the implementation code? I only
work on the checker of the static analyzer.

--
Thanks,
Lou

_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev


_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev


_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev

_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: speed up clang build after modifying checker code of static analyzer

Yvan Roux via cfe-dev
In reply to this post by Yvan Roux via cfe-dev
On 19/09/18 03:06, Lou Wynn via cfe-dev wrote:

> I do need debug information because tracing code is a great way for me
> to get familiarized with clang and the static analyzer.
>
> I've managed to run "make -j8 clang" at the top of the build directory.
> However, the final linking process takes too long now. Is there a way to
> build a thin version that only runs the static analyzer? I think that I
> can at least cut the code gen part out of the build process but have no
> idea how. The only command I run is "clang -cc1 -analyze
> -analyzer-checker=xxx" It'd be great if I could reduce the final object
> file to few hundred megabytes.

You can run cmake with -DLLVM_BUILD_TARGETS=X86 or even
-DLLVM_BUILD_TARGETS="". You won't be able to generate machine code, but
it should work.

You can also pass -DLLVM_INCLUDE_TESTS=OFF if you want a smaller
buildsystem.

Also make sure you are not building other repos such as
clang-tools-extra.

Thanks,

Stephen.

_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: speed up clang build after modifying checker code of static analyzer

Yvan Roux via cfe-dev
In reply to this post by Yvan Roux via cfe-dev

When I use clang to compile (gcc is fine), I got cmake error:

-- Performing Test HAVE_CXX_ATOMICS_WITHOUT_LIB
-- Performing Test HAVE_CXX_ATOMICS_WITHOUT_LIB - Failed
-- Looking for __atomic_fetch_add_4 in atomic
-- Looking for __atomic_fetch_add_4 in atomic - found
-- Performing Test HAVE_CXX_ATOMICS_WITH_LIB
-- Performing Test HAVE_CXX_ATOMICS_WITH_LIB - Failed
CMake Error at cmake/modules/CheckAtomic.cmake:50 (message):
  Host compiler must support std::atomic!
Call Stack (most recent call first):
  cmake/config-ix.cmake:323 (include)
  CMakeLists.txt:577 (include)


How can I fix it? I checkout source code from the git mirror. I'm running ubuntu 16.04.1 LTS.


Thanks,
Lou
 
On 09/19/2018 12:02 AM, Kristóf Umann wrote:
Hi!

A couple tips I can give:
  * Compile with clang. Make sure you use a release build for this purpose, optimized with LTO. I usually just download a prebuild binary and go with that. Clang tends to be about 2-3 times faster than gcc.
  * For linking, use LLD, a linker part of the LLVM family. Especially with -j8, it will make a significant difference. Your next best option is gnu gold.

Great to hear about a new checker! I'd encourage you to upload your work in progress early on to phabricator, even if it only contains an empty callback, because an entire working solution (if lengthy enough) can drastically increase the time it takes to review it.

I know because I made that mistake :).

Cheers,
Kristóf Umann


On 19 Sep 2018 04:06, "Lou Wynn via cfe-dev" <[hidden email]> wrote:

I do need debug information because tracing code is a great way for me to get familiarized with clang and the static analyzer.

I've managed to run "make -j8 clang" at the top of the build directory. However, the final linking process takes too long now. Is there a way to build a thin version that only runs the static analyzer? I think that I can at least cut the code gen part out of the build process but have no idea how. The only command I run is "clang -cc1 -analyze -analyzer-checker=xxx" It'd be great if I could reduce the final object file to few hundred megabytes.

[ 96%] Built target clangStaticAnalyzerCore


Scanning dependencies of target clangStaticAnalyzerCheckers
[ 96%] Building CXX object tools/clang/lib/StaticAnalyzer/Checkers/CMakeFiles/clangStaticAnalyzerCheckers.dir/IntegerOverflowChecker.cpp.o
[ 96%] Linking CXX static library ../../../../../lib/libclangStaticAnalyzerCheckers.a
[100%] Built target clangStaticAnalyzerCheckers
[100%] Linking CXX executable ../../../../bin/clang  <-- this take most time now
[100%] Built target clang

real    2m22.707s
user    2m10.812s
sys     0m14.936s

Thanks,
Lou
On 09/18/2018 06:25 PM, Reid Kleckner wrote:
If the clang binary is 2GB, I would recommend building in Release mode with assertions enabled. Linking binaries without debug information is dramatically faster. This means passing -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_ASSERTIONS=ON to cmake.

If you use debug information, I recommend explicitly building one target, the target you are testing (usually clang), until you feel the need to run the complete clang test suite.

On Tue, Sep 18, 2018 at 5:42 PM Lou Wynn via cfe-dev <[hidden email]> wrote:
Hi,

I'm writing my first clang static analyzer checker. I wrote the scaffold
code to add the checker into clang. But every time when I change the
checker implementation code, the build process after generating the
checker object file is very slow. It's over 4 minutes in my laptop,
which has the i7-6820HQ four core CPU and 32G memory. The generated
clang binary is over 2G. The affected artifacts are as follows:

---------------------

Scanning dependencies of target clangStaticAnalyzerCheckers
[ 95%] Building CXX object
tools/clang/lib/StaticAnalyzer/Checkers/CMakeFiles/clangStaticAnalyzerCheckers.dir/IntegerOverflowChecker.cpp.o
[ 95%] Linking CXX static library
../../../../../lib/libclangStaticAnalyzerCheckers.a
[ 98%] Built target clangStaticAnalyzerCheckers
[ 98%] Linking CXX executable ../../../../bin/arcmt-test
[ 98%] Linking CXX executable ../../../../bin/clang-check
[ 98%] Linking CXX executable ../../../../bin/clang
[ 98%] Linking CXX shared library ../../../../lib/libclang.so
[ 98%] Built target arcmt-test   <--- this seems to take the most time
[ 98%] Built target libclang
[ 98%] Linking CXX executable ../../../../bin/c-arcmt-test
[ 98%] Linking CXX executable ../../../../bin/c-index-test
[ 98%] Built target c-arcmt-test
[ 98%] Built target clang-check
[ 98%] Built target clang
[100%] Built target c-index-test

real    4m23.014s
user    8m4.032s
sys     0m51.992s

---------------------

How can I speed up the build process so that I don't have to wait for
almost 4.5 minute for a single change in the implementation code? I only
work on the checker of the static analyzer.

--
Thanks,
Lou

_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev


_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev




_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: speed up clang build after modifying checker code of static analyzer

Yvan Roux via cfe-dev
Are you using a prebuild binary? Are you sure you're using the clang you'd like to compile with? (`which clang`?)

On 19 Sep 2018 23:38, "Lou Wynn" <[hidden email]> wrote:

When I use clang to compile (gcc is fine), I got cmake error:

-- Performing Test HAVE_CXX_ATOMICS_WITHOUT_LIB
-- Performing Test HAVE_CXX_ATOMICS_WITHOUT_LIB - Failed
-- Looking for __atomic_fetch_add_4 in atomic
-- Looking for __atomic_fetch_add_4 in atomic - found
-- Performing Test HAVE_CXX_ATOMICS_WITH_LIB
-- Performing Test HAVE_CXX_ATOMICS_WITH_LIB - Failed
CMake Error at cmake/modules/CheckAtomic.cmake:50 (message):
  Host compiler must support std::atomic!
Call Stack (most recent call first):
  cmake/config-ix.cmake:323 (include)
  CMakeLists.txt:577 (include)


How can I fix it? I checkout source code from the git mirror. I'm running ubuntu 16.04.1 LTS.


Thanks,
Lou
 
On 09/19/2018 12:02 AM, Kristóf Umann wrote:
Hi!

A couple tips I can give:
  * Compile with clang. Make sure you use a release build for this purpose, optimized with LTO. I usually just download a prebuild binary and go with that. Clang tends to be about 2-3 times faster than gcc.
  * For linking, use LLD, a linker part of the LLVM family. Especially with -j8, it will make a significant difference. Your next best option is gnu gold.

Great to hear about a new checker! I'd encourage you to upload your work in progress early on to phabricator, even if it only contains an empty callback, because an entire working solution (if lengthy enough) can drastically increase the time it takes to review it.

I know because I made that mistake :).

Cheers,
Kristóf Umann


On 19 Sep 2018 04:06, "Lou Wynn via cfe-dev" <[hidden email]> wrote:

I do need debug information because tracing code is a great way for me to get familiarized with clang and the static analyzer.

I've managed to run "make -j8 clang" at the top of the build directory. However, the final linking process takes too long now. Is there a way to build a thin version that only runs the static analyzer? I think that I can at least cut the code gen part out of the build process but have no idea how. The only command I run is "clang -cc1 -analyze -analyzer-checker=xxx" It'd be great if I could reduce the final object file to few hundred megabytes.

[ 96%] Built target clangStaticAnalyzerCore


Scanning dependencies of target clangStaticAnalyzerCheckers
[ 96%] Building CXX object tools/clang/lib/StaticAnalyzer/Checkers/CMakeFiles/clangStaticAnalyzerCheckers.dir/IntegerOverflowChecker.cpp.o
[ 96%] Linking CXX static library ../../../../../lib/libclangStaticAnalyzerCheckers.a
[100%] Built target clangStaticAnalyzerCheckers
[100%] Linking CXX executable ../../../../bin/clang  <-- this take most time now
[100%] Built target clang

real    2m22.707s
user    2m10.812s
sys     0m14.936s

Thanks,
Lou
On 09/18/2018 06:25 PM, Reid Kleckner wrote:
If the clang binary is 2GB, I would recommend building in Release mode with assertions enabled. Linking binaries without debug information is dramatically faster. This means passing -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_ASSERTIONS=ON to cmake.

If you use debug information, I recommend explicitly building one target, the target you are testing (usually clang), until you feel the need to run the complete clang test suite.

On Tue, Sep 18, 2018 at 5:42 PM Lou Wynn via cfe-dev <[hidden email]> wrote:
Hi,

I'm writing my first clang static analyzer checker. I wrote the scaffold
code to add the checker into clang. But every time when I change the
checker implementation code, the build process after generating the
checker object file is very slow. It's over 4 minutes in my laptop,
which has the i7-6820HQ four core CPU and 32G memory. The generated
clang binary is over 2G. The affected artifacts are as follows:

---------------------

Scanning dependencies of target clangStaticAnalyzerCheckers
[ 95%] Building CXX object
tools/clang/lib/StaticAnalyzer/Checkers/CMakeFiles/clangStaticAnalyzerCheckers.dir/IntegerOverflowChecker.cpp.o
[ 95%] Linking CXX static library
../../../../../lib/libclangStaticAnalyzerCheckers.a
[ 98%] Built target clangStaticAnalyzerCheckers
[ 98%] Linking CXX executable ../../../../bin/arcmt-test
[ 98%] Linking CXX executable ../../../../bin/clang-check
[ 98%] Linking CXX executable ../../../../bin/clang
[ 98%] Linking CXX shared library ../../../../lib/libclang.so
[ 98%] Built target arcmt-test   <--- this seems to take the most time
[ 98%] Built target libclang
[ 98%] Linking CXX executable ../../../../bin/c-arcmt-test
[ 98%] Linking CXX executable ../../../../bin/c-index-test
[ 98%] Built target c-arcmt-test
[ 98%] Built target clang-check
[ 98%] Built target clang
[100%] Built target c-index-test

real    4m23.014s
user    8m4.032s
sys     0m51.992s

---------------------

How can I speed up the build process so that I don't have to wait for
almost 4.5 minute for a single change in the implementation code? I only
work on the checker of the static analyzer.

--
Thanks,
Lou

_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev


_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev




_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: speed up clang build after modifying checker code of static analyzer

Yvan Roux via cfe-dev

~/work/git-repos/llvm/release-git$ which clang
/usr/bin/clang

~/work/git-repos/llvm/release-git$ clang -###
clang version 3.8.0-2ubuntu4 (tags/RELEASE_380/final)
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin


I also tried with a self-built clang, and got the same error.


Thanks,
Lou

On 09/19/2018 02:41 PM, Kristóf Umann wrote:
Are you using a prebuild binary? Are you sure you're using the clang you'd like to compile with? (`which clang`?)

On 19 Sep 2018 23:38, "Lou Wynn" <[hidden email]> wrote:

When I use clang to compile (gcc is fine), I got cmake error:

-- Performing Test HAVE_CXX_ATOMICS_WITHOUT_LIB
-- Performing Test HAVE_CXX_ATOMICS_WITHOUT_LIB - Failed
-- Looking for __atomic_fetch_add_4 in atomic
-- Looking for __atomic_fetch_add_4 in atomic - found
-- Performing Test HAVE_CXX_ATOMICS_WITH_LIB
-- Performing Test HAVE_CXX_ATOMICS_WITH_LIB - Failed
CMake Error at cmake/modules/CheckAtomic.cmake:50 (message):
  Host compiler must support std::atomic!
Call Stack (most recent call first):
  cmake/config-ix.cmake:323 (include)
  CMakeLists.txt:577 (include)


How can I fix it? I checkout source code from the git mirror. I'm running ubuntu 16.04.1 LTS.


Thanks,
Lou
 
On 09/19/2018 12:02 AM, Kristóf Umann wrote:
Hi!

A couple tips I can give:
  * Compile with clang. Make sure you use a release build for this purpose, optimized with LTO. I usually just download a prebuild binary and go with that. Clang tends to be about 2-3 times faster than gcc.
  * For linking, use LLD, a linker part of the LLVM family. Especially with -j8, it will make a significant difference. Your next best option is gnu gold.

Great to hear about a new checker! I'd encourage you to upload your work in progress early on to phabricator, even if it only contains an empty callback, because an entire working solution (if lengthy enough) can drastically increase the time it takes to review it.

I know because I made that mistake :).

Cheers,
Kristóf Umann


On 19 Sep 2018 04:06, "Lou Wynn via cfe-dev" <[hidden email]> wrote:

I do need debug information because tracing code is a great way for me to get familiarized with clang and the static analyzer.

I've managed to run "make -j8 clang" at the top of the build directory. However, the final linking process takes too long now. Is there a way to build a thin version that only runs the static analyzer? I think that I can at least cut the code gen part out of the build process but have no idea how. The only command I run is "clang -cc1 -analyze -analyzer-checker=xxx" It'd be great if I could reduce the final object file to few hundred megabytes.

[ 96%] Built target clangStaticAnalyzerCore


Scanning dependencies of target clangStaticAnalyzerCheckers
[ 96%] Building CXX object tools/clang/lib/StaticAnalyzer/Checkers/CMakeFiles/clangStaticAnalyzerCheckers.dir/IntegerOverflowChecker.cpp.o
[ 96%] Linking CXX static library ../../../../../lib/libclangStaticAnalyzerCheckers.a
[100%] Built target clangStaticAnalyzerCheckers
[100%] Linking CXX executable ../../../../bin/clang  <-- this take most time now
[100%] Built target clang

real    2m22.707s
user    2m10.812s
sys     0m14.936s

Thanks,
Lou
On 09/18/2018 06:25 PM, Reid Kleckner wrote:
If the clang binary is 2GB, I would recommend building in Release mode with assertions enabled. Linking binaries without debug information is dramatically faster. This means passing -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_ASSERTIONS=ON to cmake.

If you use debug information, I recommend explicitly building one target, the target you are testing (usually clang), until you feel the need to run the complete clang test suite.

On Tue, Sep 18, 2018 at 5:42 PM Lou Wynn via cfe-dev <[hidden email]> wrote:
Hi,

I'm writing my first clang static analyzer checker. I wrote the scaffold
code to add the checker into clang. But every time when I change the
checker implementation code, the build process after generating the
checker object file is very slow. It's over 4 minutes in my laptop,
which has the i7-6820HQ four core CPU and 32G memory. The generated
clang binary is over 2G. The affected artifacts are as follows:

---------------------

Scanning dependencies of target clangStaticAnalyzerCheckers
[ 95%] Building CXX object
tools/clang/lib/StaticAnalyzer/Checkers/CMakeFiles/clangStaticAnalyzerCheckers.dir/IntegerOverflowChecker.cpp.o
[ 95%] Linking CXX static library
../../../../../lib/libclangStaticAnalyzerCheckers.a
[ 98%] Built target clangStaticAnalyzerCheckers
[ 98%] Linking CXX executable ../../../../bin/arcmt-test
[ 98%] Linking CXX executable ../../../../bin/clang-check
[ 98%] Linking CXX executable ../../../../bin/clang
[ 98%] Linking CXX shared library ../../../../lib/libclang.so
[ 98%] Built target arcmt-test   <--- this seems to take the most time
[ 98%] Built target libclang
[ 98%] Linking CXX executable ../../../../bin/c-arcmt-test
[ 98%] Linking CXX executable ../../../../bin/c-index-test
[ 98%] Built target c-arcmt-test
[ 98%] Built target clang-check
[ 98%] Built target clang
[100%] Built target c-index-test

real    4m23.014s
user    8m4.032s
sys     0m51.992s

---------------------

How can I speed up the build process so that I don't have to wait for
almost 4.5 minute for a single change in the implementation code? I only
work on the checker of the static analyzer.

--
Thanks,
Lou

_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev


_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev





_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: speed up clang build after modifying checker code of static analyzer

Yvan Roux via cfe-dev
Uhh. I never encountered that error, so I can't offer anything that google can not :( 

All the other tips however others and I gave should speed up compilation and linking, so gcc is fine if you are having a lot of trouble building with clang, especially if linking is the most ime consuming part. Though it would be beneficial to get to the bottom of this if you have the time.

One more thing, building shared libraries will make linking a whole lot faster (the resulting binary will be slower, but that won't hurt much in a debug build anyway) -DBUILD_SHARED_LIBS=ON.

On 19 Sep 2018 23:44, "Lou Wynn" <[hidden email]> wrote:

~/work/git-repos/llvm/release-git$ which clang
/usr/bin/clang

~/work/git-repos/llvm/release-git$ clang -###
clang version 3.8.0-2ubuntu4 (tags/RELEASE_380/final)
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin


I also tried with a self-built clang, and got the same error.


Thanks,
Lou

On 09/19/2018 02:41 PM, Kristóf Umann wrote:
Are you using a prebuild binary? Are you sure you're using the clang you'd like to compile with? (`which clang`?)

On 19 Sep 2018 23:38, "Lou Wynn" <[hidden email]> wrote:

When I use clang to compile (gcc is fine), I got cmake error:

-- Performing Test HAVE_CXX_ATOMICS_WITHOUT_LIB
-- Performing Test HAVE_CXX_ATOMICS_WITHOUT_LIB - Failed
-- Looking for __atomic_fetch_add_4 in atomic
-- Looking for __atomic_fetch_add_4 in atomic - found
-- Performing Test HAVE_CXX_ATOMICS_WITH_LIB
-- Performing Test HAVE_CXX_ATOMICS_WITH_LIB - Failed
CMake Error at cmake/modules/CheckAtomic.cmake:50 (message):
  Host compiler must support std::atomic!
Call Stack (most recent call first):
  cmake/config-ix.cmake:323 (include)
  CMakeLists.txt:577 (include)


How can I fix it? I checkout source code from the git mirror. I'm running ubuntu 16.04.1 LTS.


Thanks,
Lou
 
On 09/19/2018 12:02 AM, Kristóf Umann wrote:
Hi!

A couple tips I can give:
  * Compile with clang. Make sure you use a release build for this purpose, optimized with LTO. I usually just download a prebuild binary and go with that. Clang tends to be about 2-3 times faster than gcc.
  * For linking, use LLD, a linker part of the LLVM family. Especially with -j8, it will make a significant difference. Your next best option is gnu gold.

Great to hear about a new checker! I'd encourage you to upload your work in progress early on to phabricator, even if it only contains an empty callback, because an entire working solution (if lengthy enough) can drastically increase the time it takes to review it.

I know because I made that mistake :).

Cheers,
Kristóf Umann


On 19 Sep 2018 04:06, "Lou Wynn via cfe-dev" <[hidden email]> wrote:

I do need debug information because tracing code is a great way for me to get familiarized with clang and the static analyzer.

I've managed to run "make -j8 clang" at the top of the build directory. However, the final linking process takes too long now. Is there a way to build a thin version that only runs the static analyzer? I think that I can at least cut the code gen part out of the build process but have no idea how. The only command I run is "clang -cc1 -analyze -analyzer-checker=xxx" It'd be great if I could reduce the final object file to few hundred megabytes.

[ 96%] Built target clangStaticAnalyzerCore


Scanning dependencies of target clangStaticAnalyzerCheckers
[ 96%] Building CXX object tools/clang/lib/StaticAnalyzer/Checkers/CMakeFiles/clangStaticAnalyzerCheckers.dir/IntegerOverflowChecker.cpp.o
[ 96%] Linking CXX static library ../../../../../lib/libclangStaticAnalyzerCheckers.a
[100%] Built target clangStaticAnalyzerCheckers
[100%] Linking CXX executable ../../../../bin/clang  <-- this take most time now
[100%] Built target clang

real    2m22.707s
user    2m10.812s
sys     0m14.936s

Thanks,
Lou
On 09/18/2018 06:25 PM, Reid Kleckner wrote:
If the clang binary is 2GB, I would recommend building in Release mode with assertions enabled. Linking binaries without debug information is dramatically faster. This means passing -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_ASSERTIONS=ON to cmake.

If you use debug information, I recommend explicitly building one target, the target you are testing (usually clang), until you feel the need to run the complete clang test suite.

On Tue, Sep 18, 2018 at 5:42 PM Lou Wynn via cfe-dev <[hidden email]> wrote:
Hi,

I'm writing my first clang static analyzer checker. I wrote the scaffold
code to add the checker into clang. But every time when I change the
checker implementation code, the build process after generating the
checker object file is very slow. It's over 4 minutes in my laptop,
which has the i7-6820HQ four core CPU and 32G memory. The generated
clang binary is over 2G. The affected artifacts are as follows:

---------------------

Scanning dependencies of target clangStaticAnalyzerCheckers
[ 95%] Building CXX object
tools/clang/lib/StaticAnalyzer/Checkers/CMakeFiles/clangStaticAnalyzerCheckers.dir/IntegerOverflowChecker.cpp.o
[ 95%] Linking CXX static library
../../../../../lib/libclangStaticAnalyzerCheckers.a
[ 98%] Built target clangStaticAnalyzerCheckers
[ 98%] Linking CXX executable ../../../../bin/arcmt-test
[ 98%] Linking CXX executable ../../../../bin/clang-check
[ 98%] Linking CXX executable ../../../../bin/clang
[ 98%] Linking CXX shared library ../../../../lib/libclang.so
[ 98%] Built target arcmt-test   <--- this seems to take the most time
[ 98%] Built target libclang
[ 98%] Linking CXX executable ../../../../bin/c-arcmt-test
[ 98%] Linking CXX executable ../../../../bin/c-index-test
[ 98%] Built target c-arcmt-test
[ 98%] Built target clang-check
[ 98%] Built target clang
[100%] Built target c-index-test

real    4m23.014s
user    8m4.032s
sys     0m51.992s

---------------------

How can I speed up the build process so that I don't have to wait for
almost 4.5 minute for a single change in the implementation code? I only
work on the checker of the static analyzer.

--
Thanks,
Lou

_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev


_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev






_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: speed up clang build after modifying checker code of static analyzer

Yvan Roux via cfe-dev
In reply to this post by Yvan Roux via cfe-dev
(whoops re-adding cfe-dev)

clang-3.8 is ancient. I guess you'll need a newer clang (and probably a
newer libc++ if you're using it instead of libstdc++, because lack of
std::atomic support might be a standard library problem).

But for me on linux the -DBUILD_SHARED_LIBS=1 flag was pretty much the
answer. So i don't think you should dig too much into compiling with
clang unless you care about compilation time as much as linking time,
which is unlikely.

Also see if you're running out of memory while linking. If you're
swapping, it will be slow. Linking is the only memory-intensive part of
the build.

On 9/19/18 2:44 PM, Lou Wynn via cfe-dev wrote:

>
> ~/work/git-repos/llvm/release-git$ which clang
> /usr/bin/clang
>
> ~/work/git-repos/llvm/release-git$ clang -###
> clang version 3.8.0-2ubuntu4 (tags/RELEASE_380/final)
> Target: x86_64-pc-linux-gnu
> Thread model: posix
> InstalledDir: /usr/bin
>
>
> I also tried with a self-built clang, and got the same error.
>
>
> Thanks,
> Lou
>
> On 09/19/2018 02:41 PM, Kristóf Umann wrote:
>> Are you using a prebuild binary? Are you sure you're using the clang
>> you'd like to compile with? (`which clang`?)
>>
>> On 19 Sep 2018 23:38, "Lou Wynn" <[hidden email]
>> <mailto:[hidden email]>> wrote:
>>
>>     When I use clang to compile (gcc is fine), I got cmake error:
>>
>>     -- Performing Test HAVE_CXX_ATOMICS_WITHOUT_LIB
>>     -- Performing Test HAVE_CXX_ATOMICS_WITHOUT_LIB - Failed
>>     -- Looking for __atomic_fetch_add_4 in atomic
>>     -- Looking for __atomic_fetch_add_4 in atomic - found
>>     -- Performing Test HAVE_CXX_ATOMICS_WITH_LIB
>>     -- Performing Test HAVE_CXX_ATOMICS_WITH_LIB - Failed
>>     CMake Error at cmake/modules/CheckAtomic.cmake:50 (message):
>>       Host compiler must support std::atomic!
>>     Call Stack (most recent call first):
>>       cmake/config-ix.cmake:323 (include)
>>       CMakeLists.txt:577 (include)
>>
>>
>>     How can I fix it? I checkout source code from the git mirror. I'm
>>     running ubuntu 16.04.1 LTS.
>>
>>
>>     Thanks,
>>     Lou
>>      
>>
>>     On 09/19/2018 12:02 AM, Kristóf Umann wrote:
>>>     Hi!
>>>
>>>     A couple tips I can give:
>>>       * Compile with clang. Make sure you use a release build for
>>>     this purpose, optimized with LTO. I usually just download a
>>>     prebuild binary and go with that. Clang tends to be about 2-3
>>>     times faster than gcc.
>>>       * For linking, use LLD, a linker part of the LLVM family.
>>>     Especially with -j8, it will make a significant difference. Your
>>>     next best option is gnu gold.
>>>
>>>     Great to hear about a new checker! I'd encourage you to upload
>>>     your work in progress early on to phabricator, even if it only
>>>     contains an empty callback, because an entire working solution
>>>     (if lengthy enough) can drastically increase the time it takes
>>>     to review it.
>>>
>>>     I know because I made that mistake :).
>>>
>>>     Cheers,
>>>     Kristóf Umann
>>>
>>>
>>>     On 19 Sep 2018 04:06, "Lou Wynn via cfe-dev"
>>>     <[hidden email] <mailto:[hidden email]>> wrote:
>>>
>>>         I do need debug information because tracing code is a great
>>>         way for me to get familiarized with clang and the static
>>>         analyzer.
>>>
>>>         I've managed to run "make -j8 clang" at the top of the build
>>>         directory. However, the final linking process takes too long
>>>         now. Is there a way to build a thin version that only runs
>>>         the static analyzer? I think that I can at least cut the
>>>         code gen part out of the build process but have no idea how.
>>>         The only command I run is "clang -cc1 -analyze
>>>         -analyzer-checker=xxx" It'd be great if I could reduce the
>>>         final object file to few hundred megabytes.
>>>
>>>         [ 96%] Built target clangStaticAnalyzerCore
>>>
>>>
>>>         Scanning dependencies of target clangStaticAnalyzerCheckers
>>>         [ 96%] Building CXX object
>>>         tools/clang/lib/StaticAnalyzer/Checkers/CMakeFiles/clangStaticAnalyzerCheckers.dir/IntegerOverflowChecker.cpp.o
>>>         [ 96%] Linking CXX static library
>>>         ../../../../../lib/libclangStaticAnalyzerCheckers.a
>>>         [100%] Built target clangStaticAnalyzerCheckers
>>>         [100%] Linking CXX executable ../../../../bin/clang  <--
>>>         this take most time now
>>>         [100%] Built target clang
>>>
>>>         real    2m22.707s
>>>         user    2m10.812s
>>>         sys     0m14.936s
>>>
>>>         Thanks,
>>>         Lou
>>>
>>>         On 09/18/2018 06:25 PM, Reid Kleckner wrote:
>>>>         If the clang binary is 2GB, I would recommend building in
>>>>         Release mode with assertions enabled. Linking binaries
>>>>         without debug information is dramatically faster. This
>>>>         means passing -DCMAKE_BUILD_TYPE=Release
>>>>         -DLLVM_ENABLE_ASSERTIONS=ON to cmake.
>>>>
>>>>         If you use debug information, I recommend explicitly
>>>>         building one target, the target you are testing (usually
>>>>         clang), until you feel the need to run the complete clang
>>>>         test suite.
>>>>
>>>>         On Tue, Sep 18, 2018 at 5:42 PM Lou Wynn via cfe-dev
>>>>         <[hidden email] <mailto:[hidden email]>> wrote:
>>>>
>>>>             Hi,
>>>>
>>>>             I'm writing my first clang static analyzer checker. I
>>>>             wrote the scaffold
>>>>             code to add the checker into clang. But every time when
>>>>             I change the
>>>>             checker implementation code, the build process after
>>>>             generating the
>>>>             checker object file is very slow. It's over 4 minutes
>>>>             in my laptop,
>>>>             which has the i7-6820HQ four core CPU and 32G memory.
>>>>             The generated
>>>>             clang binary is over 2G. The affected artifacts are as
>>>>             follows:
>>>>
>>>>             ---------------------
>>>>
>>>>             Scanning dependencies of target clangStaticAnalyzerCheckers
>>>>             [ 95%] Building CXX object
>>>>             tools/clang/lib/StaticAnalyzer/Checkers/CMakeFiles/clangStaticAnalyzerCheckers.dir/IntegerOverflowChecker.cpp.o
>>>>             [ 95%] Linking CXX static library
>>>>             ../../../../../lib/libclangStaticAnalyzerCheckers.a
>>>>             [ 98%] Built target clangStaticAnalyzerCheckers
>>>>             [ 98%] Linking CXX executable ../../../../bin/arcmt-test
>>>>             [ 98%] Linking CXX executable ../../../../bin/clang-check
>>>>             [ 98%] Linking CXX executable ../../../../bin/clang
>>>>             [ 98%] Linking CXX shared library
>>>>             ../../../../lib/libclang.so
>>>>             [ 98%] Built target arcmt-test <--- this seems to take
>>>>             the most time
>>>>             [ 98%] Built target libclang
>>>>             [ 98%] Linking CXX executable ../../../../bin/c-arcmt-test
>>>>             [ 98%] Linking CXX executable ../../../../bin/c-index-test
>>>>             [ 98%] Built target c-arcmt-test
>>>>             [ 98%] Built target clang-check
>>>>             [ 98%] Built target clang
>>>>             [100%] Built target c-index-test
>>>>
>>>>             real    4m23.014s
>>>>             user    8m4.032s
>>>>             sys     0m51.992s
>>>>
>>>>             ---------------------
>>>>
>>>>             How can I speed up the build process so that I don't
>>>>             have to wait for
>>>>             almost 4.5 minute for a single change in the
>>>>             implementation code? I only
>>>>             work on the checker of the static analyzer.
>>>>
>>>>             --
>>>>             Thanks,
>>>>             Lou
>>>>
>>>>             _______________________________________________
>>>>             cfe-dev mailing list
>>>>             [hidden email] <mailto:[hidden email]>
>>>>             http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>>>>             <http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev>
>>>>
>>>
>>>
>>>         _______________________________________________
>>>         cfe-dev mailing list
>>>         [hidden email] <mailto:[hidden email]>
>>>         http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>>>         <http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev>
>>>
>>>
>>
>
>
>
> _______________________________________________
> cfe-dev mailing list
> [hidden email]
> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev

_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: speed up clang build after modifying checker code of static analyzer

Yvan Roux via cfe-dev
In reply to this post by Yvan Roux via cfe-dev

After doing lots of experiments, ranging from installing clang-6 from its official build (http://apt.llvm.org/xenial) to checking Cmake failure logs and tweaking its settings, I've finally happened to find out what was wrong.

I used

-DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang

when I ran CMake because I saw clang and clang++ are symlinks to bin/clang-x. This caused the Cmake failure, and it was gone when I happened to use:

-DCMAKE_CXX_COMPILER=clang++-x

My guess is that the clang code must check which original command was used to invoke it. When it's invoked through the clang symlink, it acts like the C compiler; when through the clang++ symlink, like the C++ compiler.

Thank all of you who helped. Now with clang++-6.0 and lld-6.0, I can build clang in 9s after changing a checker. Yeh! The entire cmake command line looks like this:

cmake -G "Unix Makefiles" -DLLVM_OPTIMIZED_TABLEGEN=true -DCMAKE_BUILD_TYPE=Debug -DBUILD_SHARED_LIBS=ON ../llvm/ -DLLVM_USE_LINKER=lld-6.0 -DCMAKE_C_COMPILER=clang-6.0 -DCMAKE_CXX_COMPILER=clang++-6.0

Thanks,
Lou
 
On 09/19/2018 02:58 PM, Kristóf Umann wrote:
Uhh. I never encountered that error, so I can't offer anything that google can not :( 

All the other tips however others and I gave should speed up compilation and linking, so gcc is fine if you are having a lot of trouble building with clang, especially if linking is the most ime consuming part. Though it would be beneficial to get to the bottom of this if you have the time.

One more thing, building shared libraries will make linking a whole lot faster (the resulting binary will be slower, but that won't hurt much in a debug build anyway) -DBUILD_SHARED_LIBS=ON.

On 19 Sep 2018 23:44, "Lou Wynn" <[hidden email]> wrote:

~/work/git-repos/llvm/release-git$ which clang
/usr/bin/clang

~/work/git-repos/llvm/release-git$ clang -###
clang version 3.8.0-2ubuntu4 (tags/RELEASE_380/final)
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin


I also tried with a self-built clang, and got the same error.


Thanks,
Lou

On 09/19/2018 02:41 PM, Kristóf Umann wrote:
Are you using a prebuild binary? Are you sure you're using the clang you'd like to compile with? (`which clang`?)

On 19 Sep 2018 23:38, "Lou Wynn" <[hidden email]> wrote:

When I use clang to compile (gcc is fine), I got cmake error:

-- Performing Test HAVE_CXX_ATOMICS_WITHOUT_LIB
-- Performing Test HAVE_CXX_ATOMICS_WITHOUT_LIB - Failed
-- Looking for __atomic_fetch_add_4 in atomic
-- Looking for __atomic_fetch_add_4 in atomic - found
-- Performing Test HAVE_CXX_ATOMICS_WITH_LIB
-- Performing Test HAVE_CXX_ATOMICS_WITH_LIB - Failed
CMake Error at cmake/modules/CheckAtomic.cmake:50 (message):
  Host compiler must support std::atomic!
Call Stack (most recent call first):
  cmake/config-ix.cmake:323 (include)
  CMakeLists.txt:577 (include)


How can I fix it? I checkout source code from the git mirror. I'm running ubuntu 16.04.1 LTS.


Thanks,
Lou
 
On 09/19/2018 12:02 AM, Kristóf Umann wrote:
Hi!

A couple tips I can give:
  * Compile with clang. Make sure you use a release build for this purpose, optimized with LTO. I usually just download a prebuild binary and go with that. Clang tends to be about 2-3 times faster than gcc.
  * For linking, use LLD, a linker part of the LLVM family. Especially with -j8, it will make a significant difference. Your next best option is gnu gold.

Great to hear about a new checker! I'd encourage you to upload your work in progress early on to phabricator, even if it only contains an empty callback, because an entire working solution (if lengthy enough) can drastically increase the time it takes to review it.

I know because I made that mistake :).

Cheers,
Kristóf Umann


On 19 Sep 2018 04:06, "Lou Wynn via cfe-dev" <[hidden email]> wrote:

I do need debug information because tracing code is a great way for me to get familiarized with clang and the static analyzer.

I've managed to run "make -j8 clang" at the top of the build directory. However, the final linking process takes too long now. Is there a way to build a thin version that only runs the static analyzer? I think that I can at least cut the code gen part out of the build process but have no idea how. The only command I run is "clang -cc1 -analyze -analyzer-checker=xxx" It'd be great if I could reduce the final object file to few hundred megabytes.

[ 96%] Built target clangStaticAnalyzerCore


Scanning dependencies of target clangStaticAnalyzerCheckers
[ 96%] Building CXX object tools/clang/lib/StaticAnalyzer/Checkers/CMakeFiles/clangStaticAnalyzerCheckers.dir/IntegerOverflowChecker.cpp.o
[ 96%] Linking CXX static library ../../../../../lib/libclangStaticAnalyzerCheckers.a
[100%] Built target clangStaticAnalyzerCheckers
[100%] Linking CXX executable ../../../../bin/clang  <-- this take most time now
[100%] Built target clang

real    2m22.707s
user    2m10.812s
sys     0m14.936s

Thanks,
Lou
On 09/18/2018 06:25 PM, Reid Kleckner wrote:
If the clang binary is 2GB, I would recommend building in Release mode with assertions enabled. Linking binaries without debug information is dramatically faster. This means passing -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_ASSERTIONS=ON to cmake.

If you use debug information, I recommend explicitly building one target, the target you are testing (usually clang), until you feel the need to run the complete clang test suite.

On Tue, Sep 18, 2018 at 5:42 PM Lou Wynn via cfe-dev <[hidden email]> wrote:
Hi,

I'm writing my first clang static analyzer checker. I wrote the scaffold
code to add the checker into clang. But every time when I change the
checker implementation code, the build process after generating the
checker object file is very slow. It's over 4 minutes in my laptop,
which has the i7-6820HQ four core CPU and 32G memory. The generated
clang binary is over 2G. The affected artifacts are as follows:

---------------------

Scanning dependencies of target clangStaticAnalyzerCheckers
[ 95%] Building CXX object
tools/clang/lib/StaticAnalyzer/Checkers/CMakeFiles/clangStaticAnalyzerCheckers.dir/IntegerOverflowChecker.cpp.o
[ 95%] Linking CXX static library
../../../../../lib/libclangStaticAnalyzerCheckers.a
[ 98%] Built target clangStaticAnalyzerCheckers
[ 98%] Linking CXX executable ../../../../bin/arcmt-test
[ 98%] Linking CXX executable ../../../../bin/clang-check
[ 98%] Linking CXX executable ../../../../bin/clang
[ 98%] Linking CXX shared library ../../../../lib/libclang.so
[ 98%] Built target arcmt-test   <--- this seems to take the most time
[ 98%] Built target libclang
[ 98%] Linking CXX executable ../../../../bin/c-arcmt-test
[ 98%] Linking CXX executable ../../../../bin/c-index-test
[ 98%] Built target c-arcmt-test
[ 98%] Built target clang-check
[ 98%] Built target clang
[100%] Built target c-index-test

real    4m23.014s
user    8m4.032s
sys     0m51.992s

---------------------

How can I speed up the build process so that I don't have to wait for
almost 4.5 minute for a single change in the implementation code? I only
work on the checker of the static analyzer.

--
Thanks,
Lou

_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev


_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev







_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev