Content ======= 0 Testenvironment A Preparing the buildsystem A.1 uncached A.1.1 uncached, cmake A.1.2 uncached, current system A.1.3 uncached, summary A.2 cached A.2.1 cached, cmake A.2.2 cached, current system A.2.3 cached, summary B Doing the build B.1 cmake B.2 current system B.3 summary C Doing the build with cached results C.1 cmake C.2 current system C.3 summary C.3.1 summary, non-parallel build C.3.2 summary, -j6 build C.3.3 summary, -j2 build D Empty builds D.1 cmake D.2 current system D.3 summary F Complexity F.1 cmake F.2 current system F.3 summary G Summary H Features H.1 cmake H.2 current system 0 Testenvironment ================= Tests below were done on a * P4, 2.60GHz HT * 1GB RAM * Linux 2.6.17.11 * Fedora Core 5 with a recent (2006-09-20) CVS snapshot of xmlrpc-c. 'cmake' patches were taken from http://ensc.de/xmlrpc-c/ and cmake-2.4.3 was used. When 'slowdown' numbers are given, this means the time needed with the current system compared with the time needed with 'cmake'. Time with 'cmake' will be assumed as 100%. Most tests were executed multiple times; results shown in this text are usually written as object AVERAGE | RESULT0 RESULT1 RESULT2 ... A Preparing the buildsystem ============================ This benchmark prepares the buildsystem out of the CVS repository, so that 'make' can be executed. For 'cmake' usually the 'cmake' command will be executed. For the current system, 'autoreconf' must be executed once and './configure' later. This case is interesting when final package will be built and when the developer adds new sources or build-time tests. A.1 uncached ============ This benchmark does not use cached information A.1.1 uncached, cmake --------------------- $ time cmake ../repo -DCMAKE_CXX_FLAGS='-O2 -g' -DCMAKE_C_FLAGS='-O2 -g' $ rm -rf * real 0m1.549s | 0m1.708s 0m1.463s 0m1.478s user 0m0.960s | 0m1.004s 0m0.916s 0m0.960s sys 0m0.478s | 0m0.552s 0m0.452s 0m0.432s A.1.2 uncached, current system ------------------------------ $ time autoreconf -i -f real 0m15.174s | 0m15.911s 0m13.924s 0m15.687s user 0m13.239s | 0m14.581s 0m12.573s 0m12.565s sys 0m0.529s | 0m0.488s 0m0.552s 0m0.548s $ time CXXFLAGS='-O2 -g' CFLAGS='-O2 -g' ./configure real 0m9.724s | 0m11.170s 0m8.988s 0m9.015s user 0m3.682s | 0m4.064s 0m3.476s 0m3.508s sys 0m4.288s | 0m4.564s 0m4.164s 0m4.136s A.1.3 uncached, summary ----------------------- cmake | current system slowdown | sum = autoreconf + configure real 0m1.549s | 0m24.898s = 0m15.174s 0m9.724s 1600% user 0m0.960s | 0m16.921s = 0m13.239s 0m3.682s 1760% sys 0m0.478s | 0m4.817s = 0m0.529s 0m4.288s 1000% A.2 cached ========== This benchmark uses cached information; this case happens usually more often then the uncached one e.g. when developer changes the Makefile or CMakeLists.txt. For the current system, 'autoreconf' is usually not needed but only when configure.in was changed. This case will be ignored therefore. A.2.1 cached, cmake ------------------- $ time cmake ../repo -DCMAKE_CXX_FLAGS='-O2 -g' -DCMAKE_C_FLAGS='-O2 -g' $ # *no* rm -rf * real 0m0.710s | 0m0.711s 0m0.716s 0m0.705s user 0m0.496s | 0m0.476s 0m0.512s 0m0.500s sys 0m0.193s | 0m0.216s 0m0.184s 0m0.180s A.2.2 cached, current system ---------------------------- $ time CXXFLAGS='-O2 -g' CFLAGS='-O2 -g' ./configure -C real 0m5.135s | 0m5.108s 0m5.142s 0m5.156s user 0m1.720s | 0m1.648s 0m1.720s 0m1.792s sys 0m2.206s | 0m2.280s 0m2.196s 0m2.144s A.2.3 cached, summary --------------------- cmake | current system slowdown real 0m0.710s | 0m5.135s 720% user 0m0.496s | 0m1.720s 350% sys 0m0.193s | 0m2.206s 1143% B Doing the build ================= This benchmark measures the real build. Because current system does not honor CFLAGS or CXXFLAGS consistently, results might not be very representative and give a raw estimation only. Current system does not allow to build only static or dynamic libraries while cmake builds only one variant. This falsifies results too. Time needed by the compiler affects the results very much. B.1 cmake --------- $ time CCACHE_DISABLE=1 make ; make clean real 0m45.857s | 0m46.944s 0m45.231s 0m45.398s user 0m40.404s | 0m40.775s 0m40.151s 0m40.287s sys 0m4.396s | 0m4.516s 0m4.412s 0m4.260s B.2 current system ------------------ $ time CCACHE_DISABLE=1 make real 0m58.870s | 0m57.545s 1m1.881s 0m57.185s user 0m48.704s | 0m47.651s 0m50.975s 0m47.487s sys 0m8.901s | 0m8.769s 0m9.161s 0m8.773s B.3 summary ----------- cmake | current system slowdown real 0m45.857s | 0m58.870s 130% user 0m40.404s | 0m48.704s 120% sys 0m4.396s | 0m8.901s 200% C Doing the build with cached results ===================================== During this benchmark, ccache (http://ccache.samba.org/) was used to cache compilation results. Hence, time needed by the compiler affects benchmark results not very much and the buildsystem can be compared better. This benchmark compares times needed for parallel and non-parallel builds too. Between every 'make' a 'make clean' was executed. C.1 cmake ========= $ time make ; make clean real 0m10.690s | 0m10.886s 0m10.606s 0m10.580s user 0m7.018s | 0m7.156s 0m6.940s 0m6.960s sys 0m2.933s | 0m2.820s 0m3.008s 0m2.972s $ time make -j6 ; make clean real 0m8.864s | 0m8.175s 0m9.246s 0m9.171s user 0m11.079s | 0m11.113s 0m11.045s 0m11.081s sys 0m4.292s | 0m4.332s 0m4.312s 0m4.232s $ time make -j2 ; make clean real 0m8.195s | 0m8.170s 0m8.227s 0m8.189s user 0m10.673s | 0m10.473s 0m10.657s 0m10.889s sys 0m4.166s | 0m4.292s 0m4.040s 0m4.168s C.2 current system ================== $ time make ; make clean real 0m20.121s | 0m20.473s 0m19.950s 0m19.942s user 0m11.991s | 0m11.973s 0m12.141s 0m11.861s sys 0m7.165s | 0m7.212s 0m7.016s 0m7.268s $ time make -j6 ; make clean real 0m17.262s | 0m16.985s 0m17.634s 0m17.167s user 0m19.821s | 0m20.049s 0m19.437s 0m19.977s sys 0m10.098s | 0m10.253s 0m9.817s 0m10.225s $ time make -j2 ; make clean real 0m16.685s | 0m17.454s 0m16.244s 0m16.358s user 0m19.195s | 0m18.897s 0m19.461s 0m19.229s sys 0m10.114s | 0m10.193s 0m10.037s 0m10.113s C.3 summary =========== C.3.1 summary, non-parallel build --------------------------------- cmake | current system slowdown real 0m10.690s | 0m20.121s 190% user 0m7.018s | 0m11.991s 170% sys 0m2.933s | 0m7.165s 240% C.3.2 summary, -j6 build ------------------------ cmake | current system slowdown real 0m8.864s | 0m17.262s 190% user 0m11.079s | 0m19.821s 180% sys 0m4.292s | 0m10.098s 240% C.3.3 summary, -j2 build ------------------------ cmake | current system slowdown real 0m8.195s | 0m16.685s 200% user 0m10.673s | 0m19.195s 180% sys 0m4.166s | 0m10.114s 240% D Empty builds ============== This test measures the time for a plain 'make' when all files were created already by a previous run. D.1 cmake --------- In opposite to the 'current system', time needed for output comes into account too. Redirecting to /dev/null would reduce time by 1-2% (see below). $ time make ## *no* clean real 0m0.382s | 0m0.376s 0m0.376s 0m0.395s user 0m0.168s | 0m0.172s 0m0.152s 0m0.180s sys 0m0.206s | 0m0.208s 0m0.216s 0m0.196s D.2 current system ------------------ Output affects performance by about 10% so it was redirected to /dev/null. 'cmake' was influenced by 1-2% and results of it were *NOT* corrected. $ time make > /dev/null real 0m0.666s | 0m0.657s 0m0.678s 0m0.664s user 0m0.349s | 0m0.344s 0m0.380s 0m0.324s sys 0m0.322s | 0m0.320s 0m0.304s 0m0.344s D.3 summary ----------- cmake | current system slowdown real 0m0.382s | 0m0.666s 170% user 0m0.168s | 0m0.349s 210% sys 0m0.206s | 0m0.322s 160% F Complexity ============= Numbers below show, how much lines of code (LOC) must be maintained for the buildsystem. It does not take into account foreign code (libtool, generated files, cmake, cmake-modules). It does not measure the complexity of the code itself (e.g. Makefile statements vs. cmake commands). F.1 cmake --------- $ wget http://ensc.de/xmlrpc-c/0003-make-cmake-transition.txt $ diffstat 0003-make-cmake-transition.txt 50 files changed, 988 insertions(+), 1 deletion(-) F.2 current system ------------------ $ find $CVS_REPO -iname 'make*' -o -name '*.make' -o -name 'xmlrpc-c-config.in' -o \ -name '*makefile*' | xargs wc -l 3449 total F.3 summary ----------- Compared with 'cmake', the current system has 350% of LOC which must be maintained. G Summary ========== Real-world usage would be something between case C (build with cached results) and D (empty builds). Numbers there show that the current system needs 170-190% of the time for 'cmake'. H Features =========== This is not covered by numbers or facts but reflects my personal preferences only. H.1 cmake ---------- + supports VPATH builds perfectly (does not work in another mode) + very good RPATH handling (rpaths set in build dir, rpaths removed in installed files) + xmlrpc-c tools and programs can be compiled with clean dependencies ('ldd -u -r *' does not give errors); supports '-Wl,--as-needed' + 'make -j' works + builds on x86_64 and PPC + honors user provided C(XX)FLAGS - does not support targets like 'make dist' or 'make distcheck' H.2 current system ------------------- + 'make -j' seems to work with CVS - VPATH builds do not work ("$REPO/configure && make" -> "make: *** No targets specified and no makefile found. Stop.") - RPATH handling is broken (no multilib support in libtool, binaries in build dir do not have rpath) - does not support targets like 'make dist' or 'make distcheck' - ignores C(XX)FLAGS at several places - broken library dependencies; e.g. | $ ldd -u ../inst/usr/local/lib/libxmlrpc.so | undefined symbol: xmlrpc_XML_ErrorString (../inst/usr/local/lib/libxmlrpc.so) | | $ ldd -u -r ../inst/usr/local/lib/libxmlrpc_client.so | Unused direct dependencies: | | /usr/lib/libgssapi_krb5.so.2 | /usr/lib/libkrb5.so.3 ? last versions did not built on x86_64 and PPC due to mixed PIC/non-PIC code; it could not be tested whether this is still the case with recent CVS snapshot.