I will begin this post by informing you not everything works yet. I’m including this post in the series because learning how to un-screw stuff is almost as important as learning how to do it right. In fact, it is probably more important.
Configuring build environments has got to be my least favorite thing to do. As a consultant I typically don’t join a project until after someone else has already done that and created a step by step document which has been tested multiple times in our development environment. In truth, in an FDA regulated world it’s pretty much required to have a 100% reproducible document/script to set up a development environment.
The sad part about trying to configure distcc on a Debian based build box for use by a Raspberry Pi is the tiny bit of documentation which does exist no longer matches the world. You end up having to cobble bits and pieces from different places and blend them with much hope. Especially when Qt is involved.
I tweaked the .distcc/hosts file on my Raspberry Pi to only have itself and roland-desktop (the 6-core AMD now referred to as build server). Then, on the build server I did the following:
mkdir piBuild cd piBuild git clone https://github.com/raspberrypi/tools.git --depth=1 pitools mkdir test cd test nano hello.cpp
#include <stdio.h>
int main( int argc , char **argv )
{
printf("Hello World!\n");
return 0;
}
$ ls ../pitools/arm-bcm2708/ arm-bcm2708hardfp-linux-gnueabi gcc-linaro-arm-linux-gnueabihf-raspbian arm-bcm2708-linux-gnueabi gcc-linaro-arm-linux-gnueabihf-raspbian-x64 arm-rpi-4.9.3-linux-gnueabihf $ ../pitools/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian/bin/arm-linux-gnueabihf-gcc hello.cpp -o hello roland@roland-desktop:~/piBuild/test$ ls hello hello.cpp
Your first attempt should fail on a 64-bit machine because you are missing 32-bit support packages.
sudo apt-get install zlib1g:i386
Back on the Raspberry Pi:
scp roland@192.168.1.132:piBuild/test/hello . roland@192.168.1.132's password: hello 100% 6413 6.3KB/s 00:00 pi@raspberrypi:~ $ ./hello Hello World! pi@raspberrypi:~ $
So, we can successfully build and copy a simple C++ program and copy it across to the Pi where it works as expected. Just for grins I also tried the other compiler.
$ ../pitools/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian/bin/arm-linux-gnueabihf-g++ hello.cpp -o hello
The executable also worked just fine. In fact, I opened a terminal window on my regular desktop so I could quick walking around my office to build and test everything.
Terminal 1:
$ ssh pi@raspberrypi The authenticity of host 'raspberrypi (192.168.1.161)' can't be established. ECDSA key fingerprint is SHA256:WdA7OyfFqp74k4YxuzCTeqelu2sSQWMDgV3IugIdkus. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'raspberrypi,192.168.1.161' (ECDSA) to the list of known hosts. pi@raspberrypi's password: The programs included with the Debian GNU/Linux system are free software; the exact distribution terms for each program are described in the individual files in /usr/share/doc/*/copyright. Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent permitted by applicable law. Last login: Fri Sep 1 19:27:03 2017 pi@raspberrypi:~ $ scp roland@roland-desktop:piBuild/test/hello . The authenticity of host 'roland-desktop (192.168.1.132)' can't be established. ECDSA key fingerprint is 93:7a:89:c4:c8:cb:fb:38:75:bc:77:f9:b4:64:bd:58. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'roland-desktop' (ECDSA) to the list of known hosts. roland@roland-desktop's password: hello 100% 6453 6.3KB/s 00:00 pi@raspberrypi:~ $ ./hello Hello World! pi@raspberrypi:~ $
For those who do not know, SCP is “secure copy.” It lets you copy files between two computers for which you have a username and password on. Some may note that this time I used the name I entered in my /etc/hosts file instead of the physical IP address. When building scripts it is always best to use a name because you only have to put a new IP address in /etc/hosts for everything to start working again when a new address gets assigned.
After such a rousing success I move back to the build server and do the following:
sudo nano /etc/init.d/distcc
Add the following below the existing PATH line. Be sure to use your own username and piBuild directory.
PATH=/home/roland/piBuild/pitools/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian/bin:"$PATH"
I’m always a bit leery about stuffing things in front of PATH like this. Since this is a daemon control script I’m less leery. You will find a million examples on-line of shell scripts which do this very thing and the second, third, etc. time they get run in the same session bad things happen because the PATH variable now has many repeated directories in it.
Now we need to create links between the actual programs we want to use for c++, cpp, g++ and gcc.
cd ~/piBuild/pitools/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian/bin ln -s arm-linux-gnueabihf-c++ c++ ln -s arm-linux-gnueabihf-cpp cpp ln -s arm-linux-gnueabihf-g++ g++ ln -s arm-linux-gnueabihf-gcc gcc
Now reload the daemon.
sudo systemctl daemon-reload
In theory, we should be good to go on the build server. That assumes no information was out of date and that the current version of pitools supports the Raspberry Pi II with the previous Raspbian distro on it. (Not a good assumption.) Back on the Raspberry Pi we clean the project and kick off our distributed build only to see lots of lines which look like this.
distcc[1516] ERROR: compile ../xpnsqt2/XpnsBrowseWindow.cpp on roland-desktop failed distcc[1516] (dcc_build_somewhere) Warning: remote compilation of '../xpnsqt2/XpnsBrowseWindow.cpp' failed, retrying locally distcc[1516] Warning: failed to distribute ../xpnsqt2/XpnsBrowseWindow.cpp to roland-desktop, running locally instead distcc[1514] ERROR: compile ../xpnsqt2/SplashDialog.cpp on roland-desktop failed distcc[1514] (dcc_build_somewhere) Warning: remote compilation of '../xpnsqt2/SplashDialog.cpp' failed, retrying locally
Now we are at the point where this post becomes important
If you turn verbose on by setting DISTCC_VERBOSE=1 in your environment you end up getting tons of trash spewing at you, but no answer as to what is wrong. This post is important if for no other reason than you don’t have to become an expert on distcc to troubleshoot your broken build environment. I spent quite a bit of time going through the expert friendly man pages before I found this plan of attack.
- In QtCreator click on Projects then “Build Environment” [Details]
Click [Add]
DISTCC_VERBOSE 1 - return to “Edit” view via toolbar
clean all
Click on hammer to build – watch for source file in distcc monitor which fails on a node and write it down along with node. - Open the compile output view, select all, then past into text editor and search for problem source file name.
Select the entire distcc command line for it and save that somewhere. - in a terminal
cd ~/build-xpnsqt2-Desktop-Debug
export DISTCC_SAVE_TEMPS=1
export DISTCC_VERBOSE=1edit ~/.distcc/hosts to change order so localhost is last and problem node is first.paste in the previously saved distcc command line and hit return
watch the monitor window to be sure it went to problem node
This doesn’t fix your problem. It allows you to generate an error file which stays around after compilation fails. In theory I could have done it by setting DISTCC_SAVE_TEMPS in QtCreator, but, I wanted the output of one source file not a deluge from N compilation threads. When you know the code compiles clean without distcc you also know whatever is configured wrong will impact most everything you try to compile. Below is what I burned most of a day to find out. No matter how I searched I could not find this information. You see, you never get to see what is actually wrong when building via distcc in QtCreator.
pi@raspberrypi:~/build-xpnsqt2-Desktop-Debug $ ls /tmp distcc_ba636540.ii qipc_systemsem_qtsingleapplicationebe297d2e336ab08fe29406a369ff59513146b603ce distcc_server_stderr_83876540.txt pi@raspberrypi:~/build-xpnsqt2-Desktop-Debug $ cat /tmp/distcc_server_stderr_83876540.txt In file included from /usr/include/c++/4.9/bits/stl_construct.h:59:0, from /usr/include/c++/4.9/bits/stl_tempbuf.h:60, from /usr/include/c++/4.9/bits/stl_algo.h:62, from /usr/include/c++/4.9/algorithm:62, from /usr/include/arm-linux-gnueabihf/qt5/QtCore/qglobal.h:89, from /usr/include/arm-linux-gnueabihf/qt5/QtCore/qnamespace.h:45, from /usr/include/arm-linux-gnueabihf/qt5/QtCore/qobjectdefs.h:49, from /usr/include/arm-linux-gnueabihf/qt5/QtCore/qobject.h:48, from /usr/include/arm-linux-gnueabihf/qt5/QtCore/qabstractanimation.h:45, from /usr/include/arm-linux-gnueabihf/qt5/QtCore/QtCore:4, from /usr/include/arm-linux-gnueabihf/qt5/QtGui/QtGuiDepends:2, from /usr/include/arm-linux-gnueabihf/qt5/QtGui/QtGui:3, from ../xpnsqt2/payeeDialog.cpp:10: /usr/include/c++/4.9/new:129:41: error: ‘operator new’ takes type ‘size_t’ (‘long unsigned int’) as first parameter [-fpermissive] /usr/include/c++/4.9/new:131:41: error: ‘operator new’ takes type ‘size_t’ (‘long unsigned int’) as first parameter [-fpermissive] /usr/include/c++/4.9/new:137:41: error: ‘operator new’ takes type ‘size_t’ (‘long unsigned int’) as first parameter [-fpermissive] /usr/include/c++/4.9/new:139:41: error: ‘operator new’ takes type ‘size_t’ (‘long unsigned int’) as first parameter [-fpermissive] /usr/include/c++/4.9/new:146:57: error: ‘operator new’ takes type ‘size_t’ (‘long unsigned int’) as first parameter [-fpermissive] /usr/include/c++/4.9/new:148:59: error: ‘operator new’ takes type ‘size_t’ (‘long unsigned int’) as first parameter [-fpermissive]
So, back on the build server we take a look:
roland@roland-desktop:~/piBuild/pitools/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian/bin$ ./gcc -v Using built-in specs. COLLECT_GCC=./gcc COLLECT_LTO_WRAPPER=/home/roland/piBuild/pitools/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian/bin/../libexec/gcc/arm-linux-gnueabihf/4.8.3/lto-wrapper Target: arm-linux-gnueabihf
Meanwhile, back on the Pi.
pi@raspberrypi:~ $ gcc -v Using built-in specs. COLLECT_GCC=gcc COLLECT_LTO_WRAPPER=/usr/lib/gcc/arm-linux-gnueabihf/4.9/lto-wrapper Target: arm-linux-gnueabihf Configured with: ../src/configure -v --with-pkgversion='Raspbian 4.9.2-10' --with-bugurl=file:///usr/share/doc/gcc-4.9/README.Bugs --enable-languages=c,c++,java,go,d,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-4.9 --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --with-gxx-include-dir=/usr/include/c++/4.9 --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-gnu-unique-object --disable-libitm --disable-libquadmath --enable-plugin --with-system-zlib --disable-browser-plugin --enable-java-awt=gtk --enable-gtk-cairo --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-4.9-armhf/jre --enable-java-home --with-jvm-root-dir=/usr/lib/jvm/java-1.5.0-gcj-4.9-armhf --with-jvm-jar-dir=/usr/lib/jvm-exports/java-1.5.0-gcj-4.9-armhf --with-arch-directory=arm --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --enable-objc-gc --enable-multiarch --disable-sjlj-exceptions --with-arch=armv6 --with-fpu=vfp --with-float=hard --enable-checking=release --build=arm-linux-gnueabihf --host=arm-linux-gnueabihf --target=arm-linux-gnueabihf Thread model: posix gcc version 4.9.2 (Raspbian 4.9.2-10)
So, tomorrow I need to look into either backing gcc down to 4.8 on the Pi or, more likely, poke around for an updated pitools (thought that _was_ supposed to be where one pulls the current from.)
At the very least, anyone who finds this will have _some_ idea how to troubleshoot a distcc configuration. Honestly I was a bit perturbed by the local retry which succeeded. This meant I didn’t see the errors I was looking for. Of course, I would have been less perturbed if I knew another piece of dark lore.
DISTCC_FALLBACK 0
Setting that environment variable to zero stops the automatic fall back to a local compile for a failed module. I have not played with it to find out if it could have saved me nearly a day of searching by showing the error messages to me the very first time it failed.
Related Posts:
Where Did My QDebug Output Go?
