Raspberry Pi II image
Information Technology, Raspberry Pi, Thank You Sir May I Have Another

How Far We’ve Come – Pt. 9

I will begin this post by informing you not everything works yet. I’m including this post in the series because learning how to un-screw stuff is almost as important as learning how to do it right. In fact, it is probably more important.

Configuring build environments has got to be my least favorite thing to do. As a consultant I typically don’t join a project until after someone else has already done that and created a step by step document which has been tested multiple times in our development environment. In truth, in an FDA regulated world it’s pretty much required to have a 100% reproducible document/script to set up a development environment.

The sad part about trying to configure distcc on a Debian based build box for use by a Raspberry Pi is the tiny bit of documentation which does exist no longer matches the world. You end up having to cobble bits and pieces from different places and blend them with much hope. Especially when Qt is involved.

I tweaked the .distcc/hosts file on my Raspberry Pi to only have itself and roland-desktop (the 6-core AMD now referred to as build server). Then, on the build server I did the following:

mkdir piBuild
cd piBuild
git clone https://github.com/raspberrypi/tools.git --depth=1 pitools

mkdir test
cd test
nano hello.cpp
#include <stdio.h>
int main( int argc , char **argv )
    printf("Hello World!\n");
    return 0;
$ ls ../pitools/arm-bcm2708/
arm-bcm2708hardfp-linux-gnueabi gcc-linaro-arm-linux-gnueabihf-raspbian
arm-bcm2708-linux-gnueabi gcc-linaro-arm-linux-gnueabihf-raspbian-x64

$ ../pitools/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian/bin/arm-linux-gnueabihf-gcc hello.cpp -o hello
roland@roland-desktop:~/piBuild/test$ ls
hello hello.cpp

Your first attempt should fail on a 64-bit machine because you are missing 32-bit support packages.

sudo apt-get install zlib1g:i386

Back on the Raspberry Pi:

scp roland@ .
roland@'s password:
hello 100% 6413 6.3KB/s 00:00
pi@raspberrypi:~ $ ./hello
Hello World!
pi@raspberrypi:~ $

So, we can successfully build and copy a simple C++ program and copy it across to the Pi where it works as expected. Just for grins I also tried the other compiler.

$ ../pitools/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian/bin/arm-linux-gnueabihf-g++ hello.cpp -o hello

The executable also worked just fine. In fact, I opened a terminal window on my regular desktop so I could quick walking around my office to build and test everything.

Terminal 1:

$ ssh pi@raspberrypi
The authenticity of host 'raspberrypi (' can't be established.
ECDSA key fingerprint is SHA256:WdA7OyfFqp74k4YxuzCTeqelu2sSQWMDgV3IugIdkus.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'raspberrypi,' (ECDSA) to the list of known hosts.
pi@raspberrypi's password:

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Fri Sep 1 19:27:03 2017

pi@raspberrypi:~ $ scp roland@roland-desktop:piBuild/test/hello .
The authenticity of host 'roland-desktop (' can't be established.
ECDSA key fingerprint is 93:7a:89:c4:c8:cb:fb:38:75:bc:77:f9:b4:64:bd:58.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'roland-desktop' (ECDSA) to the list of known hosts.
roland@roland-desktop's password: 
hello 100% 6453 6.3KB/s 00:00 
pi@raspberrypi:~ $ ./hello
Hello World!
pi@raspberrypi:~ $

For those who do not know, SCP is “secure copy.” It lets you copy files between two computers for which you have a username and password on. Some may note that this time I used the name I entered in my /etc/hosts file instead of the physical IP address. When building scripts it is always best to use a name because you only have to put a new IP address in /etc/hosts for everything to start working again when a new address gets assigned.

After such a rousing success I move back to the build server and do the following:

sudo nano /etc/init.d/distcc

Add the following below the existing PATH line. Be sure to use your own username and piBuild directory.


I’m always a bit leery about stuffing things in front of PATH like this. Since this is a daemon control script I’m less leery. You will find a million examples on-line of shell scripts which do this very thing and the second, third, etc. time they get run in the same session bad things happen because the PATH variable now has many repeated directories in it.

Now we need to create links between the actual programs we want to use for c++, cpp, g++ and gcc.

cd ~/piBuild/pitools/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian/bin
ln -s arm-linux-gnueabihf-c++ c++
ln -s arm-linux-gnueabihf-cpp cpp
ln -s arm-linux-gnueabihf-g++ g++
ln -s arm-linux-gnueabihf-gcc gcc

Now reload the daemon.

sudo systemctl daemon-reload

In theory, we should be good to go on the build server. That assumes no information was out of date and that the current version of pitools supports the Raspberry Pi II with the previous Raspbian distro on it. (Not a good assumption.) Back on the Raspberry Pi we clean the project and kick off our distributed build only to see lots of lines which look like this.

distcc[1516] ERROR: compile ../xpnsqt2/XpnsBrowseWindow.cpp on roland-desktop failed
distcc[1516] (dcc_build_somewhere) Warning: remote compilation of '../xpnsqt2/XpnsBrowseWindow.cpp' failed, retrying locally
distcc[1516] Warning: failed to distribute ../xpnsqt2/XpnsBrowseWindow.cpp to roland-desktop, running locally instead
distcc[1514] ERROR: compile ../xpnsqt2/SplashDialog.cpp on roland-desktop failed
distcc[1514] (dcc_build_somewhere) Warning: remote compilation of '../xpnsqt2/SplashDialog.cpp' failed, retrying locally

Now we are at the point where this post becomes important

If you turn verbose on by setting DISTCC_VERBOSE=1 in your environment you end up getting tons of trash spewing at you, but no answer as to what is wrong. This post is important if for no other reason than you don’t have to become an expert on distcc to troubleshoot your broken build environment. I spent quite a bit of time going through the expert friendly man pages before I found this plan of attack.

  1. In QtCreator click on Projects then “Build Environment” [Details]
    Click [Add]
  2. return to “Edit” view via toolbar
    clean all
    Click on hammer to build – watch for source file in distcc monitor which fails on a node and write it down along with node.
  3. Open the compile output view, select all, then past into text editor and search for problem source file name.
    Select the entire distcc command line for it and save that somewhere.
  4. in a terminal
    cd ~/build-xpnsqt2-Desktop-Debug
    export DISTCC_SAVE_TEMPS=1
    export DISTCC_VERBOSE=1edit ~/.distcc/hosts to change order so localhost is last and problem node is first.paste in the previously saved distcc command line and hit return
    watch the monitor window to be sure it went to problem node

This doesn’t fix your problem. It allows you to generate an error file which stays around after compilation fails. In theory I could have done it by setting DISTCC_SAVE_TEMPS in QtCreator, but, I wanted the output of one source file not a deluge from N compilation threads. When you know the code compiles clean without distcc you also know whatever is configured wrong will impact most everything you try to compile. Below is what I burned most of a day to find out. No matter how I searched I could not find this information. You see, you never get to see what is actually wrong when building via distcc in QtCreator.

pi@raspberrypi:~/build-xpnsqt2-Desktop-Debug $ ls /tmp
 distcc_ba636540.ii qipc_systemsem_qtsingleapplicationebe297d2e336ab08fe29406a369ff59513146b603ce

pi@raspberrypi:~/build-xpnsqt2-Desktop-Debug $ cat /tmp/distcc_server_stderr_83876540.txt
 In file included from /usr/include/c++/4.9/bits/stl_construct.h:59:0,
 from /usr/include/c++/4.9/bits/stl_tempbuf.h:60,
 from /usr/include/c++/4.9/bits/stl_algo.h:62,
 from /usr/include/c++/4.9/algorithm:62,
 from /usr/include/arm-linux-gnueabihf/qt5/QtCore/qglobal.h:89,
 from /usr/include/arm-linux-gnueabihf/qt5/QtCore/qnamespace.h:45,
 from /usr/include/arm-linux-gnueabihf/qt5/QtCore/qobjectdefs.h:49,
 from /usr/include/arm-linux-gnueabihf/qt5/QtCore/qobject.h:48,
 from /usr/include/arm-linux-gnueabihf/qt5/QtCore/qabstractanimation.h:45,
 from /usr/include/arm-linux-gnueabihf/qt5/QtCore/QtCore:4,
 from /usr/include/arm-linux-gnueabihf/qt5/QtGui/QtGuiDepends:2,
 from /usr/include/arm-linux-gnueabihf/qt5/QtGui/QtGui:3,
 from ../xpnsqt2/payeeDialog.cpp:10:
 /usr/include/c++/4.9/new:129:41: error: ‘operator new’ takes type ‘size_t’ (‘long unsigned int’) as first parameter [-fpermissive]
 /usr/include/c++/4.9/new:131:41: error: ‘operator new’ takes type ‘size_t’ (‘long unsigned int’) as first parameter [-fpermissive]
 /usr/include/c++/4.9/new:137:41: error: ‘operator new’ takes type ‘size_t’ (‘long unsigned int’) as first parameter [-fpermissive]
 /usr/include/c++/4.9/new:139:41: error: ‘operator new’ takes type ‘size_t’ (‘long unsigned int’) as first parameter [-fpermissive]
 /usr/include/c++/4.9/new:146:57: error: ‘operator new’ takes type ‘size_t’ (‘long unsigned int’) as first parameter [-fpermissive]
 /usr/include/c++/4.9/new:148:59: error: ‘operator new’ takes type ‘size_t’ (‘long unsigned int’) as first parameter [-fpermissive]

So, back on the build server we take a look:

roland@roland-desktop:~/piBuild/pitools/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian/bin$ ./gcc -v
Using built-in specs.
Target: arm-linux-gnueabihf

Meanwhile, back on the Pi.

pi@raspberrypi:~ $ gcc -v
Using built-in specs.
Target: arm-linux-gnueabihf
Configured with: ../src/configure -v --with-pkgversion='Raspbian 4.9.2-10' --with-bugurl=file:///usr/share/doc/gcc-4.9/README.Bugs --enable-languages=c,c++,java,go,d,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-4.9 --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --with-gxx-include-dir=/usr/include/c++/4.9 --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-gnu-unique-object --disable-libitm --disable-libquadmath --enable-plugin --with-system-zlib --disable-browser-plugin --enable-java-awt=gtk --enable-gtk-cairo --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-4.9-armhf/jre --enable-java-home --with-jvm-root-dir=/usr/lib/jvm/java-1.5.0-gcj-4.9-armhf --with-jvm-jar-dir=/usr/lib/jvm-exports/java-1.5.0-gcj-4.9-armhf --with-arch-directory=arm --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --enable-objc-gc --enable-multiarch --disable-sjlj-exceptions --with-arch=armv6 --with-fpu=vfp --with-float=hard --enable-checking=release --build=arm-linux-gnueabihf --host=arm-linux-gnueabihf --target=arm-linux-gnueabihf
Thread model: posix
gcc version 4.9.2 (Raspbian 4.9.2-10)

So, tomorrow I need to look into either backing gcc down to 4.8 on the Pi or, more likely, poke around for an updated pitools (thought that _was_ supposed to be where one pulls the current from.)

At the very least, anyone who finds this will have _some_ idea how to troubleshoot a distcc configuration. Honestly I was a bit perturbed by the local retry which succeeded. This meant I didn’t see the errors I was looking for. Of course, I would have been less perturbed if I knew another piece of dark lore.


Setting that environment variable to zero stops the automatic fall back to a local compile for a failed module. I have not played with it to find out if it could have saved me nearly a day of searching by showing the error messages to me the very first time it failed.

<Previous-part    Next-Part>

Related Posts:

QT Database book

AGILE book

Where Did My QDebug Output Go?

MOC Parse Error at “std”

KDE Neon – Distcc and Qt

CopperSpice Experiments

QtCreator – No qmlScene installed

So You Can’t Get Your Models to Work with QML?