One of the tools which was wildly touted years ago was distcc. This is a distributed compilation system which can be brutal to set up, but can also dramatically reduce compilation times for big jobs. It has fallen out of favor in recent years because most developers end up getting a quad or more core machine with a modern enough CPU to have all kinds of virtualization and hyper threading. These machines also tend to have many more Gig of RAM than they really need so, if Linux really is good about adjusting its disk cache memory usage, in theory you won’t see much boost. At least that is the argument I keep hearing and usually I hear that argument from people using laptops for development.
Flaws in the Thinking
So, please allow me to point out some flaws in that (mostly Millineals) thinking:
- Laptops, unless their battery life is measured in mere minutes, _always_ have underpowered components. Yes, you may have lots of RAM, a great sounding graphics chipset, etc., but the hardware children will have opted for the lowest power consuming version of each. Even your USB ports will operate at both lower power and slower speed because the overall design goal was to make the batter last as long as possible. Be honest. When you are thinking about a new laptop and see “atrocious battery life” in the reviews, you click to the next one don’t you?
- A sucky network isn’t going to make anything run faster. Most shops which complained profusely about distcc not returning much bang for the buck typically have a horrible network where people groan any time they have to transfer even a print job on it.
- Both make and moc have gotten much better when it comes to working with distcc. One of the big drawbacks of building really complex Qt GUI applications with distcc used to be moc didn’t distribute well. I don’t notice the particulars, but I don’t notice a problem anymore.
Any C++ Qt application with a sufficient number of source files can benefit from using distcc. Assuming your network isn’t a three legged dog running in deep snow, that is. As to hand tuning the disk cache and conducting other experiments with it, I don’t bother. You can read about a few experiments here. That mystical “sufficient number of files” threshold is much lower than you think. In order to verify this I need to install the distcc monitor.
This is a little graphical tool which lets you see how your build is using the farm. After that I needed to install distcc itself. For some reasons the software application tool doesn’t included that, but you can find it with synaptic.
I also installed distcc-pump. There are pluses and minuses here. The default configuration of distcc does not work with pump. There has been a bug with posts and reports dating back to 2007 and probably beyond where you end up with a rash of “connection refused” errors trying to let distcc dynamically find and use distcc servers on your network. I forgot about this and spent some head slamming time figuring it out.
The Next Step
Go that far on each machine which is to be part of your distcc compilation farm then open up a terminal and type the following:
On my main desktop it returned:
Now you need to know just who those machines are.
roland@roland-HP-Compaq-8100-Elite-SFF-PC:~$ nslookup 192.168.1.132 Server: 127.0.1.1 Address: 127.0.1.1#53 126.96.36.199.in-addr.arpa name = roland-desktop. roland@roland-HP-Compaq-8100-Elite-SFF-PC:~$ nslookup 192.168.1.105 Server: 127.0.1.1 Address: 127.0.1.1#53 188.8.131.52.in-addr.arpa name = roland-HP-Compaq-8100-Elite-SFF-PC.
You need to tweak distcc just a touch
sudo nano /etc/default/distcc
The above lines need to be in your distcc file. Of course you need to change 192.168.1.0/24 to be whatever your network is. Another major issue is that LISTENER needs to be blank. It just doesn’t seem to work any other way.
As I said, I had quite a bit of head slamming time trying to track down the “connection refused” issue so I made a few changes which may not be required. I’m going to list them here and we will experiment more during another post when I try to distcc from my Raspberry Pi.
$ cat /etc/hosts
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
$ cat /etc/distcc/hosts
# As described in the distcc manpage, this file can be used for a global
# list of available distcc hosts.
# The list from this file will only be used, if neither the
# environment variable DISTCC_HOSTS, nor the file $HOME/.distcc/hosts
# contains a valid list of hosts.
# Add a list of hostnames in one line, seperated by spaces, here.
Once all of that was done I was able to tweak the xpnsqt2 make parameters as follows:
Basically I added
-j40 CC=distcc CXX=distcc
You will find posts on the Internet telling you to add
QMAKE_CC = distcc QMAKE_CXX = distcc
to your .pro file. While it is true this will cause qmake to generate
CC = distcc CXX = distcc
it is also true that people tend to forget those are in there then post a project on SourceForge or their own Web site many people cannot build. The reason many wish to add it directly into the .pro file is so distcc gets used by the next user.
Those additional make options get saved in .pro.user not in the .pro file. If you are building on multiple machines or have multiple developers all using a common build environment it makes sense to put those values in the original .pro file and to look up how to force in the -j40 option as well. If you are working on something which will be released as an OpenSource project of some kind, best not to make those mods.
While I have not tried it I have seen posts stating you can define the CC and CXX environment variables to be distcc gcc and distcc g++ respectively. For someone working on many projects this is definitely the way to go, assuming it works.
Once all of that was done I fired up QtCreator, cleaned the project, then kicked off a build
The machine I’m using as the build server is horribly named “roland-desktop.” It is an AMD 6-core machine with 20Gb of RAM and an SSD. I should also mention it is running BOINC while idle. The machine which is actually on my desktop is that Compaq-8100-Elite blah blah blah machine. It is a quad-core I7 having 16Gb of RAM and an SSD.
As the monitor clearly shows, even though the build server is weighted down by BOINC, a project this small benefited from distcc.