Showing posts with label Make. Show all posts
Showing posts with label Make. Show all posts

Monday, 11 December 2017

Crashed my Build Server....

When I say I crashed it, I mean... It just locked up and I had to soft reboot it... And when I say build server I mean one of the virtual machines on one of my Xen Hosts....

So, the machine is a fairly beefy 16 core machine with 48GB of RAM running on a Dell server under the desk, this disk base is a RAID-0 200GB unit over a bunch of 2.5" 10,000 RPM SAS Drives to a Perci5 RAID Controller....

The machine is only really spooled up for big builds, and this was one of them, I wanted to build LLVM support before bed.

The problem?  When I performed the build with "make -j all" it would get to 16% and then blank the screen, and totally lock up, nothing, nada, nowt... I left it for a while but nothing happened, and yes the LLVM build is time consuming but it doesn't lock at 16% for minutes.

Soft reboot, and the same happened again...


I've started the build again with "make -j 15" rather than all sixteen cores.  And it's already up into the 35% area of the build whilst I've been typing this....

But, what the heck locked the machine up before?  It wasn't actually using all the processors all of the time, surely?  Maybe?

I might have to set up one of the older 2950's and have a play about with this, leaving my one beefy machine alone.

Just on a side note, could you imagine the mess your system would be left in if you soft rebooted this kind of kit, mid-build, with no warning... LOL.

Thursday, 4 May 2017

Software Engineering : My very bad way to set up makefiles

I may have wrote this post, but I'm just being lazy... Read on.

I've made other Boost videos, and other codeblocks videos, and I've always assumed that the user at the other end was well aware of how to compile & build code.

However, it seems there are lots of users whom are not aware of how you actual create a program from source code, at least not with C++.

So I'm going to show you the absolutely most wrong basic way of creating and using a makefile to perform a build...

The first step of C++ program creation, is that trivial matter of creating the source code, easy right?... Yep, so lets skip that step, and go to when its once  and we want to pass the code through a compiler, the compiler turns the source not into a program, but into an intermediary format called an object file or object code.

The compiler then leaves the process and another program takes over, this second program is known as the Linker, and it takes the object code and links it with all the system calls and other libraries you are wishing to use.

There are other steps involved in the process of building a program, however, compiling & linking are the two main steps; when I began programming you were made very much aware of this by the compiler and linker generally being such large complex programs and the libraries being so big you had to physically change floppy disk between the different steps of the process.

This process was also taught to me in school & later college, it was common practice to understand a compiled program was actually compiled then linked.

Unfortunately, this knowledge seems to have fallen in to the way side, it is of course more important to learn how to program than be bogged down in minutia, however, once you have learned I strongly believe you should expand your knowledge and really have a holistic knowledge of the craft of programming.

So, back to our program, the make file, what might they look like? Well they are simply text files,  which contain commands very similar to bash script... We call this make file script by passing it to the "make" program, you can get make for Linux, mac or alongside MinGW for windows.

Lets start writing a make file for any basic C++ program, which links the Boost libraries we've built somewhere on disk, and which looks like this:


#include <iostream>
#include <string>
#include <boost/filesystem.hpp>


int main (int p_argc, char** p_argv)

{
     if ( p_argc > 1 )
     {
          std::cout << "File or Folder ";
          std::string l_pathString(p_argv[1]);
          boost::filesystem::path l_path(l_pathString);

          if ( boost::filesystem::exists(l_path) )
          {
               std::cout << "Exists!";
          }
          else
          {
               std::cout << "Not Found";
          }
          std::cout << std::endl;
     }
}

Save this as "main.cpp" and then define a variable called "FILES" into which we place this filename.

FILES=main.cpp

We want to start our make file with a variable which contains the compiler we wish to use, for Linux on my test machine I'll use "g++".

CC=g++

So now we have a value called "CC" which is a string of characters with the value "g++".  The dollar sign indicates that "CC" is a variable within the script, which we might want to use later, I use these kind of variables throughout makefiles so I can place at the top any paths or references I like, or I can alter them when I move the makefile from Linux to say Windows, without needing to change hundreds of places throughout the script, I can change just one location.

I find this working with strings is 95% of what working with makefiles is about, next in the file we then define the strings for where to find both the boost headers and the binary libraries are...

BOOST_INC=/home/xelous/boost
BOOST_LIB=$(BOOST_INC)/stage/lib

Notice, I create the root folder for boost as one variable, and then in the next line I use that variable to extend into the "stage/lib" folder, so I never duplicate the root folder for the boost libraries.

We can now define at least our build command, slightly different the command is a verb, in this case "build", which is the command we will use here, but which is also the default.  If you save this makefile as "makefile" in a folder, then navigate your console to that same folder and type "make", the make program will look for "makefile" by default, find it and build the "build" verb by default!

build: 
   $(CC) -I$(BOOST_INC) -L$(BOOST_LIB)

So, what is this command doing?  Well it is saying to use the compiler, with the Include folder for boost and look for libraries in the boost library folder we've set.


The "-I" and "-L" parameters are the same parameters one would use on the command line for "g++".  This command therefore equates to "g++ -I/home/xelous/boost -L/home/xelous/boost/stage/lib", but I think you will agree it is somewhat easier to manage.

We also need to tell the program to link against some libraries, and that we want them linked statically in this case.   So our next variable will be called "CFLAGS1" to pass the Compiler flags to the command.  Back at the top of our make file I therefore add....

CFLAGS1=-fpermissive -Wl,-Bstatic

This file tells the compiler to be permissive of some common errors and treat them as warnings.  You can use whatever parameters you wish for your compile, the flags available and their meaning are quite varied, however you will need to use a second set to within the build command, so lets define that now....

CFLAGS2=-static-libstdc++ -std=c++14

The second tells the compiler and linker to link against the static versions of the libraries we select later, you will need to look at the boost invocation pages in order to go back over our previous posts and build boost static and take advantage of this option.

I'm not going to explain static linking here, nor any of the other switches, available.

Now we're linking the C++ standard library, and boost, statically we need another variable with the list of libraries to link, on the command line to g++ this is with the "-l" (lowercase L), except with we are going to make the program threaded, so we can set up a new makefile script variable called "LIBS", into which we set the libraries, like so.

LIBS=-pthread -lboost_signals -lboost_system -lboost_filesystem

We now have the compiler, the folders, the libraries, the flags... What about the target file for the build?... Well I want it to be an executable called "
fredrick".

TARGET=-o fredrick

Now we need to put this all together into a command, just running the script with the variables does nothing, we need to define a verb... Lets call this "build" a verb in the makefile has to have a colon following it...

build:

And we can tell make what to perform for a build:

build: 
   $(CC) -I$(BOOST_INC) -L$(BOOST_LIB) $(CFLAGS1) $(FILES) $(CFLAGS2) $(LIBS) $(TARGET)

We might also want to clean the project, so we'd want to delete the target file maybe, by adding another verb:

clean:
   rm $(TARGET)

We can then perform the call "make build" or "make clean" in the folder with this makefile and perform our build.

A bit rough and ready, but it works to get you up and running.

Monday, 31 October 2016

Software Engineering : "warning: defaulted and deleted functions only available with -std=c++11"

warning: defaulted and deleted functions only available with -std=c++11 or -std=gnu++11

When you're using std=c++14??!?!?!  What the heck is going on here?

I've noticed this strange bit of behaviour with g++ and warning when compiling with multiple cores... I've stripped this example back to the bare minimum, so lets just define what we're building and then we'll look at the strange warning which comes out, first of all, we need to compile two files, completely independent of one another:

Main.cpp

#include <iostream>

int main ()
{
std::cout << "Hello World" << std::endl;
}

This is our first file, the other has to be a class:

Data.h

#ifndef DATA_HEADER
#define DATA_HEADER

#include <string>

namespace Xelous
{
class Data
{
private:
std::string m_Data;
public:
Data(const std::string& p_Data);
const std::string& GetData() const;
};
}

#endif

And the code for this class looks like this:

#include "Data.h"

namespace Xelous
{
Data::Data(const std::string& p_Data)
:
m_Data(p_Data)
{
}
const std::string& Data::GetData() const
{
return m_Data;
}
}

This code so far is all fine, however, in our data header we don't want the default constructor, so we're going to just delete the default constructor, like this:

#ifndef DATA_HEADER
#define DATA_HEADER

#include <string>

namespace Xelous
{
class Data
{
private:
std::string m_Data;
public:
Data() = delete;
Data(const std::string& p_Data);
const std::string& GetData() const;
};

}

#endif

Fairly simple stuff so far, and no problems, no errors... However, I always build with "pedantic", especially as I get closer to release and I'm looking at code, so lets see my build file for the above project:

CC=g++
STD=c++14
WARNINGS=-Wall -Wfatal-errors
PEDANTIC=-pedantic
OUTPUT=example

MAIN=main
DATA=data

CompileMain: ${MAIN}.cpp
${CC} -std=${STD} ${WARNINGS} ${PEDANTIC} -c ${MAIN}.cpp -time
CompileData: ${DATA}.cpp
${CC} -std=${STD} ${WARNINGS} ${PEDANTIC} -c ${DATA}.cpp -time
Lets just stop there, and look what we have in the build file so far, we have the compiler, the STL version to use, the warnings, we're using pedantic and I'm only compiling the two files, there's no linking going on.

I always prefer this so I can trap individual compile errors or problems, speeding up the over all development... Once I'm happy both classes or files are clear I can then combine them into their link, by adding:

Link: ${DATA}.o ${MAIN}.o
${CC} -o ${OUTPUT} ${DATA}.o ${MAIN}.o -time
This will link everything up...

There isn't a problem with this code, we can save the file now and call "make" on those three targets and it will work fine... Let us just complete the make file:

clean:
rm -f ${DATA}.o
rm -f ${MAIN}.o
rm -f ${OUTPUT}
clearscreen:
clear
all: clean clearscreen CompileData CompileMain Link

These last three targets are our clear, a simple clear screen and then the "all" target, 

So, we can now "make all" and see this output:


Building one after the other is fine, however, if I use "j2" that is to make it build data and main on different cores at the same time, then link the result of both you get the above error:


Both compiles are being sent to the compiler with c++14, when we're using serial single focus compilation there is no warning, yet with two the warnings pops out?

I've actually run out of time on this very busy Sunday to look into this any further at the moment, I'm therefore going to schedule this for tomorrow and let it loose on the world...

Wednesday, 7 September 2016

Software Engineering : Building Cryptozoidberg (Boolberry)

I recently received a request via YouTube to help a fellow programmer out, they were having issues building this library against the boost libraries... https://github.com/cryptozoidberg/boolberry

The problem essentially is the lack of transparency in CMake when it looks for boost with it's "FindBoost" functionality, because neither he, nor I build boost with CMake.  I personally don't use or like CMake, I do use make however, and am happy to look around in the make files.

So, I did... And there's nothing specifically wrong with this makefile, but it is insisting on looking for boost within the system, not the version built by yourself.

My Solution therefore has been:

sudo apt-get install libboost-all-dev
sudo apt-get install git
sudo apt-get install cmake
cd ~
mkdir c++
cd c++
git clone //https://github.com/cryptozoidberg/boolberry.git
cd boolberry
make -j4

(Note, replace 4 here with as many cores as your PC has available!  More cores will speed this build up, and it will take a long while)

And this works fine:


As you can see.  However, I did this from my main build machine, which has a whole bunch of additional and historic settings.  Which is the most common pitfall for others coming to build your software later, they lack these customisation's.

My build environment: Ubuntu 16.04 Desktop, with the i3 Window Manager (just open a terminal), and I have installed gcc (sudo apt-get install gcc).  These are the only steps before running the above script in a terminal window.

I therefore decided to perform a full clean setup of this system in a new Ubuntu 16.04 (64bit) VMware Virtual machine.

From this, I realise I needed to explain how to install the boost libraries, how to set up cmake and how to get git working before the build would work....

You can see this whole process below, please accept my apologies that this was recorded on an extremely slow internet connection, it got smushed during recording over the mobile link, and I've not had chance to re-shoot or edit the video.



As you can see however, the build worked perfectly.

To Muhmad whom asked for this, you'll find the Donate page at the top right!