Showing posts with label Linux. Show all posts
Showing posts with label Linux. Show all posts

Tuesday, 2 March 2021

Linux - Detect, Format & Mount USB-Flash Stick Natively

I've had yet another drive fail in one of my old Linux servers, this was an old 2.5" mechanical just used for booting... so I'm able to go in; just about; and copy all the data off of the machine.

However, the USB-flash drive I chose as my life-boat was being a bit of a pain, here's the commands I did:

  • ls /dev/sd*
    (and note down the replies)
    (insert the USB-Stick)
  • ls /dev/sd*
    (note the label of the new drive, for me this is /dev/sdc)
  • sudo fdisk /dev/sdc
    (delete the existing exFat partition)
    (new partition max size - of Linux type)
    (write & exit)
  • sudo mkfs.ext4 /dev/sdc1
  • mkdir ~/external
  • sudo mount /dev/sdc1 ~/external
  • sudo chown <myUser> ~/external/
  • rsync -a -v /myData/ ~/external
I could "sudo" rsync and save the need to chown, but here you go.

I'm out, more from the HMS Rodney build soon!

Saturday, 10 October 2020

Twelve Drives Exit - Only Ten Drives Survive

 I've spent the morning doing a bunch of hard work, like using a surface compactor to lay a 60 meter gravel track... I was knackered come lunch, so I decided to have a play in the server world.

Now, some of you may know we're between properties at the moment, this means I had to take all my servers offline and move them.

However, I've been desperate to get the 32 core machine back online.

Booting up just now though and a couple of the drives had gone bad, like physically bad.  As such my ZFS pool was just a total write off, so I've decided to restore from my offline back up.  There's not actually that much data on that cluster, it was only running raid 2.... And with two disks dying in a simple move, I thought it better to go for raid3.


But remember folks, RAID is not a back up.  My important server is a triple mirror Zfs pool.  I can lose any 2 of the three drives over there, and they're brand new nice WD SSD's... plus the server is ONLY turned on for backups.

This serer is my scratch working/coding project server, on which I host my build slaves and nodes in network tests etc.

Tuesday, 15 January 2019

Cross Compiling Clarity

I have an important distinction to make clear... Cross Compiling is using a machine with a different processor type than the target machine for your program.

So, you have a PC with an Intel Core 2 Duo in it, and you use a compiler to output an ARM executable, that is cross compiling.

If you're on an Intel Core Duo toting 15" MacBook pro (from 2006) and you use that to produce the same ARM executable, that's still cross-compiling.  BUT if you use that MacBook to generate a windows executable it's not cross compiling, as the processor is the same in both targets, you're not crossing over...

Some folks argue this differently, because you're crossing between the Windows and Apple OS's you're cross-compiling.  In my book, and by the definition given elsewhere, you're not, in this case you are cross-platform (the platform being the OS) but you're not cross compiling because the processors are the same family.

That's all, I just wanted this out there, argue as you all see fit.


Monday, 10 September 2018

Ikea Server Rack is Finished

What kind of bloke gets to play with servers on his day off?  A damn lucky one my wife always tell me?.. Am I lucky?  Or do I have a plan?  I think I have a plan, this is a project I've been meaning to sort out for quite some time, now I have a week off of work, I'm knocking things out the park... Sooo......


Meet my new server wrack, so this is a pair of Ikea Lack Coffee tables, one with the feet fitted, and it's screwed down onto the top of another, with a pair of wheel which I also got at Ikea.

I've reinforced the leg uprights, and I'm busy fitting out the rear with a dedicated switch and power setup.

It's all on wheels so it can go anywhere in my office with relative ease.

And yes, these are my older Poweredge servers (from my fan/noise reduction videos).

Only the top unit is operational, with all the drives fitted and a full compliment of memory (64GB I think is in it).  Its socking a quad port gigabyte Ethernet at the moment but I have a whole box of fiber channel cards to go into the machines up here taking me up to 4Gbit per card (full duplex) and I plan to fit two cards in each machine so we should bond up to about 8gbits... We'll see, waiting on the cables to arrive from Hong Kong.


My other item is to get myself a dual Socket 1366 based server soon, just to play about with 6 core 12 thread CPU's (so up to 12 cores 24 threads) and perhaps in the new year I'll go up to the E5- series of Xeons to massive amounts of threads.

At the moment though, load balancing various things on my AWS instances and dropping to my own hosting is more my interests...

And a friend of mine has reminded me of a project he and I need undertake - watch this space.

Monday, 30 April 2018

Kdenlive : First Impressions

They say first impressions count and that's certainly true for any tool I utilize.  In what little free time I've had over the weekend, I've been starting to edit up all the clips I took of the workstation upgrade & rebuild.  This is my passage from the pedestrian Core i7 950 to a Xeon X5670.

I've elected to update my video editing software and Kdenlive seemed to be the answer to visual challenges I had at hand, and oh boy was I right.

Following a seamless download of the Appimage version, a simple startup and about ten minutes orientating myself, then about a further ten to just get used to the S X and M control core points I was applying effects to clips, setting up spacing in between others and saving to perform a test render.

I am utterly and totally so impressed with Kdenlive, this is the first time in a long while I've been equally surprised and delighted by a piece of software.

Wednesday, 3 January 2018

Ultra Cheap ZFS Array

Make your own ZFS array (mirrored) with USB Flash drives, for cheap!



Since this... interesting post of mine... has only about 10 views, and my tech items usually get a few hundred, I figure somewhere along the lines it got trampled by my silly New Years post....

Sunday, 31 December 2017

Using Flash Drives in ZFS Mirror

This post comes from an idea I had to allow me to easily carry a ZFS mirror away from a site and back again, we didn't need much space - only 5gb - but it had to be mirrored in triplicate, one copy to stay locally, one going into a fire safe on site and the third to be carried by the IT manager off-site each evening.

The trouble?  A near zero budget, so for a little over £45 we have a 14GB ZFS mirrored pool, across three 16 GB USB Flash drives and one three port USB 3.0 hub.

It was perfect for the task at hand, extremely portable, and cheap, I thought the same approach may help anyone trying to get to learn a little more about ZFS, a student or even someone using a laptop as a small office server - as the laptop literally has its own battery back-up system built in!

It's not the fastest solution, its in fact extremely slow, but as an entry step it's perfect.

See the full video below, throughout the commands I list were in use...



Commands:

Listing Disks by ID...

ls /dev/disk/by-id

Listing Disks to a file for use in a file script as you see me using...

ls /dev/disk/by-id -1 > disks.txt

------------------

To install ZFS on Debian/Ubuntu linux:

sudo apt-get install zfsutils-linux

------------------

To remove & purge ZFS from your system:

sudo apt-get purge zfsutils-linux

(and you will be left with "/etc/zfs/pool.cache" to remove or back up yourself).

------------------

Command to create the pool...

sudo zpool create <Name> mirror <DiskId1> <DiskId2> etc...

The name we had here was "tank", if you already have data on these disks you will need to add "-f" to force this change through.

------------------

Command to make a file executable - like our sh script:

sudo chmod +x <filename>

------------------

Zpool Commands:

sudo zpool status

sudo zpool import <name>

sudo zpool scrub <name>

sudo zpool clear <name>


You will want to "import" if you completely remove ZFS or move one of your sticks to a new machine etc, simply insert the disk and import the pool by name.

Scrub will be used whenever you return a disk to the pool, remember the point here is to allow you to replicate the data across the three sticks and be able to remove one or two to safe keeping, be that an overnight fire safe, or taking a physical copy with oneself.

Clear is used to remove any errors such as the Pool becoming locked out for writing - which it may if a drive, or all drives are removed - you simply clear the current problem with any pool.


Summary:  Remember this is NOT the optimum way to run ZFS, this is actually extremely slow, you are replicating each write over your USB, you can only cache so much in the RAM, but it is not a performance piece, this is about ensuring one replicates data for safe keeping, a small office or your dorm room server setup could be completely provided by a laptop in this manner, it has it's own battery backup, it is quite (if you get the right machine) and really this is a very cheap way to play with ZFS before you move onto other bigger hardware options.  Plus, I find the best way to learn about technology is to break it, even a little, and so constantly breaking down your pools by pulling USB sticks out of them is an excellent opener to recovering your pools.  Play about first, don't put anything critical on there until you're really happy with the results.

For an excellent post covering creating ZFS pools, cheak out programaster's post here: http://blog.programster.org/zfs-create-disk-pools

And for the official ZFS documentation you can check things out with oracle here: https://docs.oracle.com/cd/E26505_01/html/E37384/toc.html


Oh, and Happy New Year... I guess I made it to 100 posts this year...

Friday, 29 December 2017

Windows Defrag v KingDian SSD... FAIL

I'm sure you all already know this one, and many vendor specific implementations of SSD drivers/applications prevent this... Not me though.

Yes, I have a dirty cheap (£27) SSD from KingDian in my Laptop... Wait, wait wait, stop abusing me!  This is just a boot disk to hold a local OS and some scratch space, all my storage is provided to this machine over NFS.
The trouble?  Well, in the late summer the wife and I had a few days away in deepest darkest Wales... Now this was a disconnected break, however, I couldn't resist taking the laptop loaded down with a few DVD based games and a steam download or two, plus I got to take some system code with me and a slew of PDF's I could read... I got a lot of work done, even without Linux (yes I had to flatten the machine back to Windows 7).

And I pretty much put the machine away until just before Christmas, I then needed it again, so flipped it out the cabinet and fired it up, only to immediately find the battery was dead... It had been dying, I rely on a meager source of dodgy, second party batteries from China... You know the kind of Lithium cells you'd not want to leave alone too long.

But being Christmas I was skint, so I've had to wait until the end of the month to get a new battery... Which arrived today, I left it to fire up and take it's first charge, everything looked fine, it was still Windows from the holiday, so I left it to it and I've been and enjoyed some TV...

Upon coming back up however, the machine is mental... Telling me I have no boot device, I had left the machine on... It has no network connection, so it can't have downloaded a Windows update and restarted... I was a little baffled... But fear not, I'm a trained professional...


This got us into a Windows Recovery, and after not giving a shit and starting normally, back into windows, where I started to puzzle out the problem....

Event Viewer, and Applications... Nothing but the panic about the non-clean reboot... System... Hmm... System shows the last event at 16:54 and then the reboot at 23:54 (alright, don't judge, I watched three films okay, and we had a take-away!)...

 So what was the event at 16:54... "Service Control Manager".... Hmm, which service.... "The Disk Defragmenter service entered the running state"...

OH SHIT... This is an SSD, you really shouldn't defragment them (it reduces their life)... So it tried to defrag and not only crashed windows, but made the device/bios not able to inter operate upon a warm restart.  That's quite a feat.

A hot reboot sorted everything, so I'll not dig any further, however, I'm interested in this phenomenon.  The disk is a cheap chuck away one, so I start the defrag manually... BOOM, crash, same thing.

I trawl through my box of bits and find an old OCZ 128GB SSD (which has about 26% life remaining - lol) and I throw Windows 7 onto this... Let it all set up and I come back (it's about 23:35 now, if anyone's counting).  Starting the defragger on the OCZ device, no problems.... It just does nothing, the management software from OCZ stops any nastiness.



Back to the KingDian, boot up.. .and Defrag... BOOM.

It's definitely something about the very cheap KingDian drive... Any ideas?  Drop it into the comments, but its back to Arch for this machine me thinks.

Wednesday, 27 December 2017

C/C++ Stop Miss using inline.... PLEASE!

This is a plea, from the bottom of my rotten black heart, please... Please... PLEASE stop miss using the inline directive in your C and C++.

Now, I can't blame you for this, I remember back in the 90's being actually taught (at degree level) "use inline to make a function faster", and this old lie still bites today.

inline does not make your function faster, it simply forces the compiler to insert "inline" another copy of the same code whenever you call it, so this code:

#include <iostream>

inline void Hello()

{
    std::cout << "Hello";

}

int main ()
{
    Hello();
    Hello();
    Hello();
}

Turns into the effective output code of:

int main ()

{
    std::cout << "Hello";

    std::cout << "Hello";
    std::cout << "Hello";
}

What does this mean in practice?  Well, you saves yourself a JMP into the function, and the position on the stack holding the return address, and the RET from the function which pops off the stack and returns from the function.

This is WHY people were told to use inline to make things faster in the 90's, I was taught this when I a system with around 254K of working RAM for the programs I was writing, saving that space on an 8K stack was important in complex systems, especially if you were nesting loops of calls.

However, today, on a modern processor, even modern embedded processors, DO NOT DO THIS!

You're no longer saving anything, you're in fact making your code bigger and slower as suddenly your program expands in size and you are having to fetch more and more from the slower RAM layers rather than the program instructions page fitting into the lower CACHE layers.

As you get page misses you fetch more, you literally stop the program and switch context to another item and then switch back, literally halting your program in its tracks as it suddenly had to go load the N'th of possibly thousands of repeated stanza's of code.

Don't do, this, don't lumbar yourself, let the compiler handle it's own optimizations, they're pretty good at it!

Now some of you will be saying "yeah, no shit Xel, what's your point?"... My point is I recently had around 4000 lines of code handed to me, a huge long listing, and around 40% of it was a series of functions.  This whole thing could compile down to around 62K.... But when compiled it was just over 113K... This was too big to fit into the memory of the micro-controller it was for.

The developer had been working merrily over the yule tide, happy and satisfied their code would work, they went to work this morning and instead of running the code on the IDE within an emulator, they actually ran it on the metal.

It crashed, and they couldn't figure out why, the size was why.

And then they couldn't work out why the code was so big... It is tiny code.

They came, cap in hand, to myself - and I took no small satisfaction in rolling my eyes and telling them to remove the "inline" from EVERY function... "But it'll run so slowly" they decried... "REMOVE THE INLINE".

Of course it works, they have the system fitting into the micro-controller RAM, the stack is working a lot harder, their code is a lot smaller, and they are now in possession of a more balanced opine on "inline".

* EDIT *

One person, yes hello Hank, asked me "why", why was this a not a problem on the emulator, but was a problem on the bare metal, well the bare metal was using a different compiler than the pseudo compiler for the windows based IDE, the Windows based IDE was actually running the code through a compiler which ignored "inline", and so produced code a little like this:

(Image Courtesy "CompilerExplorer")

You can see that even though "int square(int)" is marked "inline" it contains the push to the stack and the "pop ret" pairing, and making it a call from main results in two function calls to the same assembler.

The bare metal compiler did not, an undocumented difference I might add.

Monday, 11 December 2017

Crashed my Build Server....

When I say I crashed it, I mean... It just locked up and I had to soft reboot it... And when I say build server I mean one of the virtual machines on one of my Xen Hosts....

So, the machine is a fairly beefy 16 core machine with 48GB of RAM running on a Dell server under the desk, this disk base is a RAID-0 200GB unit over a bunch of 2.5" 10,000 RPM SAS Drives to a Perci5 RAID Controller....

The machine is only really spooled up for big builds, and this was one of them, I wanted to build LLVM support before bed.

The problem?  When I performed the build with "make -j all" it would get to 16% and then blank the screen, and totally lock up, nothing, nada, nowt... I left it for a while but nothing happened, and yes the LLVM build is time consuming but it doesn't lock at 16% for minutes.

Soft reboot, and the same happened again...


I've started the build again with "make -j 15" rather than all sixteen cores.  And it's already up into the 35% area of the build whilst I've been typing this....

But, what the heck locked the machine up before?  It wasn't actually using all the processors all of the time, surely?  Maybe?

I might have to set up one of the older 2950's and have a play about with this, leaving my one beefy machine alone.

Just on a side note, could you imagine the mess your system would be left in if you soft rebooted this kind of kit, mid-build, with no warning... LOL.

Tuesday, 28 November 2017

CMake rather than Mammoth makefile marathons

I'm having difficulty communicating with some folks about the beauty of cmake and using ccmake to leverage that beauty.

These are folks whom are either completely ignorant of what a makefile should look like, are happy to manage their own or at worst case are folks put off of makefiles by having inherited projects which have spiralled out of control with mammoth makefiles and a propensity to being so complex as to prevent any cost-effective entry grade for new developers to key into - i.e. they're too hard to learn, or obfuscated sufficiently to allow established developers to retain their positions of glory and power.

I don't subscribe to that ethos however, and believe that as a leader in development you should facilitate everyone to be being able to do everyone else's development role, be that starting a new project or continuing an old.

It perhaps comes from my being able to work alone and defining a role which others are then keyed into, I have been forced to allow entry to my work, to make the cost of someone else bootstrapping my work into their wetware (brain) as simple as possible.

Mammoth makefile marathons are not the way for me to do that, a CMakeLists.txt file, now that's a better proposition.  However, even here you have to take care, some folks are ignorant of the tools available, to leverage cmake in this way one might use this kind of command line...


This is daunting for a newbie, and even an experienced developer has to admit...

ccmake <PATH_TO_CMAKELISTS>

This is a much more succinct and easy to access way of getting into your CMake way of working, the cost of entry being so low as to actually make introducing new developers to Linux development, or just general CMake usage, trivial.

So where am I failing to communicate this?

Well, cmake, ccmake... The naming conventions of both are so close as to confuse people, they don't hear the second C in CCmake, or they think I'm talking about the C Programming language.  This is a lack of understanding on their part, people being people however, they don't want to admit they've no idea what you're talking about.

(As an aside, folks, if you want to be a good developer, a good person, please admit when you don't know something, it causes so much less issues in at development scrum time if you are handed work, and you simply state "I know nothing about that".  Someone else can be assigned the role, or better still, you can get training and schedule the work more effectively!)

My solution to this difficulty therefore?

Rename the programs, I've created two symbolic links in /usr/bin....

sudo ln -s /usr/local/cmake/bin/ccmake /usr/bin/makefile_prep_gui
sudo ln -s /usr/local/cmake/bin/cmake /usr/bin/makefile_prep_cmd

I've essentially bamboozled the communication factor by giving cmake the working name "makefile_prep", this means that those opposed to ceasing direct use of makefiles still feel empowered, but are subtly diverted to using an automated tool.

Immediately questions and opposite to changing the status quo has ceased, and folks are talking about using the new "makefile_prep" tools.... How clean they are, how nice the builds look, how they integrate with CLion easily, and "the output from my makefile_prep looks exactly like the build going on inside the IDE (CLion)"... Little do they realise they're both cmake!

Oiling the cogs of resistance to change, this is where I'm living at present... Its not an easy task, but sometimes it's rewarding.... Now to do the same in the day-office.

Wednesday, 8 November 2017

Software Development - All Areas Stagnation

It has been said by far bigger and better minds than myself, that if you sit still, if you don't continue to learn about new things and innovate you will stagnate.  This has been a huge problem looming within the business I work, certain things have worked since the industry sector was conceived and even though more than half a century has passed it has largely passed the internals of this industry by.

That is until very recently, where market competition has sprung up, the market base itself has reduced and so pressure is on... Nowhere is this more apparent in my industry than on the software, the front-line of pushing product to customers.

The trouble however seems to be that many people have stagnated, they've stuck with the safe option, the tools which work off of the shelf, I am of course talking about Windows, the entire tool chain that is used by 99% of the company is all Windows based, I am the man on the spot waving the Linux flag.

But just a few days ago, the Windows world had to come to my desk and see their future, I had to show technically minded folks around the code of the new system, introduce them to my imposed coding standard and update them from Microsoft Specific Visual C++ thinking to thinking about platform independent Standard C++ code... I had my work cut out for me.  I prepared the cleanest desktop environment I could (i3 on Ubuntu).


I didn't want to startle them, so the editor/environment is Visual Code from Microsoft... They started to look at the system, it's structure, how the code related to the design and the diagrams they already had, we started to follow the process flow diagrams.

It was a success, certainly no-one burst into tears, they saw the kin-ship between this code on Linux and the systems they'd worked with for decades on Windows.

But then, the senior software manager leaned down, peering at the screen, and he said some fateful words...

"I've never seen that before".

Is he talking about some piece of C++14 or C++17, the lambda's, the auto's, the shared_ptr... What technical bolt has he not screwed his nut around?


"That's very good, you can see the whole code layout.  I've never seen that before, who did you say wrote this tool?... Really Microsoft, I've never ever seen that before".

This chap uses Sublime, I've seen him using Sublime... Which does exactly the same thing....


What is the lesson to be learned? When we're talking about stagnation in software we are not only talking about the language, but also the tools, and then not only the IDE, the whole environment.

Certainly I was introducing Windows users to Linux, and even then on an unusual minimalist desktop manager, but still the lack of connection between a tool I've seen people already using and what it was capable of demonstrated tools are not being leveraged to their full potential... Certainly learn your new languages, learn your language updates, but keep your tools and environment up to spec too...

Thursday, 2 November 2017

C++ : Ignored qualifiers (-Wignored-qualifiers)

What is an ignored qualifier?  Well, in a class, lets have a student class:

#include <string>

class Student
{
    public:
        const std::string Name;

    private:
        bool m_Present;

    public:

        Student(const std::string& p_Name)
            :
            Name(p_Name),
            m_Present
        {
        }        

        void SetPresent() { m_Present = true; }

        void SetAbsent() { m_Present = false; }
};

Now, we may want to access the present flag, therefore provide a function to do so:

    bool Present() const { return m_Present }

This function is telling the user of our class quite a bit, it tells them that the return type is boolean and that the calling of this function makes no changes to the class  contents (const trailing the parameter list).

However, for me, this isn't quite right, I believe that we want to inform any user that the boolean returned is also constant, it does not change unless you alter the internal value with the "Set" functions, therefore I prefer and like to see code stating:

    const bool Present() const { return m_Present; }

This is perhaps overkill and most of the time completely acceptable code to present, however, some might prefer not to see it, specifically anyone defining "-Wignored-qualifiers" as with the "const bool" the const here is technically superfluous, the return type is boolean and a new instance of it, it is not a reference to the internal value, if it were the function may look something more like this:

    const bool& Present() const { return m_Present; }

Now we are intrinsically returning a reference to the internal boolean, or even:

    const bool* const Present() const { return &m_Present; }

For speed of operation we may directly drive the reference back as a constant boolean constant pointer.

I find this much more informative to the user, they know our intent, the code though more verbose communicates its meaning much more clearly.

As ever, yes I have seen questions asked of interfaces where "bool X()", or "bool X() const" is provided but then programmers have asked "How do I change X", with the const return, ignored qualifier or not, the know not to ask this function to change X and can look up elsewhere in your API.

Monday, 19 June 2017

Bash : Power of Pipes

Subtitle: "Get IP Address Easily"

When I say easily, I mean not so easily, but with the proper tools... Let me explain, it's been one of those days... I've a remote server running some flavour of Linux, and no-one knows it's remote IP Address, they all SSH into the box run "ifconfig" and note down the value, they then plug this into a config (or worse still were baking it directly into some code) and running their services....

The trouble of course being, years later, they're no-longer the programmers nor maintainers of this machine, I am...

And to be frank whenever the IP address changes I don't want to recompile their java code, nor use vi to edit the various configuration files, I want a script to at least update the settings automatically.

I therefore changed their code to load the IP address, not hard code it, and used some other scripts to put the IP address into the config file at boot...

The first line of that script is what I'm going to document here... so it starts:

#! /bin/bash
ifconfig | grep inet | tr ' ' '\n' | sed -u '/^$/d' | head -2 | tail -1 > ipaddress.txt

This script gives me a single line of text with the IP Address in it, for the one and only adapter in the machine, if you have multiple adapters you'd have to play about with the grep inet to select the row you want with a head & tail call before moving into the  final location, or whatever...

I wrote this up however, and immediately started to use the IP address.

The net result was a request to explain all this functionality to a colleague... Here's what I came up with.

ifconfig gets us the adapter information...
grep strips off lines we don't want only giving lines for the inet adapter
translate turns spaces into new lines
the sed call removes the blank lines, giving just the IP address and some guff
the first adapter IP address is therefore always the second line of this output
we select the first two lines with head
then select only the latter of these two with tail
and write this to a file

Her reply... "What are the Lines?"....

"What lines?"...

"These | things"....

"They're pipes, I'm piping the information from one program to the next..."

"Oh"

"Do you know what pipes do in Unix and Linux?"

"No"

I sent her to this video... https://youtu.be/XvDZLjaCJuw?t=5m15s

Thursday, 4 May 2017

Software Engineering : My very bad way to set up makefiles

I may have wrote this post, but I'm just being lazy... Read on.

I've made other Boost videos, and other codeblocks videos, and I've always assumed that the user at the other end was well aware of how to compile & build code.

However, it seems there are lots of users whom are not aware of how you actual create a program from source code, at least not with C++.

So I'm going to show you the absolutely most wrong basic way of creating and using a makefile to perform a build...

The first step of C++ program creation, is that trivial matter of creating the source code, easy right?... Yep, so lets skip that step, and go to when its once  and we want to pass the code through a compiler, the compiler turns the source not into a program, but into an intermediary format called an object file or object code.

The compiler then leaves the process and another program takes over, this second program is known as the Linker, and it takes the object code and links it with all the system calls and other libraries you are wishing to use.

There are other steps involved in the process of building a program, however, compiling & linking are the two main steps; when I began programming you were made very much aware of this by the compiler and linker generally being such large complex programs and the libraries being so big you had to physically change floppy disk between the different steps of the process.

This process was also taught to me in school & later college, it was common practice to understand a compiled program was actually compiled then linked.

Unfortunately, this knowledge seems to have fallen in to the way side, it is of course more important to learn how to program than be bogged down in minutia, however, once you have learned I strongly believe you should expand your knowledge and really have a holistic knowledge of the craft of programming.

So, back to our program, the make file, what might they look like? Well they are simply text files,  which contain commands very similar to bash script... We call this make file script by passing it to the "make" program, you can get make for Linux, mac or alongside MinGW for windows.

Lets start writing a make file for any basic C++ program, which links the Boost libraries we've built somewhere on disk, and which looks like this:


#include <iostream>
#include <string>
#include <boost/filesystem.hpp>


int main (int p_argc, char** p_argv)

{
     if ( p_argc > 1 )
     {
          std::cout << "File or Folder ";
          std::string l_pathString(p_argv[1]);
          boost::filesystem::path l_path(l_pathString);

          if ( boost::filesystem::exists(l_path) )
          {
               std::cout << "Exists!";
          }
          else
          {
               std::cout << "Not Found";
          }
          std::cout << std::endl;
     }
}

Save this as "main.cpp" and then define a variable called "FILES" into which we place this filename.

FILES=main.cpp

We want to start our make file with a variable which contains the compiler we wish to use, for Linux on my test machine I'll use "g++".

CC=g++

So now we have a value called "CC" which is a string of characters with the value "g++".  The dollar sign indicates that "CC" is a variable within the script, which we might want to use later, I use these kind of variables throughout makefiles so I can place at the top any paths or references I like, or I can alter them when I move the makefile from Linux to say Windows, without needing to change hundreds of places throughout the script, I can change just one location.

I find this working with strings is 95% of what working with makefiles is about, next in the file we then define the strings for where to find both the boost headers and the binary libraries are...

BOOST_INC=/home/xelous/boost
BOOST_LIB=$(BOOST_INC)/stage/lib

Notice, I create the root folder for boost as one variable, and then in the next line I use that variable to extend into the "stage/lib" folder, so I never duplicate the root folder for the boost libraries.

We can now define at least our build command, slightly different the command is a verb, in this case "build", which is the command we will use here, but which is also the default.  If you save this makefile as "makefile" in a folder, then navigate your console to that same folder and type "make", the make program will look for "makefile" by default, find it and build the "build" verb by default!

build: 
   $(CC) -I$(BOOST_INC) -L$(BOOST_LIB)

So, what is this command doing?  Well it is saying to use the compiler, with the Include folder for boost and look for libraries in the boost library folder we've set.


The "-I" and "-L" parameters are the same parameters one would use on the command line for "g++".  This command therefore equates to "g++ -I/home/xelous/boost -L/home/xelous/boost/stage/lib", but I think you will agree it is somewhat easier to manage.

We also need to tell the program to link against some libraries, and that we want them linked statically in this case.   So our next variable will be called "CFLAGS1" to pass the Compiler flags to the command.  Back at the top of our make file I therefore add....

CFLAGS1=-fpermissive -Wl,-Bstatic

This file tells the compiler to be permissive of some common errors and treat them as warnings.  You can use whatever parameters you wish for your compile, the flags available and their meaning are quite varied, however you will need to use a second set to within the build command, so lets define that now....

CFLAGS2=-static-libstdc++ -std=c++14

The second tells the compiler and linker to link against the static versions of the libraries we select later, you will need to look at the boost invocation pages in order to go back over our previous posts and build boost static and take advantage of this option.

I'm not going to explain static linking here, nor any of the other switches, available.

Now we're linking the C++ standard library, and boost, statically we need another variable with the list of libraries to link, on the command line to g++ this is with the "-l" (lowercase L), except with we are going to make the program threaded, so we can set up a new makefile script variable called "LIBS", into which we set the libraries, like so.

LIBS=-pthread -lboost_signals -lboost_system -lboost_filesystem

We now have the compiler, the folders, the libraries, the flags... What about the target file for the build?... Well I want it to be an executable called "
fredrick".

TARGET=-o fredrick

Now we need to put this all together into a command, just running the script with the variables does nothing, we need to define a verb... Lets call this "build" a verb in the makefile has to have a colon following it...

build:

And we can tell make what to perform for a build:

build: 
   $(CC) -I$(BOOST_INC) -L$(BOOST_LIB) $(CFLAGS1) $(FILES) $(CFLAGS2) $(LIBS) $(TARGET)

We might also want to clean the project, so we'd want to delete the target file maybe, by adding another verb:

clean:
   rm $(TARGET)

We can then perform the call "make build" or "make clean" in the folder with this makefile and perform our build.

A bit rough and ready, but it works to get you up and running.