XVE Game Engine: Thinking about AOE Damage Effects

I'm back on my home game engine and I need an area damage effect, which can apply to all the members of a team/party.... The trouble?  How to synchronize this over a network connection.

First of all, I know for certain that the server running my game world knows the center of the effect, and it knows the radius of the effect.  And it does tick the effect application against each player whether their client can visually see the effect - it appears in their combat log text immediately.

So the mechanics are there, the clients however, all run at different rates, they're in different locations world wide, with different latencies to the servers source of truth.... And so I need to now think about how I am going to tackle the issue of each player seeing an effect.

Simplicity first, the effect is just going to be represented by a wireframe sphereoid, it'll be a smooth shaded sparkles dripping effect later.

This quiets my engine engineer smooth brain some what and I now just need to synchronize the expansion from the center to the maximal radius of the effect, this will turn up at the clients from the server at any time after the effect is registered as affecting the players on the server... This maybe unfair, however, there is reason behind this in the design - the player will know the effect is going to be cast as there is pre-warning from the caster... They need to learn that and move before they see the effect basically.... That is, they need to be smart.... I know I know this is a cheap answer, but I am all about cheap answers to problems today - and in my gameplay loop, seeing the caster "spit" this effect and moving before it expands into a cloud of hell... is pretty rewarding.

So I have a time point, on all clients they know when the effect begins, they know the maximal extent and they know the center of it....

What I am thinking about now is the client side only, so without any network traffic, an entity definition for the effect which the client has pre-loaded into some dictionary of effects to execute, and this contains a rate per millisecond from the start time and the effect system has to linear interpret along the line.

I am thinking about designing some form of easing curve authoring for the client effect system - and I'll likely reuse this in the animation system - whereby I can try different easing curves.

 The system interpreting along the line could be more than a lerp too, again that'll be down the line.

Replaying, reversing or otherwise on such a defined effect is going to be quite interesting as reversing the delta will allow me to contract it again.... Though time-line ordering is to be forwards chronologically only for my game play, that maybe a feature I want to play with in future.

What's the problem then?  Yeah, it sounds like I have a handle on what I want, except.... I've assumed all the way through my thinking that the effect is fixed, known and defined up front.... The problem?

I want to scale the effect with the power of the caster, and this is not known up front, sure I can just add a float and multiply by 0.5 for half power or anywhere along that scope, heck I could even scaled the enemies power by some easing function as well and combing the power function along that line with the rate of the effect.... So there are answers to be had.

My problem is randomness, when it's random, things look wrong, they don't feel right when playing them and the rewarding feeling I had from the fixed function gameplay doesn't turn up... It's in some fort of metaphysical uncanny valley which I can't quite explain doesn't feel right.

I will have to play about some I feel.... 

Home Game Engine : Physics Debugging Tool Progress

The last week has been invested into the Physics Debugging tool, it connects to the main game world; whether that is running in the client with it's own vulkan renderer, or into the server, for I am planning to have the server be authorative over player position at least, to prevent some of the strange hackery doo of the client being in charge.

Here's a silent video of that progress, as I'm recording from my Ryzen Workstation and I don't have a mic in here.

There is actually a lot going on in the background here, though the scene looks largely unchanged.

The most obvious addition since the last scene update is the replication of the rotating object, which is just a box, but that is being rotated on the client.  My controls here send a message to the client, which then sends the status update to it's GameObject.  The same frame the GameObject queues a message out and the scene updates to the physics representation; which means I get near real time (minimum of network delay + 1 frame) of the replication of an object back and forth.

However, I don't plan on expanding that too much, the number of messages is getting silly.  For example, I have an object for a position update (3 floats) then I have one for a position and a rotation (6 floats) and then a whole other one for position, rotation and scale (9 floats) which I call a transform.  I could just replicate a transform each frame, but then the amount of data gets very much larger.

I also have flash backs of delivering data from a server on a provisioned box, where you pay by the megabyte per monthly usage (and I honestly don't know why I couldn't host a service like this server publically on my massive 500mbit home fiber, which has no cap on data, it's all I can eat) but practically, I'd not like to host this here, except for debug, and if I were to make a game, it'd need to be on an AWS instance or something.

Anyway, that's all very much future stuff, my next problems are all about the game, improving and cleaning up my game object authoring, improving my model making skills (I might actually have to double down and actually learn Blender) and the working out a few issues I know exist in my control scheme.

More about all that in March though, for the rest of February, I have to tidy this stuff up.

If you want to know more, or follow some of my other older projects and videos, my YouTube Channel exists https://www.youtube.com/@LordXelous and of course this blog is always ticking over!

That Subscribe button really helps!

Naming a new Storage Server?

 Server names?  In practice I've always been in places or projects where folks are bad at naming things and I am as guilty as the next person in that realm.  I've been on projects naming releases for obscure lord of the rings characters, for characters in a random fantasy story the lead developer had written, for birds, biomes, planets and stars.... Generally these are very niche, if you know the reference you're fine, and it brings a little character to your server.

The trouble arises when a name becomes attached to the history of that thing, the worst I never had was a project with a release named Elrond, where the release itself was re-released, so we ended up with Elrond-2, 3 and 4, then Elrond4RC1 for an update to that successful update, it was just a minefield.

In servers, the worst I ever managed was a box called "Diomede", which would often fail because of bad (read that as cheap) hardware, so the name became inextricably linked to bad performance and issues, despite being a really nice Tomcat & IBM Websphere host with a near 99% up-rate, but that 1% let it down because EVERYONE worked on that one box.

At home at the moment I have:

* Wellington

* Napoleon

* Elba

* Ney

* Waterloo

I'm sure you can tell the reference, but I've had about three versions of "Waterloo", I've also had two versions of "Ney" and at least five "Napoleons".  The names are getting mixed up, as I leave a home project and come back I'll go look up my notes, pull up that machine storage only to realise "oh this was on the OLD version of that name" which was long whisked away into the ethereal world of my replication server or just lost to time.

As such, I want to have ONE big disk server, and I really want to make it nice and neat.   I've already retired all my older machines, like this one:

But I do now have TWO cases of this specification, with the many internal drive bays:

In one of these I have a low-power Core i3 T class CPU, I forget the exact spec, but it's a low-power chip (35W) just enough power to run storage backups IMO, so this is where I'm going to start my quest.

Spinning rust drives are the challenge, I have four 4TB drives just sat idle, plus a host of smaller 1TB drives lying around, these need sorting out, mounting, checking and I then need a strategy.

I'm very likely to go with ZFS creating a mirror across three 1TB for critical storage and then some striped storage across the 4TB drives.  I then want this to make files available across the network, both to my 10GBit and my 1Gbit LAN's.... That's going to require a bit of work to install the former, but the latter should be available out of the box.

This starts to completely retire my 4Gbit fiber channel interconnects too.

So what to name this first stepping stone?.... Comments below:

Elba

Helena

Since it's about storage, I think my theme of names points to one of the islands which hosted Napoleon in exile... but lets see what you guys come up with.

Home Network Upgrade - Part 1

This is going to have to be a multi-part series, firstly as it's a complex task we've got on, but also time constraints from both my side and also from suppliers delivering me kit means it's already spread itself over three weeks... and I can only imagine it's going to take a lot longer yet.

What am I up to?  Well, we're all settled in the new house and it's time to sort out the networking... Now I only have wireless at the moment, you may have seen my prior posts about the lost router?  Well, I found it... And I was wrong, it wasn't lost at the in-laws, it was lost here.... After searching the garage again, with the wife for that fourth time she finally believed me that it was lost (not just that I was being a stupid man) and she helped look elsewhere... We still didn't find it, however the next day at work my phone rang and on the other end was a joyous voice "I found it, the router, I found it"...

Yes, it was in a box in the garden!

Not open to the elements, but not fabulous either, anyway, router found I stuck it in place.  The problem now, I had two wireless networks the router provided one and the BT Homehub one... Easy, turn the BT Wifi off.... Which I did.

Except, that doesn't turn off the BT Wifi with Fon thing, the free access point for any BT customer to use.  And, I saw folks using my Wifi*

So, I hit the old interwebs and looked up a decent ADSL/VDSL modem replacement, the one which came up, from a video by Cameron Grey actually.  I ordered a Netgear DM200.

It looks quite nice, this is just an ADSL jack in and RJ45 jack out, it is NOT a wifi router.  But, this is an utter and total pig to set up folks, really a total and utter pain in the bum.... I tried three weekends in a row in six different sessions each of over an hour before it finally sorted itself out.

For those interested, flash the firmware to v1.61.  Give it power but no network connections plugged in and hold the factory reset button on the back for 30 seconds, then kill the power to it... Plug in the ADSL feed and connect it with a single ethernet cable directly to your laptop or PC.... Give it power and wait... and wait... and wait... until all three lights go green... If they don't after like fifteen minutes (seriously wait that long), only then remove the connections, but leave the power on factory reset again and power off, connect and power on etc until you get three green lights.

Then on your PC browser open to "http://routerlogin.net" and get in there, and just run the Genie wizard, if ANY thing else comes up, factory reset!  Let it detect BT, enter the username as bthomehub@btbroadband.com no password, nothing else and wait literally tens of minutes, it will get somewhere.  And when it's asking you to register on the netgear site pacify it with the "next" button until it is totally and utterly and completely done.  Only then open another tab and try a website.

When I finally got a website I then removed this hunk of trash.

Finally, no BT Wifi with FON sucking on my juices.  I set the DM200 into modem only mode, wired it to my Router and set up PPPoE with the username and password for BT again... Anyone wondering bthomehub@btbroadband.com and BT, with VLAN set to 101.

And voila, I was back to where I was with my virgin router at the old address, I now have my nice router working nicely with a nice modem on this BT line... And it's about 5mbps quicker on my speed tests, the DNS resolutions (set to Google and CloudFire are around 30ms quicker too).  And I have more control over everything, I am so much happier.

But I'm not altogether happy, because the next phase of my work has been massively massively delayed, by like a year... Like this was planned for the old house... Fiber Optic.

Yeah, remember this box?

Hang on, lets be clear... THIS BOX...

Yes, they are fiber optic network cards... You might ask "Xel why have you so many?"... And I would have to blush and admit, I thought I was buying two as a test, instead I bought TWO JOB LOTS OF 10 CARDS!  So  have this huge box full of 20 fiber optic network cards.

The plan therefore is to put together a small form factor PC by the router and fit a fiber connection to it.   Carry that fiber line up into the loft and up there have a fiber optic router.  That's the ultimate aim.  But I want to do that test first, machine to machine... So, I fitted a card to my workstation and another to my my Xeon home server... All fine, drivers and everything fine, and I set about ordering a pair of short 2 meter cables...

And I wait...

And wait....

And a month later finally....

Oh yeah baby... Except, hang on...

I ordered a pair of cables didn't I... Didn't I... Yes, yes I did... How many are in that baggy?... Just one... OH NO JUST ONE!!!?!?!??!?!!  So, this is where I leave this project for now, it's been a long time coming, but I hope to get the optics test done, then we've got two runs of ether cat 5 or fiber to run from the router feed in down stairs to the loft and then to my office and to the garage.

As owning a garage, this is where I now plan to have the server rack, and running fiber to it may be my optimal solution, it may even be possible to get cat 5 into the garage easier than to the loft, so all the fiber might go in the garage, this is literally a plan in the process of being figured out as I find out the structure and layout of the house.

Stick around for more.

* I actually spoke to a neighbour about this, he said "Oh that'll be my lad, he connects to the as many BT Wifi connections for free, we don't pay for the internet that way".... Nice, thanks... 

My Home Network

My home network has been neglected, this is one of the problems of working in Technology, when you get home you are not going to be doing very much technology... Or maybe I'm just too interested in other things... Whatever the reason, neglect has set in.

Lets take a look at my home setup:

The network is broadly speaking split into two, the parts downstairs and the parts upstairs.

Downstairs is pretty much the Cable Modem box, a home hub set to Modem Only mode, connected to a Linksys router with some network attached storage for dumping files or media.

Upstairs is where the trouble begins, the critical path is to the left two upstairs items, my Xen server and especially my main Workstation, without these I can't work at all.  So thats two connections which I do not mess with, moving down we see the dotted yellow line, this is a single cable which lays on my desk that pays double duty to my powerful laptop or a raspberry Pi, so that's a spare I generally always need around.

We've used up four of our eight ports on the switch.  The final four are each going to the DRACs on my Dell Servers, which means they have no data connection.  If I want data to them (which is pretty much every time I use them) I either pull the wire from a neighbours DRAC, or I pull the cable from the IP Camera; it being the eighth and final port on the switch as it stands.

I have to ask what my options are... Well, I'm not able to change the cable trail from the lower to upper floor, so only one Cat6e cable there still, which means its not yet work my moving a box downstairs and dedicating it to pfSense & Squid.

I also don't have any rack space, so a racked switch with more ports is really a waste of money at the moment.

I took a look at another Netgear GC108 unmanaged switch, but I wondered about sub-net masking out some of the server stuff and thought for the few pounds different I'd go for a managed switch.

The desk then gets a new dedicated managed switch, and the more server stuff all stays together on the unmanged switch...

With the Dell servers immediately taking up six ports between their DRAC and data NICS I have two left, one for the Xen server and one to cross-connect with the managed switch.  When I come to rack these machines in a better manner, then I'll be able to co-locate the 8 port switch into whatever solution I have there without rewiring my main working station & desk!

This leaves me two ports free at the desk, in a unit which is light and small enough to screw to the underside of the unit (no way I could do that with a larger unit).

The real beauty for my needs here is the switch interlink, I can unplug just one wire and take away all my servers for re-positioning or remounting.  I could even take them all out of my immediate workspace now and hang them off of the WRT1900 downstairs (as it has two RJ45 ports spare).

Note to all the haters... On the topic of the Linksys WRT1900ACS, there is a lot of talk on the internet of it being flaky, unstable, crashing, resetting to factory defaults... Mine has been nothing but stable, like really stable, it's been reset once due to my needing to clear the cumulative six months of network map details and I think it locked up once with the WiFi not coming online.  This doesn't mean to go out there and buy one (though I did review it on Amazon, go take a look) but rather the unit I have was very good, and remains very good.

P.S. Neither Netgear nor Linksys sponsored my usage of their equipment, but you know, if you want to... Get in touch at the link near the Tip Jar!

Story Time : Fantastic Rack Mount Mistakes #1

This is the first in a little series, which I'm afraid will have no nice pictures, but trust me, these are all true and if not first hand, at least second hand accounts from trusted sources...

You may or may not have seen a server room in your time, if not then they're basically rooms, generally locked, with at least air conditioning, generally a fire extinguisher nearby and all your company server assets inside.

When I started out in business the server room was a large cupboard with an air conditioner and the 386 based NCR mini-computer inside.  It also had set of aluminium shelves, but it had no rack mounting, everything stood on the floor.

This was true for me, for a few years, the IBM AS400's we moved onto were huge machines on the floor, aside from the colour you felt like you were walking into the set of "Flight of the Navigator"

However, I soon moved over to Compaq Proliant machines and these were rack mounted... I walked into the server room, and took a look at the rack, only to be greeted by the UPS battery pack being at the top of the rack... Yes, the batteries, full of acid, were at the top...

I came out and went to speak to the nice chap in charge, pointing out, as gently as I could, that the batteries may leak, or if there was a fire could be ruptured, and it was not unknown for batteries to get hot swell & burst...

He stared at me, blinking, "What's wrong with that, they're in a case".

And he was damn right, they were... a Plastic case, this particular battery pack had no metal sheeting, no casing other made of metal, just metal corner brackets, it was essentially plastic...And full of acid.

In story number 2, you will hear about the server room fire I'd dealt with the previous year.  But this chap didn't seem to register a problem with having the battery pack at the top of the rack.

I did however lodge my complaint in the service book, and went about my work.  Months passed, and I didn't often go to this site, we moved all the other company assets to rack mounted ethernet switches, a few smaller rack mounted Pentium II based machines and all new desktops.

When one day, a user came to my office door and said they could not get onto the central server, checking my desktop serial terminal sure enough nothing.  Checking a physical VT100 nothing... Oh no.

I call the other site, no answer...

I get a secretary to start calling them every five minutes and I start to check back through our logs, it seems from our end the link was active at 6:03am about two hours before business would ramp up during the day, and the last items were a set of data from the design department at a Derby based location.

So I call Derby, asking "Did your upload complete at 6am?" a little bit of silence and I finally get the right chap on the line, he'd come in very early to get a jump on the day and to the best of my recall he said "No, it suddenly stopped and I could not get back on".

I cut him off as the secretary suddenly pages me (yes, rocking the pager) and I pick up my internal phone to hear I've got the guy from the server site on the line, he's coughing... "What happened?" is my instant question.

"Halon was active when I arrived"...

This is pretty bad, the server room was protected by an automatic halon fire extinguisher, if it detects smoke or heat there is a 10 second alert, the door closes and halon is released... 

Remember the lab scene in Terminator 2?...

A lot like that!... Staff in the room, or entering, need to use a breathing apparatus, which we'd all been trained on.

This guy was short of breath because he'd been in the halon room.

He says that when he arrived (at 7am) the room was in close down, the building fire alarm was going and a fire engine was already present... The security staff had let them in and they'd used breathing kit to enter the server room, to find....  And this was the best he could describe:

"A smouldering brown mess of plastic encasing the top of the Proliant server, stuck in all the front vents and solid into the quick release drive catches, all the network wires above are only copper and the battery pack is black"

I dive in a company car and drive up there, I arrive as the fire brigade are happy it's electrically isolated, that the halon is clear, we've got a guy from the fire suppression system coming to strut his stuff, but in the mean time the company is offline.

So, I set up a 10Mbit ethernet cable from the main input multiplexer, set up a backup system on an AST dual core machine (literally, it was a PC with two 486 CPU sockets) and 16mb of ram... power!

This was to be the tide over machine, we cycled the nightly backup onto that and at around noon, people were working, if extremely slowly.  That machine was our stop gap for the next 48 hours.

So, now to the server room, initially the most obvious problem (because the rack had a glass door closed) was the ceiling and floor... The ceiling tiles were seemingly not flame proof... DOH.... They'd melted at least and there were droplets of melted whatever they were all over the place, blown around by the AC.

The AC itself had valiantly kept fanning the flames, so it had black soot marks neatly blown all down the wall.

And the floor directly around the cabinet base was charred, they were fibre carpet tiles on concrete, but had smouldered...

Our fire plan had fired, and the halon tanks were empty, both had fired, despite perhaps this only needing one... The vent out had worked and we'd used up one of our internal breathing kits, so two spares, one inside one outside.

The room is also electrically dead, nothing is on inside.  We had redundant feeds in from sites, one inside the server room and the one outside down the corridor which the AST machine was now connected to.

We're using torches, so I opened the front door of the server rack, and it's just black, the glass is coated in black, the once cream coloured Compaqs are black, the silver rack arms are black, going up the patch panel above is black but you can see the wires look a lot thinner, reaching up they're charred, the plastic has sloughed off of them and later when I have a step ladder I see it's mostly pooled onto the top of the first computer, but it's also seeped into and set within the vents on top and spilled over into the front of the machine.

Above this is the crispy remains of the UPS, there is a hole about 2 by 3 inches around 8 inches from the left of the case bottom, and I can see into the UPS... It's a charred vision of hell.

We take this UPS out of the rack, as it is the ONLY thing in there that had been put in by the local chap, the ONLY things not on the service list, the ONLY thing I had had an issue with.... And we pop the lid, one battery had gone, catastrophically, it had melted acid out, the batteries lay on their side and so the corrosive acid had eaten the rubber off the outside of the wires inside, shorted and an electrical fire had ensued.

The battery acid had then, xenomorph style, eaten out of the bottom of the case and dripped onto the ethernet patch cables, they'd shorted, got hot, dripped more, the acid had also cut through the plastic on the power cables, more electrical shorts... And the gates of hell had opened swallowing the nice working machine.

The clean up was epic, we lifted the servers out to the cleaner area of the IT room, we cleaned the outside and had a pair of Compaq chaps come and clean them... I think they were called Mark and Jeff, and they did an excellent job.

I ordered new cable and thousands of RJ45 connectors and a crimper, setting about rewiring the room, in a better manner.

And I set the guy in charge of the site, Mr "It'll do" to taking out all the racking, jet washing it, putting it back in, redoing the power lines with a guy from the engineering department and I also had him remove all the carpet and the ceiling tiles, and find a supplier for a proper fireproof ceiling.

I also checked with the fire extinguisher guy, and he reckons that we were lucky the room was an hour outside of working hours, because the roof was just ceiling tiles and should never had been fitted with this strange fire system... Despite his being from the same company!... Basically most all the halon had exited via the ceiling tiles!

Two twenty hour days after all this I had one clean Compaq server, one clean rack and enough ethernet connections to put one patch panel and the server back online, this let our users stop moaning about how slow things were.

The other Compaq, the one closest the mess, was going off to be re-cased, the damage was so bad, but the unit itself worked.

We also now had a new fire safe, as the original had proven to not be smoke proof!

And I had a new certification on the walls being fire proof.

It took three months to recover from this, my end report to the owners, was that it'd cost about £8,500.  Not a bad deal considering the whole rebuild of the whole room would have been £40K+

I'd instigated better carpet, better ceiling, better procedure, and I had the brand new branded, supported, service contracted UPS in the bottom of the rack.

The final wash down, was the UPS which had failed, was known to be bad, three weeks before it'd gone up, the site manager had had it out and replaced two of the four battery cells within with a pair he'd bought cheap, he'd done this himself, I was not even aware, and he'd had the bottle to charge the company twice for the batteries, pocketing the difference.

Embezzlement, bad attitude, bad procedures and simply his considering me below him, resulted in his dismissal soon after.

He was the first person I enjoyed firing, he was the first person I'd actually had to actively fire from a role for incompetence, and he was the last person I let dictate to me where kit in a rack sat.

All for the placement of a UPS in a rack...

(Paul, Andy, Max if you recognise this story, you know you were a big part of that clear up, but you were in London, Leicester and Morocco... I was up to my arm pits in that mess!)

Software Engineering with 252117761

What's with this strange number Xel?... Well, this is a very useful number to help you determine how a remote system, or your network, is presenting numeric values.

When we program, we generally stick to one system or one platform at a time, however, life is never always so vanilla, and we've had a problem in the office today of talking to a raw network connection from a remote system.  We didn't know anything about this remote system, through a combination of lingual differences and a total lack of documentation (SNAFU).

So, we didn't know what endianess the remote processor was treating numbers as, and we also didn't know if the network was doing conversion between endian settings.

Apparently someone else had puzzled over this for a few weeks before giving up.

I however, channelled the power of the number 252117761.

What's so special about this number?... Well, it's a 32bit number, so we have four 8 bit bytes in there, and it's binary pattern is:

00001111000001110000001100000001

If you can't see the use of this pattern already in checking your networking and endieness, then you might have a problem.

It helps you check out the received values back, if you know you assign a value on one side as this integer, and we call for it across the link, we can see whether we get the above, or a change of order:

00000001000000110000011100001111

For example was the return we had, which is a change or ordering.

But you can also see whether you get:

00000011000000010000111100000111

Which is a short reordering.

Why 32 bit?... Well in this case, because we did know that the word size of the remote system was 32bits, but you can use this trick with any number of bits, just total up your values...

111111101111111001111110001111100001111000001110000001100000001

Might be a good 64bit pattern, and has the value 9187131167487755009.

Personal Update: Network In Action

Today, well tonight, has been the first time I've put any stress onto my new network infrastructure.  Going form a wireless N solution to a true wired 1Gbps solution has been like night and day, and of course, unlike the office network this is 1Gbps where I know exactly what is going on.

So, I've had my old Pentium 4 based server, done a stream of 30mbps from it, whilst pulling 50mbps down from EA online (the Star Wars battlefront Beta), and I've also been streaming HD video from YouTube.

It has been an extremely lush experience.

Playing Payday 2 I've also been enjoying that, with the added stability of the network,

Unfortunately, my workstation needs a lot of work, and I think this weekend it's going to have to be a down time, I need to clean the PC, I've got a fan squeaking, and I've also got one of the mechanical HDD's making a real noise.

I am thinking of sneaking an order for a £30 120GB SSD from Amazon into the mix, and the same next month, to sort something better out for my Stream storage drive, unfortunately, I have two sets of car tax to buy, so this maybe nothing more than a pipe dream.

Personal Update: Networking the House

About six months ago, I told you all I'd bought some really nice shielded cat 5 cable, and I planned to install this between the router in the living room and the office, well today I've finally taken that leap, drilling through victorian ceiling and soot, missing twentieth century power lines and an awkwardly placed floor joist....

My ping which was round 83ms TTL is now 7ms, I have my full download speed from my supplier, and indeed I am.  But it was a bloody messy process.

I have sacrificed a few ethernet plugs to the gods getting the wire down and through, the first wire I pushed through just disappeared, I think along the back of the victorian cornice in the living room; one day I may find it.

My eventual trick was to use a 20mm polymer pipe to encase the wire, and also to guide it down the wall in the lower room, this resulted in the wire being out of sight (least colourwise) against the light coloured walls.

Network Hardware Time

Yes, we've been back in the old house for a while now, and since I stripped all the networking gear out of the man lab in the other house and returned I've not touched it.  It's all been sat on my desk doing not a lot for over six months.

This weekend however, it's time to run a new Ethernet cable from the virgin cable feed, up through the first floor into the front bedroom, this cable is a bit of a beat, I've bought a double shielded cat5e, so I should finally be getting an unfettered full bore connection from my main rig.

The big tasks are going to be drilling the hole, from above this is simply through the floor board, and I've got a wire/pipe detector to make sure it is safe.  But from below the hole in the ceiling has a pelmet and the room is immaculately plastered... I kind of wish Ken had of listened to me when I asked for wiring conduits to be added, but hey ho.  So, hopefully I'll get through the pelmet and border and through the plaster board and into the lath & plaster Victorian ceiling without issue... Once I've gone up like that I'm going to use a plastic pipe to feed it through the hole and cut it off, giving a clean smooth drop for the cable.

Finally, I'm going to feed this shielded cable along the skirting in the room to the desk, and then spend Hours... and I mean HOURS.... sorting the cables.  My cabling solution is just all hanging at the moment, so I might look for some cable tied, tie a bunch into position and then hot-glue those bundles onto the underside of the desk.

Wish me luck...

In other news, you can view my last play sessions on Twitch: http://www.twitch.tv/xelous/manager/past_broadcasts

The Hack

I've been off work, obviously with my feet, however, I've also been off because I've moved house, which took just over a week.  There are a couple of bits and bobs left at the old house and the sale is progressing, whether it all gets done for Christmas and the wife and I can actually celebrate is anyones guess.

Those events are however not what I want to talk about today, no todays topic is one of the systems under my control being the subject of a hack, a successful hack too.

I've been in this game a long time, I've administered Unix, VAX VMS, Sun Solaris, Windows and Linux servers, I've seen access attempts, heard social engineers (cold callers) and even found evidence of folks trying to pick locks to server and switching equipment cabinets.

Until this hack however I'd never lost control of a machine, never had any damage done, and never expected such a strange attack.

The target machine for these problems was a Windows 7 32 Bit virtual machine, being used as a development box with an externally open port to a fixed IP on the local LAN, which the IT Admin had wired upto the company external IP Address and a third party company routed a tunnel through our firewall.

Through this single IP/Port combination the Windows machine was compromised...

Oddly, and quite annoyingly, the IT Admin immediately pointed his finger at one of the pieces of software I was running "Apache"... Could you imagine the uproar in the world at large if Apache web server had a vulnerability which allowed the kind of hack I'll detail later?... Let me just say, I ribbed him back about it, but his inert and instant finger pointing away from what I believe the true problem child annoyed me.

The other software being run on the machine was Visual Studio 2013, whether that is vulnerable, I don't know, perhaps however one would expect some evidence that online/connected services being used by Visual Studio were accessed, some log or trace, and there aren't any, the only traces left were in the Windows logs (event viewer).  Reinforcing my annoyance at the finger being pointed away from Windows.

The user software being operated did not contain any code to do the actions carried out on the machine, indeed only Windows contains the ability to do what was done, so this was the final nail in the coffin for me Windows was to blame.

What was the hack?  You know what, I have no idea, having gone through the evidence and checked I have no idea what allowed the activity, however here's my general pattern pulled from the event viewer.

Thursday 13:30  Machine left in Idle.

Friday 03:01  External Access Logon Requested

Friday 03:03  Logon negotiated with null GUID

(This logon was never listed as successful, only that it was negotiated, I therefore suspect an IP Spoofing attack or some kind).

Friday 03:04  Logon elevated to grant all tokens

Whoever this was, they spent about three hours poking about insude the machine, then logged off.

Saturday 03:01  Logon negotiated with null GUID

Saturday 03:01  Logon elevated to grant all tokens

Saturday 03:03  All user accounts set locked

Saturday 03:13  Volume Shadow Copy started

Saturday 03:23  Volume shadow Copy crashed

Saturday 03:24  Unknown Application reported 100% CPU

Saturday 03:38  New user account created with name "hahaha"

Several more shadow volume copies occur, then this person logs off.

Sunday 04:13   hahaha logs on

Sunday 04:15 All user accounts set to disables, except hahaha

Sunday 06:38 hahaha logs off

Sunday 06:39   Machine shutdown by hahaha

That's what they did, not a lot I admit, however, we have the times this happened, we have the Event View log as proof, and we pay a company to look after the company firewall... So they have access logs right?... Right?... RIGHT?...

Nope, apparently it is too much hassle to check the logs, and according to one source "we get port scanned 100 times a day, what's the point of chasing down one hack?"... erm because they got in... they didn't just scan they got in, and for the life of me I can't figure out how.

Windows was patched, firewall was on, only a none-standard port was open and it was only listening with a custom application with a specific and narrow role to carry out...

Everything else on the machine was not available to that port, or on the second network adapter in the machine!

Network Guru Fails at Monitoring his own Bandwidth

Do I come on here to bemoan the seeming incompetence of others a lot?... Because I have  just had an annoying 20 minutes talking to one of our "Network Experts", there's a server box running Ubuntu, its actually a virtual machine, and it was being blamed for crippling the network on the host machine - which has to run several other Windows Server 2008 Virtual Machines.

For some reason, this came across my desk, because someone has decided that though I can't advise the company on Linux server usage or adoption, I can be called into sort out the problems other fools create.

So, this server was running, the administrator chap is sat there with PuTTY connected to it and he's saying there's too much network traffic coming from the box.

Looking at the process list the machine is doing nothing, when I ask for one of its users to be made to use it, the CPU usage is a time flitter and then the usage is over, it appears to me the Virtual machine host is waking the VM, using it, and closing it.

The processes being used are simple webservices, so there should be no network traffic unless someone is using the server.

So, I ask.. "What's your idea that this machine is causing the issue?"  Very smugly the admin presented me with an A4 sheet with two graphs, one shows all his servers running with the Ubuntu machine present, and there's a whole load of traffic going on... The next is the same graph, with the Ubuntu box removed, and there's nearly no network traffic.

Then within PuTTY connected to the box he opens iptraf...

And he points to the statistics and declares, "See, this is running multiple kilobytes a second"...

Lets recap... "This machine is crippling our network usage on this box".... "using up kilobytes a second"... If any of you don't grasp how absurd this is please stop reading now, because the guy was clearly serious.  He's not that old, he's younger than me, surely he realises kilobytes a second on a modern gigbit backbone is nothing?  No, he seriously wants the ubuntu box silent.

So, what's causing this tiny trickle of bandwidth... Of course, he's connected to this machine with PuTTY...

"Close this window and take me to the machine in the server room"

In we go, and from the console plugged into the machine I run iptraf... look what we see now...

Yes, its zero...

When he is running his measures and checks he's connected over the network to the machine, he's been measuring his own bandwidth administrating the machine, and apparently the kilobytes per second that took were too much for him, I feel like this guy should be thrown from the building, not be earning far more than me in the "superior" position of "network guru god" which hs holds...

So knowing the machine is idle, I wonder what causes the bandwidth, so with the iptraf still running, I call up the user and ask him to get working again... and sure enough there's little trickles of bandwidth from the Ubuntu box, no spikes, no major blocks of transmission.  Hence after a few minutes I conclude the Ubuntu box though active, is only taking a tiny amount of the total bandwidth available to the machine, and so I start to look at the other machines...

And with just perfmon on the two Windows Server 2008 boxes with the user operating his end, I can see that the Windows boxes are spiking their network traffic, just with task manager you can see one box taking 20% the total available to it, and the other over 45%... This is a gigbit connection, and 20% of it is taken for maybe 30-50 seconds each minute and then goes quiet, then the 45% hog is there for maybe every 10 seconds of each minute.

They're scheduled tasks, as the users input items into their programs, the programs process and send out instructions, e-mails and update other databses.  The trickle of data I/O to the ubuntu machine is just such a feed, the Ubuntu box sucks/queries data from MS SQL Server on one of the machines and squirts it into MySQL where a very old (like 8 year old) unmaintained program written in C with MySQLConnector picks it up and uses sendmail to e-mail a load of people.  Such notifications take maybe 0.25 seconds and run every 5 minutes, the amount of data is small, and looking at the physical name of the Ubuntu box "e-mailer relay" I think it's safe to say its not using a lot of bandwidth and the other machines are to blame.

How I educate the "network expert"... This is my next task.