Planet Code

by James S. A. Corey

ISBN-13: 978-0316333429

 

I was apprehensive about reading Leviathan Wakes as a friend had suggested it was boring compared to the TV series, which I loved. It wasn’t! The Protomolecule is brown goo, rather than bright glittery stuff, but that really didn’t matter. Chrisjen Avasarala, one of my favourite characters from the TV series, and the earth government don’t feature at all. It’ll be interesting to see if she appears later in the series. Some of the other bits invented for TV I didn’t feel were necessary. The main characters were mostly the same and I felt like I already knew them.

I struggle to think of Leviathan Wakes as a space opera. There are only really two threads and the scope isn’t particularly broad. However, there is loads of potential for the future books and I’m really looking forward to them.

In the last few weeks, another friend of mine, John Cricket, has been setting weekly code challenges via linkedin and his new website, https://codingchallenges.fyi/. They were all quite interesting, but one in particular on writing load balancers appealed, so I thought I’d kill two birds with one stone and write up a worked example.

You’ll find my worked example below. The challenge itself is italics and voice is that of John Crickets.

The Coding Challenge

https://codingchallenges.fyi/challenges/challenge-load-balancer/

Write You Own Load Balancer

This challenge is to build your own application layer load balancer.

A load balancer sits in front of a group of servers and routes client requests across all of the servers that are capable of fulfilling those requests. The intention is to minimise response time and maximise utilisation whilst ensuring that no server is overloaded. If a server goes offline the load balance redirects the traffic to the remaining servers and when a new server is added it automatically starts sending requests to it.

Load balancers can work at different levels of the OSI seven-layer network model for example most cloud providers offer application load balancers (layer seven) and network load balancers (layer four). We’re going to focus on a layer seven - application load balancer, which will route HTTP requests from clients to a pool of HTTP servers.

A load balancer performs the following functions:

• Distributes client requests/network load efficiently across multiple servers
• Ensures high availability and reliability by sending requests only to servers that are online
• Provides the flexibility to add or subtract servers as demand dictates

Therefore our goals for this project are to:

1. Build a load balancer that can send traffic to two or more servers.
2. Health check the servers.
3. Handle a server going offline (failing a health check).
4. Handle a server coming back online (passing a health check).

Step Zero

As usual this is where you select the programming language you’re going to use for this challenge, set up your IDE and grab a coffee (or other beverage of your choice).

If you’re not used to building systems that handle multiple concurrent events you might like to grab your beverage of choice and do some background reading on multi-threading, concurrency and asynchronous programming as it relates to your programming language of choice.

This is an easy choice for me as most of the work I’m doing at the moment is in node and it supports the sort of concurrency I need. To make getting started easy, I created a node project template which supports typescript, because all good developers love type safety, right?

https://github.com/pjgrenyer/node-typescript-template

Step 1

In this step your goal is to create a basic server that can start-up, listen for incoming connections and then forward them to a single server.

The first sub-step then is to create a program (I’ll call it ‘lb’) that will start up and listen for connections on on a specified port (i.e. 80 for HTTP). I’d suggest you then log a message to standard out confirming an incoming connection, something like this:

./lb
Received request from 127.0.0.1
GET / HTTP/1.1
Host: localhost
User-Agent: curl/7.85.0
Accept: */*

For when I sent a test request to our load balancer like this:

curl http://localhost/

Cool! Time to write some code. Fortunately it’s really easy to create a program which can listen for connections in node, with https://expressjs.com/ and really easy to get going, by making a copy of the node typescript template project I created:

https://github.com/pjgrenyer/coding-challenges-lb

And creating a new feature branch:

git flow feature start listener

Then all I needed was to install express:

npm install express --save
npm install @types/express --save-dev

and fire up an express server:

// index.ts

import express, { Request, Response } from "express";

const port = 8080;
const app = express();

app.get('/', (req: Request, res: Response) => {
  res.send('Code Challenge!')
})

app.listen(port, () => {
  console.log(`Listening on port ${port}`)
});

run the app:

npm run dev

> coding-challenges-lb@1.0.1 dev
> npm run build && node dist/index.js


> coding-challenges-lb@1.0.1 build
> rimraf dist && tsc

Listening on port 8080

and check it works with curl:

curl localhost:8080

Code Challenge!

The code challenge suggests using port 80 for the load balancer, but I’m developing on Linux where this port is reserved, so I’ve gone for port 8080 instead.

The code challenge suggests logging the request, which is easily done with node and express, by modifying the request handler:

app.get('/', (req: Request, res: Response) => {
    console.log(req);
    res.send('Code Challenge!');
});

I’ll not show the output here as, from express, it’s very large.

That’s it for this part of the step, so I committed the code:

git add .
git commit -m"feat: added listener"

And completed the feature branch:

git flow feature finish

before going back to the code challenge instructions.

Next up we want to forward the request made to the load balancer to a back end server. This involves opening a connection to the back end server, making the same request to it that we received, then passing the result back to the client.
In order to handle multiple clients making requests you’ll need to add some concurrency, either with your programming language’s async framework or threads.

I decided to use a modified version of my code from the beginning of this step as my backend. It was changed to respond with a HTTP status of 200 and the text: ‘Hello From Backend Server’.

I decided to do much the same thing. Node can handle multiple clients making calls out of the box. I created another new project:

https://github.com/pjgrenyer/coding-challenges-be

which is much the same as the load balancer, but with a few changes. I know from reading ahead in the code challenge I’m going to need to run the backend server on multiple ports, so I made that an environment variable and returned it as part of the response message:

// index.ts

import express, { Request, Response } from 'express';

const port = process.env.PORT ? +process.env.PORT : 8081;
const app = express();

app.get('/', (req: Request, res: Response) => {
    res.send(`Hello From Backend Server (${port})`);
});

app.listen(port, () => {
    console.log(`Listening on port ${port}`);
});

Now, if I run the app as normal it will run on port 8081 by default, but if I set the port environment variable:

PORT=8082 npm run dev

then the app will run on the specified port:

> coding-challenges-be@1.0.1 dev
> npm run build && node dist/index.js

> coding-challenges-be@1.0.1 build
> rimraf dist && tsc

Listening on port 8082

and the port number is returned as part of the response:

curl localhost:8082

Hello From Backend Server (8082)

Next, I need to modify the load balancer to call the backend server.

Node doesn’t support fetch natively like a browser’s implementation of JavaScript does and even though there is a node-fetch package, I prefer Axios for no other reason than its implementation makes a little more sense to me:

npm i axios --save
npm i @types/axios --save

With Axios installed I can call the backend directly and return the response:

import express, { Request, Response } from 'express';
import axios from 'axios'

const port = 8080;
const app = express();

app.get('/', async (req: Request, res: Response) => {
    const response =
await axios.get('http://localhost:8081');
    res.send(response.data);
});

app.listen(port, () => {
    // eslint-disable-next-line no-console
    console.log(`Listening on port ${port}`);
});

Now with the backend running on port 8081 and the load balancer on 8080, making a request to the load balancer from curl gives the backend response:

curl localhost:8080

Hello From Backend Server (8081)

And now it’s on to step 2.

Step 2

In this step your goal is to distribute the incoming requests between two servers using a basic scheduling algorithm - round robin.

Round robin is the simplest form of static load balancing. In a nutshell it works by sending each new request to the next server in the list of servers. When we’ve sent a request to every server, we start back at the beginning of the list. 

You can read more about load balancing algorithms on the Coding Challenges website.

So to do this we’ll need to do several things:

1. Extend our load balancer to allow for multiple backend severs.
2. Then route the request to the servers based on the round robin scheduling algorithm.

The code challenge goes on to suggest that a Python server could be used as the backend, but I’ve built my own which can be easily started on different ports, so I’m going to stick with that.

Although I came up with quite a few different ways of storing and iterating through backend server urls, including using a database, the simplest idea I had is to maintain an array of the servers and, each time, take the server off the top, use it and push it back on from the bottom. Node modules allow arrays and functions to be easily shared between requests, so I created a new module with the array, initialised it and exported a function to get the next server:

// backend.ts

let backends: Array<string> = [];
[
    'http://localhost:8081',
    'http://localhost:8082',
    'http://localhost:8083'
].forEach((url) => backends.push(url));

export const nextBackend = (): string | undefined => {
    const nextBackend = backends.shift();
    if (nextBackend) {
        backends.push(nextBackend);
    }
    return nextBackend;
};

Then I modified the request to get the next backend server and use its url each time a request is made:

import { nextBackend } from './backend';
…
app.get('/', async (req: Request, res: Response) => {
    const backend = nextBackend();
    if (backend) {
        const response = await axios.get(backend);
        res.send(response.data);
    } else {
        res.status(503).send('Error!');
    }
});

When you ‘shift’ (remove from the top) an element out of an array, it will return undefined if the array is empty, so undefined is a possible value returned by nextBackend and must therefore be handled, in this case, by returning status code 503 and a simple error message.

This should be all which is needed, so I fired up three backend servers on three different ports:

PORT=8081 npm run dev
PORT=8082 npm run dev
PORT=8083 npm run dev

started the modified load balancer and called it a few times:

> curl localhost:8080

Hello From Backend Server (8081)

> curl localhost:8080

Hello From Backend Server (8082)

> curl localhost:8080

Hello From Backend Server (8083)

> curl localhost:8080

Hello From Backend Server (8081)
…

And it worked! I got a response from each of the backed servers in turn! It works from a web browser too.

Step 3

In this step your goal is to periodically health check the application servers that we’re forwarding traffic to. If any server fails the health check then we will stop sending requests to it.

For this exercise we’re going to use a HTTP GET request as the health check. If the status code returned is 200, the server is healthy. Any other response and the server is unhealthy and requests should no longer be sent to it.

Typically the health checks are sent periodically, I’d suggest you make this configurable via the command line so we can set a short period for testing - say 10 seconds. You will also need to be able to specify a health check URL.

So in summary the key tasks for this step:

Allow a health check period to be specified on the command line.
Every period make a GET request to the health check URL if the result is 200 carry on. Otherwise take the server out of the list of available servers to handle requests.
If the health check of a server starts passing again, add it back to the list of servers available to handle requests.

It would be a good idea to run the health check as a background task, concurrently to handling client requests.

In my experience a health endpoint is usually a slightly oddly named endpoint with ‘health’ in the name. One I see frequently is ‘_health’. A health endpoint can be any endpoint which returns 200 and should have zero or minimal side effects. For example, it shouldn’t be calling a database, but it may log. The backend service already has such an endpoint, but that might change in the future, so I’m going to create a dedicated one.

…
app.get('/_health', (req: Request, res: Response) => {
    res.send();
});
…

Then, back in the load balancer I need to call the health check endpoint on each backend every X seconds where X is configurable. Creating a background tasks in the node is really easy using setInterval:

const healthCheckInterval =
process.env.HEALTH_CHECK_INTERVAL ? +process.env.HEALTH_CHECK_INTERVAL : 10;

…
const timer = setInterval(() => {
    console.log('Health check!');
}, 1000 * healthCheckInterval);
timer.unref();
…

Starting the load balancer will print the console message every 10 seconds. Next I need to get a list of the backends and call the health check endpoint on each. First I’m going to create a health check function and put it in a module of its own to keep things clean:

// healthChecks.ts
…
export const healthChecks = async () => {
    console.log('Health check!');
}

And then call it on startup, so the first thing which is done is the health check on each backend, and then from the interval function:

// index.ts
…
healthChecks();
const timer = setInterval(async () => {
    await healthChecks();
}, 1000 * healthCheckInterval);
timer.unref();

Next I need a list of backends. The current backends array changes. Backends are pulled off the top of the array and pushed back on the bottom. In future they’ll also be removed from the array when the backend isn’t healthy, so I need a constant array of backends. I made a few changes for this:

// backend.ts

export const backends = [
'http://localhost:8081',
'http://localhost:8082',
'http://localhost:8083'];

let activeBackends: Array<string> = [];
backends.forEach((url) => activeBackends.push(url));

export const nextBackend = (): string | undefined => {
    const nextBackend = activeBackends.shift();
    if (nextBackend) {backends
            activeBackends.push(nextBackend);
    }
    return nextBackend;
};

Now the backend array is constant and a new activeBackends array is used for round robin load balancing. I also exported the  array so that I can use it elsewhere.

Now I want to iterate through the backends:

// healthChecks.ts

export const healthChecks = async () => {
    for (const backend of backends) {
        }
};

Next I want a function I can call to determine if the backend is healthy:

// healthChecks.ts

const healthCheckPath =
process.env.HEALTH_CHECK_PATH ?? '_health';
…
const isHealthy = async (url: string): Promise<boolean> => {
    try {
        const response =
axios.get(`${url}/${healthCheckPath}`);
        return (await response).status === 200;
    } catch (error: any) {
            return false;
    }
};

The path to the health check endpoint should be configurable, so I’ve made it an environment variable. The response status from the endpoint is checked and if it isn’t 200 then the health check fails. If an exception is thrown, which can happen with Axios if the backend isn’t there, it’s caught and considered a health check failure.

Now I can iterate through the backends and, check each one and output a message about its health:

export const healthChecks = async () => {
    for (const backend of backends) {
        if (await isHealthy(backend)) {
            console.log(`${backend} is healthy`);
        } else {
            console.log(`${backend} is not healthy`);
        }
    }
};

This is the point where, if you’re like me, you fire up three instances of the backend (if you haven’t still got them running), fire up the load load balancer and spend hours (well, several minutes) starting and stopping the backends and watching the health check messages:

Listening on port 8080
http://localhost:8081 is healthy
http://localhost:8082 is not healthy
http://localhost:8083 is healthy
http://localhost:8081 is healthy
http://localhost:8082 is not healthy
http://localhost:8083 is healthy
http://localhost:8081 is healthy
http://localhost:8082 is healthy
http://localhost:8083 is healthy
…

And while that’s a lot of fun, it’s not getting us anywhere as we’re not actually removing dead backends from the list or re-adding them when they’re revived. I added two new functions to do that:

// backend.ts

export const removeBackend = (url: string) => {
    activeBackends = activeBackends
.filter((backend) => backend != url);
};

export const addBackend = (url: string) => {
    if (!activeBackends.find((backend) => backend == url)) {
            activeBackends.push(url);
    }
};

The removeBackend function simply iterates through the activeBackends array and filters out the unhealthy backend. addBackend looks to see if the backend already exists and only adds it to activeBackends if it doesn’t so I don’t end up with duplicates.

Then, all that remains to do in this step is to use these methods in the the healthChecks function:

// healthChecks.ts

export const healthChecks = async () => {
    for (const backend of backends) {
        if (await isHealthy(backend)) {
            console.log(`${backend} is healthy`);
            addBackend(backend);
        } else {
            console.log(`${backend} is not healthy`);
            removeBackend(backend);
        }
    }
};

Then restart the load balancer and test…

When it comes to testing this I suggest you start up a third backend server.

Then connect to your load balancer three to six times to verify that it is rotating through backend servers as expected. Once you’ve verified that kill one of the servers and verify the request are only routed to the remaining two, without you, the end user receiving any errors.

Once you’ve verified that, start the server back up, wait just a little longer than the health check duration and then check it is now back to serving content when requests are made through the load balancer.

As a final test, check your load balancer can handle multiple concurrent requests, I suggest using curl for this. First create a file containing the urls to check - for this they’ll all be the same:

url = "http://localhost:8080"
url = "http://localhost:8080"
url = "http://localhost:8080"
url = "http://localhost:8080"
url = "http://localhost:8080"
url = "http://localhost:8080"
url = "http://localhost:8080"
url = "http://localhost:8080"

Then invoke curl to make concurrent requests:

curl --parallel --parallel-immediate --parallel-max 3 --config urls.txt

Tweak the maximum parallelisation to see how well your server copes!

If that all works, congratulations, you’ve built a basic HTTP load balancer!

I tried all the tests against the load balancer and they all passed. I put more than 90 calls in the urls file and had 30 parallel calls before I got bored. It all worked really well.

Finally

Having built a rocking load balancer and got the recommended tests to pass, I’m going to leave this worked example here. However, the code challenge does suggests some further steps:

Beyond Step 3 - Further Extensions You Could Build

Having gotten this far you’ve built a basic working load balancer. That’s pretty awesome!

Here are some other areas you could explore if you wish to take the project further and dig deeper into what makes a load balancer useful and how it works:

1. Read up about HTTP keep-alive and how it is used to reuse back end connections until the timeout expires.
2. Add some Logging - think about the kinds of things that would be useful for a developer, i.e. which server did a client’s request go to, how long did the backend server take to process the request and so on.
3. Build some automated tests that stand up the backend servers, a load balancer and a few clients. Check the load balancer can handle multiple clients at the same time.
4. If you opted for threads, try converting it to use an async framework - or vice versa.

These seem like a lot of fun, especially writing the automated tests. I may well return to and progress this code challenge in the future.

Thank you to Michael Davey, John Cricket and Stephen Cresswell.

Sleepover
Alistair Reynolds
ASIN: ‎ B0097AXWUY

I often think that Alastair Reynolds must have encountered some truly unpleasant people in his life, as he creates such nasty characters so well. Sleepover starts with Gaunt awakened from a long sleep, but soon the backstory of why those who awoke him are so unpleasant unfolds. Gaunt awakes to a dystopian future quite unlike any other I’ve read about or imagined and this is where the story really starts to get interesting.

It was clear from the outset that this story was just the beginnings or a larger work, which was pieced together from notes, and may or may not be developed into a longer novel. I hope it does.

The Silmarillion
by J. R. R. Tolkien, Christopher Tolkien
ISBN: 978-0007523221

I read the Lord of the Rings when I was about 8, having loved the BBC radio play starring Michael Hordern as Gandalf. Of course I wanted to read the Silmarillion too, but was always told it was hard going, which put me off. Then I tried it in my early 20s and didn’t get past the first few pages. Today, at (nearly) 46 I finished it.

The Silmarillion is hard going and, for the most part, unpleasant to read. It’s mostly the language used and that it reads more like a technical history than a story. It’s quite repetitive with battle after battle and no real progress for good or evil.  There are so many different names and places and this makes it difficult to follow.

It gets better around Beren and Lúthien.  Where it really gets interesting, and more enjoyable, is when it reaches the Third Age and there’s more about the rings of power and the characters I’m more familiar with from the Lord of the Rings.

I was disappointed that the scenes from the Hobbit film series with Radagast weren’t described in the Silmarillion, as I’d been led to believe they were. Someone must have made them up.

The recent Amazon TV series, the Rings of Power, doesn’t seem to be consistent with the Silmarillion either, for example, Gandalf didn’t drop from a ball in the sky and there’s no mention of Sauron rescuing Galadriel and sailing to Numenor.

The New One Minute Manager
by Kenneth Blanchard and Spencer Johnson 

I rarely read a book more than once (unless it’s set in Alastair Reynolds’ Revelation Space Universe), but this is the third time I’ve read the New One Minute Manager, and It’s not just because it’s a quick and easy book to read, with clear, concise digestible advice.

Many years ago, when I ran my own business, I was working with the conflicting practice of asking my employees to assume that everything was ok unless I said otherwise, and the desire for them to be happy in their work. This meant that most of the time feedback was sparse and, when it did come, it was predominantly negative - although I didn’t operate a blame culture.

The New One Minute manager has quite a different approach. Now that I’m leading a team again, I read it through to remind myself of the approach. I’m currently experimenting with the One Minute Goals in a software engineering context. I use something similar to the One Minute Praisings already and the book has reminded me to structure how I give feedback, generally and for the individual, by applying the One Minute Redirect.

And of course, I’ve also written out the details of the secrets and stuck them to my wall.

Visual Lint 8.0.12.359 has now been released. This is a maintenance update for Visual Lint 8.0, and includes the following changes:

• Added support for $([MSBuild]::GetDirectoryNameOfFileAbove()), $([MSBuild]::GetPathOfFileAbove()), $([System.IO.Path]::Combine()) and $(MSBuildToolsVersion) to the Visual Studio 2010-2022 project (.vcxproj) file parser. Together, these allow Visual Lint to correctly load Directory.Build.props property files located in the solution file folder.

• Updated the PC-lint Plus compiler indirect files co-rb-vs2017-shared.h, co-rb-vs2019-shared.h and co-rb-vs2022-shared.h to check if _MSVC_LANG is defined before checking its value.

• Added support for PC-lint Plus 2.0:

  • Visual Lint is now compatible with PC-lint Plus installations containing pclp64.exe but not pclp32.exe. This change is necessary to support PC-lint Plus 2.0, which does not include pclp32.exe.

  • Dedicated support files for PC-lint Plus 2.0 are now installed into the C:\Program Files (x86)\Riverblade\Visual Lint\Tools\PC-lint Plus\2.x folder.

  • Updated the PC-lint Plus Visual Studio 2022 compiler indirect file co-rb-vs2022.lnt to enable C++ 20 support.

  • Added a PC-lint Plus 2.x message database.

  • Updated the user interface and online help to reflect the fact that PC-lint Plus is now a product of Vector Informatik GmbH, following their acquisition of the assets of Gimpel Software LLC.

Finally, if you are a C++ developer who has not yet tried PC-lint Plus, it is worth noting that Vector Informatik GmbH have a policy of transparent pricing, and that team licences for PC-lint Plus start from just €336 per developer per year, with no minimum team size.

Download Visual Lint 8.0.12.359

Diamond Dogs, Turquoise Days
by Alistair Reynolds
ISBN: ‎ 978-0575083134

Diamond Dogs

I don’t get full satisfaction from stories that leave unanswered questions, unless those questions get answered in future stories. I don’t like that I don’t know why the Spire can levitate. I don’t like that I don’t know where the Spire came from, who built it or what it was for. I don’t like it that I don’t know if Richard and/or Childe completed all the puzzles and reached the top.  I don’t like that I don’t know what was at the top and I don’t like the implication that it might be the weapon used to kill Pattern Jugglers, because that asks even more unanswered questions.

I loved the story so much more on second reading and I think that’s because I was so much more familiar with the Revelation Space universe, specifically the eighty, and the other stories within it this time. No longer do I feel it was for people who really enjoy maths and I enjoyed the characters and their motivations immensely.

Turquoise Days

Turquoise Days is a strange story to pair up with Diamond Dogs. The only real connections between them are the Revelation Space universe, the suggestion that the weapon used against the Pattern Jugglers might have come from the Spire and, of course, that Celestine was modified by the Pattern Jugglers.

The first part of the story takes some getting through with the detailed description of swimming with the Pattern Jugglers. The rest of the story moves quite quickly and could probably have been a much longer story with more about what the fanatical leader of the scientists had done. Ultimately a very enjoyable read. 

Visual Lint 8.0.11.357 has now been released.

This is a maintenance update for Visual Lint 8.0, and includes the following changes:

• Fixed an MSBuild parsing issue which affected the definitions of the vcpkg project variables $(_ZVcpkgRoot), $(_ZVcpkgManifestRoot) and $(_ZVcpkgInstalledDir).

• Added support for Codegear C++ Builder 11.2 project files.

• Updated the PC-lint Plus compiler indirect files co-rb-vs2022/19/17.lnt to prevent a warning (553 - "undefined preprocessor variable, assumed 0") being issued for the Visual C++ built-in compiler symbol _MSVC_LANG in some circumstances.

• Updated the PC-lint Plus compiler indirect file co-rb-vs2022.lnt to support Visual Studio 2022 v17.4.3.

• Updated the PC-lint Plus compiler indirect file co-rb-vs2019.lnt to support Visual Studio 2019 v16.11.22.

• Updated the PC-lint Plus compiler indirect file co-rb-vs2017.lnt to support Visual Studio 2017 v15.9.51.

Download Visual Lint 8.0.11.357

The Great Dune Trilogy
Frank Herbert
ISBN-13: 0575070706

I remember distinctly reading Dune in 1992 after seeing the 1980s film. In fact I can still picture myself lying on a bed in a holiday cottage in a small French village near Carcassonne reading the book. I went on to read Dune Messiah, but couldn’t get into Children of Dune. I tried it again several years later, but still couldn’t get into it. Dune has been on my list to reread for a while. When searching for Dune in the Amazon Kindle store the trilogy came up as one book, so I decided to read all three straight through and I’m glad I did!

There’s no getting away from the fact that Dune is a great story. I discovered recently that it’s two stories glued together and it shows. The first half of the book has lots of details and then there appears to be a large gap in the story, which at least one of the films attempted to fill, and then you get the end of the story. I don’t really like the way Frank Herbert explains what’s going to happen at the beginning and then that’s what happens, with a small twist at the end. I’d rather be kept in suspense.

Dune Messiah is a great little story, but it mostly leaves science fiction behind in favour of feudalism and politics. This is also where I think Herbert starts getting indulgent. Most of us love well developed characters, but I found myself in their heads far too much. The twelve year gap in the story between Dune and Dune Messiah where a whole Jihad rages across the galaxy isn’t done justice.

I enjoyed, mostly, Children of Dune too, but it was very much more of the same. I didn’t see the end coming, which was a good surprise, but I don’t think I liked it.

It will be interesting to see how God Emperor of Dune shapes up, especially as it’s set several thousand years in the future.

So You Are Uncomfortable with Duplicate Data? 

If, like me, you’ve spent a reasonable amount of your career working with relational databases,  where data is rationalised to avoid duplication, the idea of duplicating data across microservices is probably anathema to you.

Even if you’ve worked with a noSql database like MongoDB, where data is often duplicated across the documents, you probably still struggle with the idea of a service keeping a copy of data owned by another service.

Discomfort with duplication doesn’t need to come from databases. The Don't Repeat Yourself (DRY) principle of software engineering states that "Every piece of knowledge must have a single, unambiguous, authoritative representation within a system". Even the process of Test Driven Development (TDD) includes a step for refactoring to remove duplication as part of the cycle.

As software developers we are programmed to detest duplication in all its forms.

It’s ok, I have felt your pain and as soon as you come to terms with the idea of duplicating data across microservices being the best way to make your microservices more robust, independent and decoupled from other services your pain will go - forever.

Making Microservices more Robust and More Independent

A Microservices architecture consists of a collection of different services which each provide a well defined, loosely coupled and independent business function. You can find out more at: https://microservices.io/. Let’s have a look at an example from part of a previous project of mine, an app for finding local cafes that serve great tea, called Find My Tea.

Find My Tea had a handful of microservices, but two of the most important ones were the Location service, which managed the locations (cafes, restaurants, etc.) where great tea could be found and the Brand service which managed the brands of tea served by the locations. Managing locations and brands consists of creating, reading, updating and deleting (CRUD) them. The Location service was the Single Source of Truth for Locations and the Brand service for brands.

In the first iteration, when the app requested a location, the Location service looked-up the location in its database, found the IDs of the brands served by location and then requested the details of those brands from the Brand service. The Brand service then looked up the brands in its database and passed them back to the Location service. The Location service then enriched the location with the brand details and passed it back to the app.

It is arguable that the more independent a microservice is, the more robust it becomes. Here we have a very clear dependency on the Brand service by the Location service when performing a location lookup request. If the Brand service is down or otherwise unavailable, either the brands are not returned with the location or, depending on how the location service is designed to handle partial failure, the entire request fails. There is also the potential latency of inter-service communication and two database lookups, although in a simple request such as this, it is likely to be negligible.

To remove this dependency the Location service can keep a copy of the data maintained by the Brand service up-to-date in its own database, so that when a location is requested, a call to the Brand service is not necessary.  This does not make the Brand service redundant as it is still required to maintain the brand data and remains the Single Source of Truth for brands. The copy of the brand data kept in the Location service, although updatable from the Brand service, is effectively a read-only copy. The advantage is that if the Brand service is unavailable the location request will still succeed and the brand data will be present in the response.

Distribute the Data

However; this does beg the question of how to keep the brand data up-to-date in the Location service. Could this be a source of coupling? I’ve seen this done in two ways, but there are other approaches too.

One way is to have the Location service poll the Brand service every so often to get brand data and update it in its own database. There are a number of drawbacks with this approach. We all know polling is evil. In the case where you have multiple instances of the polling service, you have to specify one instance as the one that does the polling, or all of the instances could be polling for the data unnecessarily and all at the same time. The data in the Brand service may get updated in-between polls, meaning that the data is out of date, for a period of time, in the Location service - whatever method of data synchronisation you use, there will always be an element of Eventual Consistency. You either need to devise a clever mechanism for determining which brands have been updated since the last poll or always send back all of the data resulting in potentially large requests. The polling approach requires the Location service to know where to find the Brand service, creating an unnecessary dependency. It is also required to handle the error caused by the Brand service being down or unreachable. The polling approach doesn’t tend to scale very well for all of these reasons.

The approach I favour is to use a message broker. When a brand is created, updated or deleted, the Brand service can put a message onto a topic with only the details of that change. The Location service can listen to a queue, which is subscribed to the topic, and only update its database with a single brand when a message is received. There is no polling necessary. Message brokers are usually very fast and the amount of time the Location service would be out of date is likely to be negligible. The Location service only needs to know where the queue is that it is listening to. When there are multiple instances of the Location service, the queue can be configured to only deliver each message to the first instance which requests it. An added advantage is that the Brand service only needs to know where to find the topic. It doesn’t need to know anything about the Location service, or any other services, which may want to consume the messages via a queue subscribed to the topic. Of course both the sending and receiving services are slightly coupled by the format of the message and the data contract, potentially, becomes as important as any API contract.

More Robust, Independent and Loosely Coupled

It can be as simple as that. Maintaining a copy of data in one service, which is maintained by another service, can make the services more robust and independent. Distributing that data via a message broker makes the services loosely coupled, although not entirely decoupled, it keeps the data up-to-date and reduces the size of messages.

As with most things in software development, maintaining a local copy of data which is managed elsewhere is a tradeoff. As a software developer you must consider, for example, the security concerns which come with duplicating data. Especially if it is considered personal data. You must also consider the complexity of keeping the data up-to-date. For example, when does the data expire or become invalid. Does the data need to be versioned? Does the order of applied updates need to be taken into account?

I hope it goes without saying that in almost every other context duplication should still be avoided, detested and possibly even hated. However, it should also be clear that the trade off of duplicating data in microservices can make for better microservies.

Much of my early understanding of microservices, including the advantages of sharing data and some of the possible ways to do it came from Microservices Patterns by Chris Richardson (ISBN-13: 978-1617294549). If you’re interested in learning more about  microservices, I would strongly recommend giving it a read. The rest has come from trial and error, failure, eventual success and quite a lot of arguing with colleagues.

Last summer, in Ossun-ez-Angles, Haute Pyrenees, I came across a House Mark above a barn door.

10 months ago I wrote a blog post Air-Source Heat Pump – our experience so far, 2 months in about our new air source heat pump. Have a look back at that for photos of the device itself and more detail about installation etc.

Less energy

We used a lot less energy this year than last year. Here’s the graph for 2 years:

As you can see, we used a lot less energy in kWh this year than last year. Air source heat pumps work!

More money

Our energy cost more this year than last year. I’ve calculated this graph based on fixed prices, and I used 2021 prices to keep consistency with the last blog post, but the real-prices story is similar. Here is the graph for the last 2 years:

Why did our cost go up when our energy usage went down so dramatically?

Because electricity is too expensive!

Electricity is the right way to power our cars and homes, because it can be sourced sustainably, and in fact much of it really is being sourced sustainably right now.

Artificially-high electricity prices are preventing people switching to a better way.

Solar panels help!

In September this year we had solar panels installed. They look great and are working incredibly well. We were getting 10kWh per day from them in September. We don’t have a battery yet, but when it arrives we think we will be able to cover most of our summer energy costs using these panels.

Since the panels were installed, our reduced use of grid electricity meant that our energy costs for this year dropped below last year. Obviously it’s too soon to say for sure what the full impact is, but I can say we are very happy with our solar panels.

Our house is cozy

Our new heat pump heats our leaky house very well, and we are nice and cozy, even when temperatures outside drop below zero. The heat pump is less efficient when it’s cold outside, but still way better than a gas boiler.

Installed by Your Energy Your Way

[My wife used to be director of the company, so I declare an interest.]

Our heat pump, radiators and panels were installed by Your Energy Your Way and I can recommend them for good communication, service and quality.

In the time honored tradition of using one’s blog as an Internet-enabled notepad, here’s a quick not on how I build GNU Emacs on macOS using homebrew and the emacs-mac port cask: brew upgrade -s railwaycat/emacsmacport/emacs-mac --with-mac-metal --with-imagemagick --with-native-comp --with-modern-icon --with-natural-title-bar This - amongst other features - turns on some experimental macOS-relevant features and most importantly, the optional native compilation of Elisp code.

From The Little Bookroom by Eleanor Farjeon via Eldrbarry

There were five Tims all told, Old Tim, Big Tim, Little Tim, Young Tim and Baby Tim, and they were all born wise. So whenever something was the matter in the village, or anything went wrong, or people for some cause were vexed or sorry for themselves, it became their habit to say "Let's go to the Tims about it, they'll know, they were born wise."

For instance, when Farmer John found out that the gypsies had slept without leave in his barn one night, his first thought was not, as you might suppose, "I'll fetch the constable and have the law on them!" but, "I'll see Old Tim about it, so I will!"

Then off he went to Old Tim, who was eighty years old, and he found him sitting on a gate, smoking his clay pipe.

"Morning, Old Tim," said Farmer John. "Morning, Farmer John," said Old Tim taking his clay pipe from his mouth. "I've had gypsies in my barn again, Old Tim," said Farmer John "Ah, have you now!"said Old Tim. "Ay, that I have," said Farmer John. "Ah, to be sure!," said Old Tim. "You were born wise, Old Tim," said Farmer John. "What would you do if you was me?"

Old Tim put his clay pipe in his mouth again and said, "If I was you, I'd ask Big Tim about it, for he was born wise too and is but sixty years old. So I be twenty years further off wisdom than he be."

Then off he went to Big Tim, who was Old Tim's son, and he found him sitting on a gate, smoking his briar.

"Morning, Big Tim," said Farmer John. "Morning, Farmer John," said Big Tim taking his briar from his mouth. "I've had gypsies in my barn again, Big Tim, and Old Tim sent me to ask what you would do if you was me, for you were born wise" said Farmer John. Big Tim put his briar in his mouth again and said, "If I was you, I'd ask Little Tim about it, for he was born wise too and is but forty years old which is twenty year nigher to wisdom than me."

Then off he went to Little Tim, who was Big Tim's son, and he found him lying in a haystack chewing a straw.

"Morning, Little Tim," said Farmer John. "Morning, Farmer John," said Little Tim taking the straw from his mouth. Then Farmer John put his case again, saying, "Big Tim told me to come to you about it, for you were born wise".Big Tim put the straw back in his mouth again and said, "Young Tim was born wise too and he's but twenty years old. You'll get wisdom fresher from him than from me."

Then off went Farmer John to find Young Tim, who was Little Tim's son, and he found him staring into the mill pond, munching an apple.

"Morning, Young Tim," said Farmer John. "Morning, Farmer John," said Young Tim taking the apple from his mouth. Then Farmer John told his tale for the fourth time, and ended by saying, "Little Tim thinks you'll know what I'd best do, for you were born wise".Young Tim took a new bite of his apple and said, "My son who was born last month was born wise too, and from him you'll get wisdom at the fountain-head, so to say."

Off went Farmer John to find Baby Tim, who was Young Tim's son, and he found him in his cradle with his thumb in his mouth.

"Morning, Baby Tim," said Farmer John. Baby Tim took his thumb out of his mouth and said nothing. "I've had gypsies in my barn again, Baby Tim," said Farmer John, "and Young Tim advised me to ask your advice upon it, for you was born wise. What would you do if you was me? I'll do whatever you say."Baby Tim put his thumb back in his mouth and said nothing. So Farmer John went home and did it.

And the gypsies went on to the next village and slept in the barn of Farmer George, and Farmer George called in the constable and had the law on them; and a week later his barn and his ricks were burned down, and his speckled hen was stolen away.

But the happy village went on being happy and doing nothing, neither when the Miller's wife forgot herself one day and boxed the Miller's ears, nor when Molly Garden got a bad sixpence from the pedlar, nor when the parson once came home singing by moonlight. After consulting the Tims, the village did no more than trees do in a wood or crops in a field and so all these accidents got better before they got worse.

Until the day when Baby Tim died an unmarried man at one hundred years of age. After that the happy village became as other villages and did something.

Eversion

by Alastair Reynolds
ISBN-13 ‏ : ‎ 978-0575090781

There’s little to no sci-fi in the first 30% of this book and you’d be forgiven for thinking Reynolds was just indulging himself in a seafaring romp. There’s only a hint of sci-fi up to about half way through, when it develops into groundhog day. I didn’t enjoy this. I put the book down for a week or two until I forced myself to pick it up again and finish it. I couldn’t put the second half down!

There isn’t the usual Alastair Reynolds scope, but that doesn’t matter as there is plenty of his trademark exploration and discovery and a fantastic plot twist I didn’t see coming. By far my favourite character is Ada for her sharp sense of humour and general attitude to life. The other characters are convincing and, for once, there’s a good ending, even if it is a little corny.

Mini-bottles in hotel bathrooms...
Am I the only one who can't read the writing on them?
I mean the font size is tiny. You can't read them.
There's just no way to tell which bottles are which.
Shampoo? Conditioner? Body lotion? Moisturiser? Shower gel?
It's unbelievable how many different bottles there are!
(But no toothpaste).
Even if you have your reading glasses on you still can't read the labels.
Not that you have your reading glasses on when you're about to take a shower!
Not at first anyway.
You get in the bathroom and you see the 18 mini-bottles!
And you think - "figgins - which one is shampoo?"
You pick one hoping the bottles in this hotel will be different.
That you'll be able to read the label.
But no. You can't.
So you walk back into your room to try and find your glasses.
Now you're buck naked, wearing nothing but your glasses, about to take a shower!
You start reading the mini-bottle labels.
The colour is #3422DE and the background colour is #3422DD.
They're so similar you cannot read anything!
It's just not physically possible.
A small defeat in todays modern world!

the sad state of debug performance in c++

ARD

Following a nightmare which is parking in Manchester and getting a meal on a Saturday night without booking, we walked into the Bread Shed just as ARD were getting going, minus my vinyl and CDs for signing!. The masterpiece which is Take Up My Bones was instantly recognisable, as were composer and multi instrumentalist Mark Deeks and fellow Winterfyleth band mate Chris Naughton, both on guitar. The latter was centre stage, where surely Deeks should have been?

From the off the band, who were put together to perform an album which was never intended to be performed live, were a little loose with the drums too prominent and the guitars not clear enough. There appeared to be a lot retuning necessary, especially from Chris and the lead guitarist who appeared hidden away a lot of the time. This didn’t really detract from enjoyment of the incredible compositions from the album.  By the time the final 10 minutes, consisting of Only Three Shall Know, came along something had changed, the band was as tight as anything and I wished they could have started again from the beginning. 45 minutes had flown by and I’ll certainly go and see them again.

Winterfyleth

I think I’ve seen Winterfyleth four times now, including the set which became their live album recorded at Bloodstock and earlier this year supporting Emperor at Incineration Fest. They never disappoint.

Winterfyleth are one of those bands that are so consistent with their music, without being boring or repetitive, that it doesn’t matter what they play or how familiar I am with the songs, it’s just incredible to listen to. Having said that, disappointingly, they didn’t play A Valley Thick With Oaks, which is my favourite. Who can resist singing along “In the heart of every Englishman…”? However, I did come away with a new favourite in Green Cathedral!

We only got an hour, but at least they didn’t bugger about going off and coming back for an encore. There were old songs, new songs and never before played live songs. Loved every second of it and, for the first time for me, the final song wasn’t preceded with “Sadly time is short and our songs are long, so this is our last one.” Until next time!

 

Visual Lint 8.0.10.355 has now been released.

This is a maintenance update for Visual Lint 8.0, and includes the following changes:

• The VisualLintConsole command line parser now accepts spaces between the name and value of a command line switch. As a result, switches of the form. /switchname = <value> are now accepted, whereas previously only /switchname=<value> would work.

• Entries in the file list within manually created custom project (.vlproj) files can now use wildcards. Note however that if the project file is subsequently edited by VisualLintGui, the file list will be expanded to explicitly reflect the files actually found. As such, this feature is intended for use with hand-created .vlproj files (which are really just .ini files, and as such lend themselves nicely to hand-editing).

• Fixed a bug opening analysis tool manual PDFs located on UNC drives.

• Fixed a bug in VisualLintGui which could cause some file save operations to fail.

• Updated the PC-lint Plus compiler indirect file co-rb-vs2022.lnt to support Visual Studio 2022 v17.2.6.

• Updated the PC-lint Plus compiler indirect fileco-rb-vs2019.lnt to support Visual Studio 2019 v16.11.17.

• Updated the PC-lint Plus Boost library indirect file lib-rb-boost.lnt.

• Removed nonfunctional "Print" menu commands from VisualLintGui.

• Updates to various help topics.

Download Visual Lint 8.0.10.355

Glory! Hammer! Were fantastic!

They played well and were lots of fun as you’d expect.  I mean who doesn’t like a gig to start with a cardboard cutout of Tom Jones and Delilah playing on the PA. The band were all dressed up - it must have been very hot - and playing their parts.

I did find some of the gaps between songs and the interplay with the audience felt a little too Steel Panther. It was too frequent, superfluous and added time to a set which could have been shorter.

Sozos Michael is a phenomenal singer and makes it seem effortless and perfect. I’m a big fan of widdly guitar and it doesn’t stand out as much on record as it did live which was a really nice surprise.

It’ll be great to see them again when the promised new album is out and they tour again.

React Cookbook: Recipes for Mastering the React Framework

by David Griffiths and Dawn Griffiths
ISBN: 978-1492085843

This is a book of about 100 recipes across 11 sections. The sections range from the basics, such as creating React apps, routing and managing state to the more involved topics such as security, accessibility and performance.

I was especially pleased to see that the section on creating apps looked at create-react-app, nextjs and a number of other getting started tools and libraries, rather than just sticking with create-react-app.

I instantly liked the way each recipe laid out the problem it was solving, the solution and then had a discussion on different aspects of the solution. It immediately felt a bit like a patterns book. For example, after describing how to use create-react-app, the discussion section explains in more depth what it really is, how it works, how to use it to maintain your app and how to get rid of it.

As with a lot of React developers, the vast majority of the work I do is maintaining existing applications, rather than creating new ones from scratch. I frequently forget about how to setup things like routing scratch and would usually reach for Google. However, with a book like this I can see myself reaching for the easy to find recipes again and again.

Letter to my MP on deporting refugees to Rwanda, 2022-06-06.

Dear Ben Spencer,

Please do what you can to reverse the policy of sending asylum seekers to Rwanda.

We are breaking our proud tradition of commitment to refugees.

This policy seems to have the intention of preventing people from drowning while attempting to enter the UK. Instead of ruling people’s claims “inadmissable” because they were desperate enough to enter by a dangerous route, we should provide safe routes for people escaping war and harm.

I am sure this policy was drafted with good intent, but immediately it started we have seen disproportionate numbers of Sudanese people being deported to Rwanda [1] verses other nationalities. Even within the parameters of its own flawed morality, this policy is unfair in practice. It should be stopped immediately.

[1] https://www.theguardian.com/uk-news/2022/jun/06/home-office-offers-asylum-seekers-choice-between-war-zones-they-fled-and-rwanda

Yours sincerely,

Andy Balaam

If you want to write a similar letter, feel free to use any of the above if it’s helpful. I used WriteToThem.com as a very easy way to find your MP and send a message.

Visual Lint 8.0.9.353 has now been released.

This is a maintenance update for Visual Lint 8.0, and includes the following changes:

• Added support for Codegear C++ Builder 11 Alexandria.

• Fixed a crash in the Visual Studio plugin when a file was opened in Visual Studio 2022 v17.2.0 Preview. The crash was apparently caused by a change or regression in the VS2022 implementation of EnvDTE::DocumentEvents::DocumentOpening().

• When analysing Visual Studio projects the appropriate -std and -m32 or -m64 switches are now automatically added to generated Clang-Tidy command lines.

• Fixed a bug in the Clang-Tidy analysis results parser which caused issues containing multiple square brackets to be parsed incorrectly.

• Updated the PC-lint Plus compiler indirect file co-rb-vs2022.lnt to support Visual Studio 2022 v17.1.6.

• Updated the PC-lint Plus compiler indirect file co-rb-vs2019.lnt to support Visual Studio 2019 v16.11.13.

• Updated the PC-lint Plus compiler indirect file co-rb-vs2017.lnt to support Visual Studio 2017 v15.9.47.

• Updated the PC-lint Plus compiler indirect files co-rb-vs2017.lnt, co-rb-vs2019.lnt and co-rb-vs2022.lnt to limit the value of _MSVC_LANG to 201703L. This is necessary as the version of the Clang front-end used by PC-lint Plus 1.4.x and earlier does not include C++ 20 compiler intrinsics.

• Updated the installer to terminate any instances of mspdbsrv.exe (the Visual Studio program database symbol server) before installing the Visual Studio plugin to VS2017/19/22.

• Updated the text on the "Analysis Tools" page of the installer.

• Updated the readme and help documentation to reflect the fact that Windows 11 is now supported.

Download Visual Lint 8.0.9.353