Air-Source Heat Pump – 1 year later

10 months ago I wrote a blog post Air-Source Heat Pump – our experience so far, 2 months in about our new air source heat pump. Have a look back at that for photos of the device itself and more detail about installation etc.

Less energy

We used a lot less energy this year than last year. Here’s the graph for 2 years:

Graph showing 2 years of energy usage on gas and electricity. Gas usage stops halfway through because a heat pump was installed. The second year's gas usage is much lower than the first, especially during the heavy use, cold months.

As you can see, we used a lot less energy in kWh this year than last year. Air source heat pumps work!

More money

Our energy cost more this year than last year. I’ve calculated this graph based on fixed prices, and I used 2021 prices to keep consistency with the last blog post, but the real-prices story is similar. Here is the graph for the last 2 years:

Graph showing energy cost per day over 2 years. The first half shows gas usage, which drops to zero in the middle when a heat pump was installed. The second half is higher, showing that the cost of the electricity this year was more than last year. Towards the end, solar panels were installed and the cost drops below last year.

Why did our cost go up when our energy usage went down so dramatically?

Because electricity is too expensive!

Electricity is the right way to power our cars and homes, because it can be sourced sustainably, and in fact much of it really is being sourced sustainably right now.

Artificially-high electricity prices are preventing people switching to a better way.

Solar panels help!

In September this year we had solar panels installed. They look great and are working incredibly well. We were getting 10kWh per day from them in September. We don’t have a battery yet, but when it arrives we think we will be able to cover most of our summer energy costs using these panels.

Since the panels were installed, our reduced use of grid electricity meant that our energy costs for this year dropped below last year. Obviously it’s too soon to say for sure what the full impact is, but I can say we are very happy with our solar panels.

Our house is cozy

Our new heat pump heats our leaky house very well, and we are nice and cozy, even when temperatures outside drop below zero. The heat pump is less efficient when it’s cold outside, but still way better than a gas boiler.

Installed by Your Energy Your Way

[My wife used to be director of the company, so I declare an interest.]

Our heat pump, radiators and panels were installed by Your Energy Your Way and I can recommend them for good communication, service and quality.

Deleted my Twitter account

This evening I deleted my Twitter account. I’m feeling surprisingly unsettled by doing it, to be honest.

The Twitter mobile interface showing that my account has been deactivated.

I can’t participate in a platform that gives a voice to people who incite violence. I said I would leave if Trump were re-instated, and that is what I am doing. I know Twitter has failed in the past in many ways, but this deliberate decision is the last straw.

I have downloaded the archive of my tweets, and I will process it and upload it to my web site as soon as possible.

You can find me at @andybalaam@mastododon.social.

If you’d like help finding your way onto Mastodon, feel free to ask me for help via Mastodon itself, or by email on andybalaam at artificialworlds.net

IETF115 Trip Report (Can Matrix help messaging standardisation through MIMI?)

Geeks don’t like to be formal, but we do like to be precise. That is the
contrast that comes to mind as I attend my first IETF meeting, IETF 115 in
London in November 2022.

Like most standards bodies, IETF appears to have been formed as a reaction to
stuffy, process-encumbered standards bodies. The introductory material
emphasises a rough-consensus style, with an emphasis on getting things done
and avoiding being a talking shop. This is backed up by the hackathon organised
on the weekend before the conference, allowing attendees to get a taste of
really doing stuff, writing code to solve those thorny problems.

But the IETF has power, and, rightly, with power come checks and balances, so
although there is no formal secretary for the meeting I was in, a volunteer was
found to take minutes, and they made sure to store them in the official system.
It really matters what happens here, and the motley collection of (mostly
slightly aging) participants know that.

There is an excitement about coming together to solve technical problems, but
there is also an understandable caution about the various agendas being
represented.

We were at the meeting to discuss something incredibly exciting: finding a
practical solution for how huge messaging providers (e.g. WhatsApp) can make
their systems “interoperable”. In other words, making it possible for people
using one instant messenger to talk to people using another one.

Because of the EU’s new Digital Markets Act (DMA), the largest messaging
providers are going to be obliged to provide ways for outside systems to link
into them, and the intention of the meeting I went to (“More Instant Messaging
Interoperability”, codenamed MIMI) is to agree on a way they can link together
that makes the experience good for users.

The hardest part of this is preserving end-to-end encryption across different
instant messengers. If the content of a message is encrypted, then you can’t
have a translation layer (“bridge”, in Matrix terminology) that converts those
contents from one format into another, because the translator would need to
decrypt the message first. Instead, either the client (the app on the user’s
device) needs to understand all possible formats, or we need to agree on a
common format.

The meeting was very interesting, and conversation on the day revolved around
what exactly should be in scope or out of scope for the group that will
continue this work. In particular, we discussed whether the group should work
on questions of how to identify and find people on other messaging networks,
and to what extent the problems of spam and abuse should be considered. Mostly
the conclusion was that the “charter” of the “working group”, when/if it
is formed, should leave these topics open, so that the group can decide the
details when it is up and running.

From a Matrix point of view, this is very
exciting, because we have been working on exactly how to do this stuff for
years, and our goal from the beginning has been to allow interoperability
between other systems. So we have a lot to offer on how to design a system that
allows rich interactions to work between very different systems, while
providing effective systems to fight abuse. One part that we can’t tackle
without the co-operation of the dominant messengers is end-to-end encryption,
because we can’t control the message formats that are used on the clients, so
it’s really exciting that MIMI’s work might improve that situation.

I personally feel a little skeptical that the “gatekeepers” will implement DMA
in a really good way, like what we were discussing at IETF. I think it’s more
likely that they will do the bare minimum, and make it difficult to use so
that the experience is so bad that no-one uses it. However, if one or two major
players do adopt a proper interoperable standard, it could pick up all the
minor players, and become something genuinely useful.

I really enjoyed going to the IETF meeting, and am incredibly grateful to the
dedicated people doing this work. I really hope that the MIMI group forms and
successfully creates a useful standard. I’m also really optimistic that the
things we’ve learned while creating Matrix can help with this process.

Whether this standard begins to solve the horrible problems we have with closed
messaging silos and user lock-in to potentially exploitative or harmful
platforms is probably out of our hands.

If you’d like to get involved with MIMI, check out the page here:
datatracker.ietf.org/group/mimi/about.

Setting the text selection in a browser: just use setBaseAndExtent

The Selection API is confusing and weird.

But, here’s what I’ve discovered: just use setBaseAndExtend, and when (rarely) needed, extend.

Summary

Every selection in a browser consists of:

  • an “anchor” – the beginning, where you started dragging, and
  • a “focus” – the end, where your stopped dragging, and where your cursor is

To select some text in a browser, find the from and to DOM nodes you want, and how far through them you want to be, and set the selection like this:

const sel = document.getSelection();
sel.setBaseAndExtent(from_node, from_offset, to_node, to_offset);

If you want just a cursor with nothing selected, use the same node and offset for both from and to.

The Internet has a lot of recipes that involve creating Ranges and adding them to the Selection, but there is no need to do that.

Finding the from and to nodes

When the locations are inside a text node in the DOM, it’s easy to find the from and to nodes – they are just the text nodes, and the offsets are how many UTF-16 code units through the text you want to be. [Not sure what a code unit is? See my video Interesting Characters].

“But isn’t your cursor always in a text node?”

No – for example, when I have two <br>s next to each other, and I want the cursor to be in between them (so it’s on the blank line). In this case there is no text node, and browsers handle it weirdly.

In Firefox, when my cursor is between two br nodes, the node is set to the tag (e.g. a div) that contains the brs, and the offset is the index of the second br node in the list.

Other browsers may well do it differently.

So do expect that the nodes may not be text nodes, and where that is the case, the offset will be the index of one of their children.

To select the blank line (between two brs) you can actually specify the br node directly, and give an index number of 0, and it works too. But, it does not match exactly what the browser sets when you click on the blank line, so I can’t guarantee it works exactly the same.

Backwards selections

The fly in the ointment is backwards selections: when you click and drag the mouse from right to left, the selection that is created in the browser is backwards: the anchor is on the right and the focus is on the left.

However, if you call setBaseAndExtent attempting to set up a selection like that, the focus and anchor will be swapped so that the anchor is on the left, and the focus is on the right.

But never fear! We can use extend to force the selection to be what we wanted:

const sel = document.getSelection();
sel.setBaseAndExtent(from_node, from_offset, from_node, from_offset);
sel.extend(to_node, to_offset);

Job done.

Go deeper

For more info, check out my demo page: Browser Selections Inventory.

Tips for contenteditables

I’ve been working a bit with contenteditable tags in my HTML, and learnt a couple of things, so here they are.

Why can’t I see the cursor inside an empty contenteditable?

If you make an editable div like this:

<div contenteditable="true">
</div>

and then try to focus it, then sometimes, in some browsers, you won’t see a cursor.

You can fix it by adding a <br /> tag:

<div contenteditable="true">
<br />
</div>

Now you should get a cursor and be able to edit text inside.

Programmatically selecting text inside a contenteditable

It’s quite tricky to get the browser to select anything. Here’s a quick recipe for that:

<div id="ce" contenteditable="true">
Some text here
</div>
<script>
const ce = document.getElementById("ce");
const range = document.createRange();
range.setStart(ce.firstChild, 6);
range.setEnd(ce.lastChild, 10);
const sel = document.getSelection();
sel.removeAllRanges();
sel.addRange(range);
</script>

This selects characters 6 to before-10, i.e. the word “text”. To select more complicated stuff inside tags etc. you need to find the actual DOM nodes to pass in to setStart and setEnd, which is quite tricky.

Whenever you setHTML on a contenteditable, add a BR tag

If you use setHTML on a contenteditable you should always append a <br /> on the end. It doesn’t appear in any way, and it prevents weird problems.

Most notably, if you want to have an empty line at the end of your text, you need two <br /> tags, like this:

<div id="ce" contenteditable="true">
Some text here
</div>
<script>
const ce = document.getElementById("ce");
ce.innerHTML = "a<br /><br />"
</script>

If you only include one br tag, there will be no empty line at the end.

Selecting the end of a contenteditable

It’s surprisingly tricky to put the cursor at the end of a contenteditable div but here is a recipe that works:

const range = document.createRange();
range.selectNodeContents(ce);
range.collapse();
const sel = document.getSelection();
sel.removeAllRanges();
sel.addRange(range);

(Where ce is our contenteditable div.)

More tips?

Any more tips? Drop them in the comments and I’ll include them.

Outreachy August 2022 update

I had the pleasure of being a mentor this summer for an Outreachy internship for the Matrix organisation. Outreachy provides internships to people subject to systemic bias and impacted by underrepresentation in the technical industry where they are living.

Outreachy is a fantastic organisation doing a brilliant job to try and make our sometimes terrible industry a little bit better.

Outreachy logo

Mentoring was great fun, mainly because it was such a pleasure working with my awesome intern Usman. There is lots of support available for interns and mentors through Outreachy’s Zulip chat (when will we persuade them to use Matrix? ;-) so you always have somewhere to turn if you have questions.

If you want to read more about the internship from Usman’s point of view, check out his blog posts:

  • Outreachy Blog – Introducing Myself
  • Wrap-up: Summary of my journey to being an outreachy intern at Element

    We talked every day on video calls, and really enjoyed working together. Some days we would just chat, sometimes I would give pointers for things to try in the code, or people to talk to. Some days we worked through some code together, and that was the most fun. Usman is incredibly enthusiastic and bright, so it was very satisfying making suggestions and seeing him put them into practice.

    Success!

    The work went very well, and Usman succeeded in creating a prototype that will help us design the Favourite Messages feature:

    Note: the feature isn’t ready to be fully release because it needs to be implemented on mobile platforms as well as changing where it stores its information: currently we use the browser’s local storage, but we plan to store things in Matrix, meaning it is automatically synchronised between devices.

    Things that went well

    • Meeting every day: we talked on a short video call every morning. This meant if we misunderstood each other it was quickly resolved, without lots of time being wasted.
    • Having a clear list of tasks: we kept a tracking issue on Github. This meant were clear what Usman was supposed to be doing now, and what was coming next.
    • Being flexible: we worked together to change the list of tasks every week or so. This meant we were being realistic about what could be achieved, and able to change in response to things we found out, or feedback from others.
    • Getting design input: we talked to Element’s designers several times during the project, showing them prototypes and early implementations. This meant we didn’t waste time implementing things that would need redesign later.
    • Support for me: I was able to work with Thib, who is our Outreachy Matrix community organiser, especially during the selection process. This meant I was not making decisions in isolation, and had support if anything tricky came up.
    • The Element Web community: Usman got loads of support from our community. Special thanks to Šimon, Olivier, Shay and t3chguy for your help!
    • Element the company: Element paid for this internship, and gave great support to Usman, integrating him into all our systems, inviting him to introductory meetings etc. He had every opportunity to see what working at Element is like, and to make an impression on everyone here. Element did a great job here.

    Things I would do differently

    • Managing the contribution period: before the project began, applicants are invited to contribute to the projects, allowing us to choose an intern based on those contributions. I felt slightly disorganised at this stage, and there was a lot of activity in issues and pull requests in the project from applicants. I think I should have warned our community and explained what was going to happen up-front, and maybe enlisted help from people willing to triage the contributions a little. Contributions varied in quality and understanding level, so having some volunteers who were primed to spend a little more time explaining and helping contributors get started would have prevented this impinging on the time of the team as a whole. Nevertheless, our team responded really well, and we got some useful contributions, and I hope the contributors had a good experience too.
    • Integrating Usman into the team: we chose a project that was independent from what other team members were doing, meaning he mostly interacted with others when he needed help. While it is sensible to make sure interns are decoupled from the main work (because it’s hard to predict how much progress they will make, and they are going stop after their internship), I do also wish we could have found a project that gave more opportunity to work with other people, not just “stealing” their time to help out, but actually working together on shared pieces of work. This is a tricky one to figure out, but food for thought.

    Conclusions

    The experience of being a mentor was really fun, and I would recommend it to anyone working on an open source project.

    I would emphasise, though, that you need to put aside enough time: the internship will not be successful if you don’t make time to work with your intern, get to know them, and introduce them to your community. Since interns may be new to the world of work, or shy about taking your time, as a mentor, you need to take responsibility for giving them enough support.

    Final note: as a mentor, you are NOT responsible for the work going well! Your responsibility is to help and support your intern, and give them everything they need to be successful (including feedback about things that are not working well), but it is up to the the intern themself to do the work, and how much work gets done is going to be the combination of a number of factors, including the intern’s experience and abilities. Don’t worry if you don’t get as far as you expected – after all, that happens in nearly all software projects…

Transcoding video files for playback in a browser

ffmpeg -i original.mkv -c:v libx264 -c:a aac -ac 2 -ab 384000 -ar 48000 new.mp4

(Short answer: use the above ffmpeg command line. Read on for how I did this in Tdarr.)

I recently discovered Jellyfin, which gives me a Netflix-like UI for viewing my own videos, and seems great.

The only problem I had was that some videos were in formats that can’t be played natively in a web browser. Jellyfin heroically tries to transcode them on the fly, but my server is very lightweight, and there’s no way it can do that.

So, I needed to transcode those videos to a more suitable format.

Tdarr allows transcoding large numbers of files, and with a little head-scratching I worked out how to get it running, but I still needed the right ffmpeg options to make the videos work well in Firefox, without needing transcoding of video, audio or the container.

Here are the Tdarr settings that I found worked well:

Output file container: .mp4
Encoder: ffmpeg
CLI arguments: -c:v libx264 -c:a aac -ac 2 -ab 384000 -ar 48000
Only transcode videos in these codecs: hevc

Explanation:

  • Output file container: .mp4 – wraps the video up in an MP4 container – surprisingly, Firefox doesn’t seem to support MKV.
  • -c:v libx264 – re-encodes the video as H.264. Firefox can’t do H.265, and H.264 is reasonably compatible with lots of browsers. If you don’t care about Safari or various Microsoft browsers, you might want to think about VP9 as it’s natively supported on Firefox, so should work on weird architectures etc.
  • -c:a aac -ac 2 -ab 384000 -ar 48000 – re-encodes the audio as AAC with the right bitrates etc. Jellyfin was still transcoding the audio when I just specified -c:a aac, and it took me a while to work out that you need those other options too.
  • Only transcode videos in these codecs: hevchevc means H.265 encoding, and the videos I had problems with were in that encoding, but you might have different problems. If in doubt, you can choose “Don’t transcode videos in these codecs:” and uncheck all the encodings, meaning all your videos will be re-encoded.

If you are not using Tdarr, here is the plain command line to use with ffmpeg:

ffmpeg -i original.mkv -c:v libx264 -c:a aac -ac 2 -ab 384000 -ar 48000 new.mp4

Matrix is a Distributed Real-time Database Video

Curious to know a bit more about Matrix? This video goes into the details of what kinds of requests you need to send to write a Matrix client, and why it’s interesting to write a Matrix server.

Slides: Matrix is a Distributed Real-time Database Slides

Really excited that since I started my job working on Matrix, I have become more enthusiastic about it, rather than less.

Building cross-platform Rust for Web, Android and iOS – a minimal example

One of the advantages of writing code in Rust is that it can be re-used in other places. Both iOS and Android allow using native libraries within your apps, and Rust compiles to native. Web pages can now use WebAssembly (WASM), and Rust can compile to WASM.

So, it should be easy, right?

Well, in practice it seems a little tricky, so I created a small example project to explain it to myself, so maybe it’s helpful to you too.

The full code is at gitlab.com/andybalaam/example-rust-bindings, but here is the general idea:

crates/example-rust-bindings - the real Rust code
bindings/ffi - uniffi code to build shared objects for Android and iOS
bindings/wasm - wasm_bingen code to build WASM for Web
examples/example-android - an Android app that generates a Kotlin wrapper, and runs the code in the shared object
examples/example-web - a web page that imports the WASM and runs it

Steps for WASM

Proof that I did this on Web - Firefox showing "This string is from Rust!"

Variation: if you modify the build script in package.json to call wasm-pack with --target node instead of --target web you can generate code suitable for using from a NodeJS module.

Steps for Android

Proof that I did this on Android: Android emulator showing a label "This string is from Rust!"

Steps for iOS

I am working on this and will fill it in later.

Deporting desperate people from the UK

Letter to my MP on deporting refugees to Rwanda, 2022-06-06.

Dear Ben Spencer,

Please do what you can to reverse the policy of sending asylum seekers to Rwanda.

We are breaking our proud tradition of commitment to refugees.

This policy seems to have the intention of preventing people from drowning while attempting to enter the UK. Instead of ruling people’s claims “inadmissable” because they were desperate enough to enter by a dangerous route, we should provide safe routes for people escaping war and harm.

I am sure this policy was drafted with good intent, but immediately it started we have seen disproportionate numbers of Sudanese people being deported to Rwanda [1] verses other nationalities. Even within the parameters of its own flawed morality, this policy is unfair in practice. It should be stopped immediately.

[1] https://www.theguardian.com/uk-news/2022/jun/06/home-office-offers-asylum-seekers-choice-between-war-zones-they-fled-and-rwanda

Yours sincerely,

Andy Balaam

If you want to write a similar letter, feel free to use any of the above if it’s helpful. I used WriteToThem.com as a very easy way to find your MP and send a message.

Improving my vimrc live on stream

I was becoming increasingly uncomfortable with how crufty my neovim config was getting, and especially how I didn’t understand parts of it, so I decided to wipe it clean and rebuild it from scratch.

I did it live on stream, to make it feel like a worthwhile activity:

Headline features of the new vimrc:

  • A new theme, using the base16 theme framework
  • A file browser (NERDTree)
  • A minimalist status line with vim-airline
  • Search with ripgrep
  • Rust language support with Coc

Note: after the stream I managed to resolve the remaining issues with highlight colours not showing by triggering re-applying them after the theme has been applied:

augroup tabs_in_make
    autocmd!
    autocmd ColorScheme * highlight MatchParen cterm=none ctermbg=none ctermfg=green
augroup END

You can find my current neovim config at gitlab.com/andybalaam/configs/-/tree/main/.config/nvim.

Comparison of Matrix events before and after “Extensible Events”

(Background: Matrix is the awesome open standard for messaging that I get to work on now that I work at Element.)

The Extensible Events (MSC1767) Matrix Spec Change proposal describes a new way of structuring events in matrix that makes it easy to send events that have multiple representations (e.g. something clever like an interactive map, and something simpler like an image of a map).

The main purpose of the change is to make it easy for clients that don’t support some amazing new feature to display something that is still useful.

Since there is an implementation of this change out in the wild (in Element), it seems reasonably likely that this change will be accepted into the Matrix spec.

I really like this change, but I find it hard to understand, so here is a simple example that I have found helpful to think it through.

An old event, and a new event

Here is an old-fashioned event, followed by a new, shiny, extensible version:

{
    "type": "m.room.message",
    "content": {
        "body": "This is the *old* way",
        "format": "org.matrix.custom.html",
        "formatted_body": "This is the <b>old</b> way",
        "msgtype": "m.text"
    },
    ... other properties not relevant to this, e.g. "sender" ...
}
{
    "type": "m.message",
    "content": {
        "m.message": [
            {"mimetype": "text/plain", "body": "This the *new* way"},
            {"mimetype": "text/html", "body": "This is the <b>new</b> way"}
        ],
    }
    ... other properties not relevant to this, e.g. "sender" ...
}

Notice that in the new extensible events, the property within content is the same as the message type (here: m.message).

The point is that as well as the primary event type (here, m.message) we can other representations of the same message, such as an image, location co-ordinates, or something completely different. The client will render the primary event type if it understands it (and is able to show it), but if not, it can look for other types that it does understand.

For example, in Polls when you send a new poll question, it could look like this:

{
    "type": "m.poll.start",
    "content": {
        "m.poll.start": {
            ... The actual poll question etc. ...
        },
        "m.message": [
            ... A text version of the question ...
        ]
    },
    ... other properties not relevant to this, e.g. "sender" ...
}

So clients that don’t know m.poll.start can still display the poll question (if they understand extensible events), instead of completely ignoring event types they don’t know about.

An abbreviated form of the new event

Of course, life is not quite as simple as that.

Because this is a lot of typing:

{
    "type": "m.message",
    "content": {
        "m.message": [
            {"mimetype": "text/plain", "body": "This the *new* way"},
            {"mimetype": "text/html", "body": "This is the <b>new</b> way"}
        ],
    }
    ... other properties not relevant to this, e.g. "sender" ...
}

We have an abbreviated form:

{
    "type": "m.message",
    "content": {
        "m.text": "This the *new* way",
        "m.html": "This is the <b>new</b> way"
    }
    ... other properties not relevant to this, e.g. "sender" ...
}

These two are exactly equivalent.

m.text is an abbreviation for an m.message containing an entry with "mimetype": "text/plain" and the relevant body. Similarly, m.html is an abbreviation for an m.message containing an entry with "mimetype": "text/html" and the relevant body. If you declare both, they effectively get squashed together into one m.message with both entries.

Those 2 are the only abbreviations listed, so they are special cases.

Backwards compatibility

Of course, life is way more complicated than that, so what we’re likely to see around if/when this gets widely adopted is some kind of mashed-together event like this:

{
    "type": "m.room.message",
    "content": {
        "msgtype": "m.text",
        "body": "Hello World",
        "format": "org.matrix.custom.html",
        "formatted_body": "<b>Hello</b> World",
        "m.text": "Hello World",
        "m.html": "<b>Hello</b> World"
    }
}

Note that the type here is m.room.message, where extensible events says it should be m.message. The idea is that an extensible-events-aware client will see "msgtype": "m.text" and know to look for m.message as the primary type. (This is further complicated here by the fact that there isn’t actually a m.message property – this is because m.text and m.html are abbreviated forms of it.)

Also, clients that want to display old events will need to preserve their code that parses the old event types in perpetuity.

Air-Source Heat Pump – our experience so far, 2 months in

Summary: less energy, more money

2 months ago, we replaced our gas boiler with an air-source heat pump, which uses electricity to heat our home and boiler. This is a report of our experience so far.

We expected it to reduce our environmental impact, and cost us more money, and we were right.

It works: our house is comfortable. We use a lot less energy, and it costs us significantly more money (because electricity costs way more than gas).

The house

Our house is a beautiful, leaky old house, with a modern extension. Half of it is well-insulated. The other half was built around 1890, and while we do have double-glazing and decent loft insulation, the walls have no cavities and feel cold to the touch, and there are drafts everywhere.

The new half has underfloor heating. The old half and the upstairs are heated by radiators. We have a hot water cylinder.

The air-source heat pump

Our air-source heat pump uses electricity to extract heat from the outside air and heats water for radiators and hot water, directly replacing our gas boiler.

Our heat pump was installed by Your Energy Your Way and I must declare in interest: my wife is a director of the company.

The heat pump is an LG 16kW “THERMA V” model. It looks like a very large air conditioning unit, which sits outside our house in the yard to the side. It is about as tall as my shoulder height, with two big fans on it.

A large air-source heat pump

It stands on a soak-away area with some stones on it that the installers made by removing some patio tiles. This is needed because it drips a small amount of liquid as part of its normal operation. The outdoor unit makes noise, but our house is next to the main road, so we don’t hear it. It is not audible indoors.

Standing next to the outdoor unit you can feel a cold breeze, like opening the fridge door. This is unpleasant on cold days.

That outdoor unit connects through the wall to an indoor part that is a bit smaller than our old boiler.

The controller box has a terrible user interface and is very hard to decipher, but we did eventually manage to programme it to turn the target temperature up in the daytime and down at night. Your Energy Your Way advised us that it is more efficient to keep the house at a cool-ish 17 degrees at night, rather than letting it get cold and having to work hard heat it up again in the morning, so that is how we have set it up.

The controller box’s built-in thermostat does not work properly (it reports the wrong temperature), so we had to add an external thermostat, which works well.

We didn’t need to change anything about our hot water cylinder, or our underfloor heating.

When planning the installation, Your Energy Your Way estimated the heat loss of our rooms, and recommended upgrading our radiators. In an old house like ours this is sometimes needed, because it is way more efficient to heat a house with cooler water running through the radiators, but if the water is cooler, you need more radiator surface area to heat the house effectively. In a newer house with existing radiators, they are probably fine as-is.

We kept most of the existing radiators, and added some more in the coldest rooms.

How comfortable is the house?

The house is more comfortable than it was before, for two reasons: firstly the radiators we had were not really adequate, and secondly the cooler water in the radiators makes a less irritating heat, meaning the house is nicely comfortable most of the time, instead of bouncing between feeling cold and feeling oppressively over-heated.

On cold days, the old part of the house is a bit cold, but I think on average it’s a little better than it was before.

We do find mornings can be chilly, particularly because the system stops heating the radiators if the hot water cylinder needs heating up after people have had showers. We could improve this situation by getting a larger cylinder, which we are considering.

However, it’s worth pointing out that we needed engineers to visit four or five times to make adjustments before we felt the system was working well enough. There are a lot of things that can be tweaked, and it took some time for it to work well.

My advice: don’t pick the cheapest quote – pick the people you think you can trust to do the work well: especially the heat loss calculations before installation and the adjustments afterwards.

How much energy are we using? (The good news)

So far, it looks like we are using about two-thirds less energy in our household than we were before:

The above chart is stacked, so the top line represents the total energy usage. We switched to the air-source heat pump exactly when our gas usage was about to skyrocket (because it’s cold in winter), and it remained relatively low.

This is absolutely fantastic: our house is more comfortable than before, and we have reduced the amount of energy we are using by 66%. This is the total energy usage of our house, not just for heating, so the reduction of energy used for heating is even more dramatic than it looks.

Even better, the energy we use is at least partly produced from renewable sources, so our carbon footprint is much lower. Previously we were directly releasing carbon by burning imported gas – now we use mostly UK-produced electricity, and as the grid decarbonises, our carbon footprint reduces even if we make no further changes.

How much money are we spending? (The bad news)

Excluding standing charges*, we are spending about one third more on energy than we were before. This is because electricity is so much more expensive than gas: our electricity costs 19p per kWh and our gas costs 4p per kWh.

* Note: our energy provider wanted to charge us £350 to remove our gas meter, so we refused, and are still paying the gas standing charge. I’m not sure how we’re going to resolve this, especially since our energy provider is now in administration.

The above chart is stacked, so the top line represents the total cost (excluding standing charges). When we switched to the air-source heat pump, our energy costs increased faster than they did the same time last year, and were consistently higher. We think the peak in November might be misleading as it may have been when the system was not set up correctly, but we are not sure.

Because air-source heat pumps are more efficient when the weather is warmer, we do expect to fare better in the summer than we are right now.

I would not suggest getting a heat pump if you want to save money. Maybe this will change as gas prices are expected to rise significantly this year.

An installation like ours, including new radiators, costs £10-15K. A decent chunk of that will be paid back to us by the government, spread out over the next 7 years, under the soon-to-be-gone Renewable Heat Incentive (RHI). RHI will be replaced by the
Boiler Upgrade Scheme (BUS), which will be limited to a £5K grant for air-source heat pumps, although it is paid up-front. We would have received much less money under BUS than RHI. It is almost certainly too late for you to get a heat pump under RHI, by the way – all the installers are booked up until end of March 2022, when it ends.

Thoughts

If you think it’s surprising (and deeply concerning) that taking the step of significantly reducing our carbon footprint cost us a one-third increase in our energy bills, I would agree with you.

I am told that the tax taken on electricity is much higher than on gas, even though these taxes are apparently intended help decarbonise our energy.

Meanwhile, the government is replacing (with great fanfare) RHI with the much less generous (although more timely) BUS, making it even more economically punishing to reduce your carbon footprint.

I think this should be addressed urgently: money should be provided to help people install heat pumps, and the tax regime should be changed to make it cheap to use low-carbon fuels.

The technology is available, but the financial situation makes this a vanity project for people like me who can afford it, instead of what it could be: a feasible plan to get our national carbon usage down, fast.

On a positive note, our house is nice and warm, and I feel a bit less guilty about how much carbon we’re using to keep it cosy.

Providing MapLibre-compatible style JSON from openstreetmap-tile-server

[Previous: Self-hosting maps on my laptop]

In the previous post I showed how to run OSM tile server stack locally.

Now I’ve managed to connect a MapLibre GL JS front end to my local tile server and it’s showing maps!

(It’s running inside Element Web, the awesome Matrix messenger I am working on. NOTE: this is a very, very early prototype!)

In the previous post I ran a docker run command to launch the tile server.

This time, I had to create a file style.json:

{
  "version": 8,
  "sources": {
    "localsource": {
      "type": "raster",
      "tiles": [
        "http://127.0.0.1:8080/tile/{z}/{x}/{y}.png"
      ]
    }
  },
  "layers": [
    {
      "id": "locallayer",
      "source": "localsource",
      "type": "raster"
    }
  ]
}

and then I launched the tile server with that file available in the document root:

docker run \
    -p 8080:80 \
    -v $PWD/style.json:/var/www/html/style.json \
    -v openstreetmap-data:/var/lib/postgresql/12/main \
    -v openstreetmap-rendered-tiles:/var/lib/mod_tile \
    -e THREADS=24 \
    -e ALLOW_CORS=enabled \
    -d overv/openstreetmap-tile-server:1.3.10 \
    run

Now I can point my MapLibre GL JS at that style file with code something like this:

this.map = new maplibregl.Map({
    container: my_container,
    style: "http://127.0.0.1:8080/style.json",
    center: [0, 0],
    zoom: 13,
});

Very excited to be drawing maps without any requests leaving my machine!

Self-hosting maps on my laptop

[See also: Providing MapLibre-compatible style JSON from openstreetmap-tile-server]

As part of my research for working on location sharing for Element Web, the Matrix-based instant messenger, I have been learning about tile servers.

I managed to get OSM tile server stack working on my laptop:

Here are a couple useful pages I read during my research:

Today I managed to run a real tile server on my laptop, using data downloaded from OpenStreetMap in a way that I think complies with their terms of use.

To run these commands you will need Docker, and hopefully nothing much else.

Download the data

I downloaded the UK data like this:

wget 'https://download.geofabrik.de/europe/great-britain-latest.osm.pbf'

You can find downloads for other regions at download.geofabrik.de/

Import it

Then I ran an import, which converts the PBF data into tiles that can be shown in a UI:

docker volume create openstreetmap-data
docker volume create openstreetmap-rendered-tiles
docker run \
    -v $PWD/great-britain-latest.osm.pbf:/data.osm.pbf \
    -v openstreetmap-data:/var/lib/postgresql/12/main \
    -v openstreetmap-rendered-tiles:/var/lib/mod_tile \
    -e THREADS=24 \
    overv/openstreetmap-tile-server:1.3.10 \
    import

(Change “great-britain” to match what you downloaded.)

On my quite powerful laptop this took 39 minutes to run.

Run the tile server

Finally, I launched the server:

(Make sure you’ve done the “Import it” step first.)

docker run \
    -p 8080:80 \
    -v openstreetmap-data:/var/lib/postgresql/12/main \
    -v openstreetmap-rendered-tiles:/var/lib/mod_tile \
    -e THREADS=24 \
    -e ALLOW_CORS=enabled \
    -d overv/openstreetmap-tile-server:1.3.10 \
    run

This should launch the docker container in the background, which you can check with docker ps.

Test it

You can now grab a single file by going to http://127.0.0.1:8080/tile/0/0/0.png, or interact with the map properly at http://127.0.0.1:8080.

It was quite unresponsive at first, but once it had cached the tiles I was looking at, it was very smooth.

Streaming to Twitch and PeerTube simultaneously using nginx on Oracle cloud

Simulcasting RTMP using NGINX

I want people to be able to watch my Matrix and Rust live coding streams using free software, so I’d like to simulcast to PeerTube as well as Twitch.

This is possible using NGINX and its RTMP module. It does involve building NGINX from source, but I actually found that reasonably easy to do.

Why Oracle cloud?

I would never recommend using Oracle for anything, but they do provide up to two virtual machines in their cloud for free, and the one I am using has been consistently available with very good connectivity, in a London data centre since I set it up several months ago.

So, we are making our lives more difficult by trying to do this on Oracle Linux, which is a derivative of RHEL.

Building NGINX and its RTMP module on Oracle Linux

I ran these commands on my Oracle cloud instance (running Oracle Linux):

sudo yum install git pcre-devel openssl-devel
mkdir nginx
cd nginx
wget http://nginx.org/download/nginx-1.21.4.tar.gz
git clone https://github.com/arut/nginx-rtmp-module.git
cd nginx-1.21.4
./configure --add-module=../nginx-rtmp-module/
make
sudo make install

After all this NGINX was installed to /usr/local/nginx/.

Creating the NGINX config file for RTMP simulcasting

Next I edited the NGINX config file by typing:

sudo nano /usr/local/nginx/conf/nginx.conf

And pasted in this config at the bottom of the file:

rtmp {
    server {
        listen 1935;
        chunk_size 4096;
        application live {
            live on;
            record off;
            push rtmp://live.twitch.tv/app/live_INSERT_TWITCH_STREAM_KEY;
            push rtmp://diode.zone:1935/live/INSERT_PEERTUBE_STREAM_KEY;
        }
    }
}

Notice that you will need to get your Twitch stream key from Twitch -> Creator Dashboard -> Settings -> Stream, then Copy next to the Primary Stream Key.

To get a PeerTube stream ID, you will need to go to your PeerTube page and click Publish, then Go Live, choose your channel and choose Go Live. Note that if you want the streams to record and be available later, you have to create a new stream key each time you start a stream, and change it in nginx.conf.

If you use a different PeerTube server (I use diode.zone) then you’ll need to change the server name in the config file above too.

Make sure your config file is saved with the right URLs in it.

Opening ports

To send RTMP traffic to my server, I needed to open the right port to the Oracle cloud instance. That involved creating an ingress rule, and adding a firewall rule.

Creating an ingress rule

In the web interface, I went to the menu in the top left, clicked Compute, then Instances.

I clicked on my instance’s name, then I clicked on the name of the subnet in the details (on the right).

I clicked on Default security list for…, then Add Ingress Rules.

I made an ingress rule with Source Type=CIDR, Source CIDR=0.0.0.0/0, IP Protocol=TCP, Source Port Range=(blank, meaning all), Destination Port Range=1935

Adding a firewall rule

Then I ssh’d into the machine and ran these commands to create a firewall rule allowing the traffic:

sudo firewall-cmd --zone=public --permanent --add-port=1935/tcp
sudo firewall-cmd --reload

Stop and Start NGINX

After creating the config file and opening the right port, I needed to start NGINX.

Every time I change the config file, I need to restart it.

If it’s already running, I stop it with:

sudo /usr/local/nginx/sbin/nginx -s stop

and then I start it up again with

sudo /usr/local/nginx/sbin/nginx

I can check whether it’s happy by looking at the log files, for example to see any errors:

less /usr/local/nginx/logs/error.log

Starting the stream

Now I go into OBS and go to File -> Settings -> Stream and choose the type as Custom, and the Server as rtmp://1.1.1.1/live. (But instead of 1.1.1.1 I put the public IP address of my instance, which I found by clicking the name of the instance in the Oracle cloud management console.)

New game: Tron – frantic multiplayer retro action

My newest game is out now on Smolpxl Games – Tron:

Pixellated lines fight each other to stay alive

Play at smolpxl.gitlab.io/tron.

It’s a frantic multiplayer retro pixellated thingy playable in your browser. Try to stay alive longer than everyone else!

This version allows many players (up to 16 if you can manage it), and is quite pure in its implementation.

There are bots to play against, and you can gather your friends around a keyboard to play together.

Part of the motivation for writing this game was to test my new smolpxl-remote remote-play system, but this is not enabled yet, so watch this space…

I love playing games with other people – preferably at least 3 other people. In theory you could have 8 players around a keyboard playing this – send me a picture if you try!

One feature I worked on in the Smolpxl library for this game: saving configuration to local storage (and asking permission to do so). I ended up with a very ugly hack to do this, so a bit more work is needed before I merge it into the library.

Preventing Virgin Media hijacking my DNS

Yesterday I learned that Virgin Media is inserting itself into some of my DNS requests. Much as I am not a fan of how powerful Cloudflare are, if they are telling the truth about their DNS, then it’s safe, so I followed their instructions on how to use their DNS and then removed the default DNS and hopefully my Internet will work now.

From the serverfault answer by lauc.exon.nod:

nmcli con mod "Wired connection 1" ipv4.dns "1.1.1.1 1.0.0.1"
nmcli con mod "Wired connection 1" ipv4.ignore-auto-dns yes
nmcli con down "Wired connection 1"
nmcli con up "Wired connection 1"

Letter to my MP about climate emergency

[Introduction including details about my own air source heat pump install, and mention of the ending of the RHI funding in April 2022.]

After I have installed an air source heat pump, I will pay more money to heat my home, even though I am using less energy, because electricity is more expensive than gas. So this change will hurt me financially over both the short and longer terms.

Do you agree with me that climate emergency is the most important issue the government is now facing?

Do you also agree with me that we urgently need people to switch their heating and home insulation to reduce our dependence on burning gas?

Please do all you can to persuade the Prime Minister to introduce initiatives before COP26 that make it financially viable for families without spare cash to insulate their home and heat them with renewable energy.

Please pass my letter on to the Prime Minister and any government departments you consider relevant.

Thank you very much for your time.

New Job at Element (Matrix)

I started a new job today at Element!

It has been a long-standing ambition of mine to work in Free and Open Source software, and I am very excited to work for a company that is the main developer of a really important project: the Matrix communication network.

I don’t know much about what I’ll be doing yet, but finding an open source company with a decent business model that is prepared to pay me is very exciting. The fact that they have offices that are close enough for me to go for is another huge bonus.

Wish me luck, and I’ll let you know what I’m working on when it becomes more clear.

What to cache when building Rust using Gitlab CI or similar

When building your project with Gitlab CI or a similar build tool, you can end up spending a lot of time watching your build repeat the same steps over and over. This is especially frustrating when it mostly consists of downloading and compiling the same things we downloaded and compiled last time.

To mitigate this, we can ask Gitlab CI to cache things that will be the same next time.

For a Rust project, the most important thing to cache is target in the local directory.

But, if you are installing tools using rustup or cargo, it will really help if you cache those too. Fortunately, Rust knows where those are by using environment variables, and these are defined in the standard Rust Docker image.

We can make sure we’re caching as much as possible by adding a section like this to .gitlab-ci.yml:

    cache:
        key: shared-cache
        paths:
            - target/
            - $CARGO_HOME/
            - $RUSTUP_HOME/

If you add this to all your jobs, they will share a single cache between them, and cache the local target directory as well as any tools installed with rustup or cargo.

Here is a full example from my Evolve SVGs project:

image: rust:latest

before_script:
    - rustup component add rustfmt
    - rustup target add wasm32-unknown-unknown
    - cargo install trunk wasm-bindgen-cli

pages:
    stage: deploy
    script:
        - echo "Publishing pages to" $CI_PAGES_URL
        - make deploy
        - mv dist public
    artifacts:
      paths:
        - public
    only:
        - main
    cache:
        key: shared-cache
        paths:
            - target/
            - $CARGO_HOME/
            - $RUSTUP_HOME/

test:
    stage: test
    script:
        - make test
    cache:
        key: shared-cache
        paths:
            - target/
            - $CARGO_HOME/
            - $RUSTUP_HOME/

Minimal example of a Maven pom for a mixed Kotlin and Java project

The Kotlin docs describe some things you need in your pom.xml to create a project that is a mix of Kotlin and Java code, but there is no complete example, so here is mine:

pom.xml:

<project>
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.example.kj</groupId>
    <artifactId>kotlin-and-java</artifactId>
    <version>1.0.0-SNAPSHOT</version>

    <properties>
        <kotlin.version>1.5.21</kotlin.version>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
    </properties>

    <build>
        <plugins>
            <plugin>
                <groupId>org.jetbrains.kotlin</groupId>
                <artifactId>kotlin-maven-plugin</artifactId>
                <version>${kotlin.version}</version>
                <executions>
                    <execution>
                        <id>compile</id>
                        <goals>
                            <goal>compile</goal>
                        </goals>
                        <configuration>
                            <sourceDirs>
                                <sourceDir>${project.basedir}/src/main/kotlin</sourceDir>
                                <sourceDir>${project.basedir}/src/main/java</sourceDir>
                            </sourceDirs>
                        </configuration>
                    </execution>
                    <execution>
                        <id>test-compile</id>
                        <goals> <goal>test-compile</goal> </goals>
                        <configuration>
                            <sourceDirs>
                                <sourceDir>${project.basedir}/src/test/kotlin</sourceDir>
                                <sourceDir>${project.basedir}/src/test/java</sourceDir>
                            </sourceDirs>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.5.1</version>
                <executions>
                    <!-- Replacing default-compile as it is treated specially by maven -->
                    <execution>
                        <id>default-compile</id>
                        <phase>none</phase>
                    </execution>
                    <!-- Replacing default-testCompile as it is treated specially by maven -->
                    <execution>
                        <id>default-testCompile</id>
                        <phase>none</phase>
                    </execution>
                    <execution>
                        <id>java-compile</id>
                        <phase>compile</phase>
                        <goals>
                            <goal>compile</goal>
                        </goals>
                    </execution>
                    <execution>
                        <id>java-test-compile</id>
                        <phase>test-compile</phase>
                        <goals>
                            <goal>testCompile</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>
    <dependencies>
        <dependency>
            <groupId>org.jetbrains.kotlin</groupId>
            <artifactId>kotlin-stdlib</artifactId>
            <version>${kotlin.version}</version>
        </dependency>
    </dependencies>
</project>

src/main/java/MyJava.java:

public class MyJava {
    public static void main(String[] args) {
        MyKotlin k = new MyKotlin();  // Use Kotlin from Java
        System.out.println(k.message());
    }
}

src/main/kotlin/MyKotlin.kt:

class MyKotlin : MyJava() {  // Use Java from Kotlin
    fun message(): String {
        return "Hello from Kotlin!"
    }
}

src/test/java/MadeInJavaTest.java:

class MadeInJavaTest {
    public void testCanUseJava() {
        MyJava j = new MyJava();
    }

    public void testCanUseKotlin() {
        MyKotlin k = new MyKotlin();
        assertEquals(k.message(), "Hello from Kotlin!");
    }

    static void assertEquals(String left, String right) {
        if (!left.equals(right)) {
            throw new AssertionError(left + " != " + right);
        }
    }
}

src/test/kotlin/MadeInKotlinTest.kt:

class MadeInKotlinTest {
    fun testCanUseJava() {
        MyJava()
    }

    fun testCanUseKotlin() {
        val k = MyKotlin();
        assertEquals(k.message(), "Hello from Kotlin!");
    }
}

fun assertEquals(left: String, right: String) {
    if (left != right) {
        throw AssertionError("${left} != ${right}");
    }
}

Importing/migrating from one peertube server to another

My Peertube server is shutting down, so I need to move my videos to another one. The official scripts don’t seem to cover this case very well, so here is what I did.

My script fetches videos and their details and uploads them to the new server via the Peertube API.

Contributions welcome: I was not able to copy video descriptions across, and I was too lazy so I hard-coded the number of tags. Also, I didn’t make a git repo for all this because I felt it needs more thought, but feel free to start one and I will happily contribute this.

This script copies all videos in a single Peertube channel to a different server. You must find the numeric ID of the channel on the new server to copy into, which I did by looking at the responses in the Network tab of Firefox’s developer tools when I clicked on its name in the web interface. It requires bash, curl, youtube-dl and jq.

Here’s peertube-migrate-channel.bash:

#!/bin/bash

set -u
set -e

# Modify these for your setup
OLD_SERVER="INSERT OLD SERVER e.g. https://peertube.social"
OLD_CHANNEL="INSERT CHANNEL URL-NAME e.g. trials_fusion"
NEW_SERVER="INSERT NEW SERVER e.g. https://video.hardlimit.com"
NEW_CHANNEL="INSERT NEW CHANNEL ID e.g. 4103"
USERNAME="INSERT_USERNAME for new server e.g. trials"
PASSWORD="INSERT PASSWORD for new server"
TAG1="INSERT_A_TAG e.g trials"
TAG2="INSERT_A_TAG e.g. gaming"
TAG3="INSERT_A_TAG e.g. gameing"

DIR=$(mktemp -d)
API_PATH="${NEW_SERVER}/api/v1"

# Find out how many videos are in the channel
curl -s \
    "${OLD_SERVER}/api/v1/video-channels/${OLD_CHANNEL}/videos/?count=1" \
    > "${DIR}/videos-total.json"

TOTAL=$(jq .total < "${DIR}/videos-total.json")
CURRENT=0
PAGE_SIZE=10

# Get a list of URLS for all the videos

echo -n "" > "${DIR}/urls.txt"

while (( CURRENT < TOTAL )); do
    FILE="${DIR}/videos-page-${CURRENT}.json"

    curl -s \
        "${OLD_SERVER}/api/v1/video-channels/${OLD_CHANNEL}/videos/?start=${CURRENT}&count=${PAGE_SIZE}&skipCount=true" \
        > "${FILE}"

    jq ".data | map(.uuid) | .[]" -r < "${FILE}" >> "${DIR}/urls.txt"

    CURRENT=$((CURRENT + PAGE_SIZE))
done

# Log in to the new server

client_id=$(curl -s "${API_PATH}/oauth-clients/local" | jq -r ".client_id")
client_secret=$(curl -s "${API_PATH}/oauth-clients/local" | jq -r ".client_secret")
token=$(curl -s "${API_PATH}/users/token" \
  --data client_id="${client_id}" \
  --data client_secret="${client_secret}" \
  --data grant_type=password \
  --data response_type=code \
  --data username="${USERNAME}" \
  --data password="${PASSWORD}" \
  | jq -r ".access_token")

# Download and upload each video

tac "${DIR}/urls.txt" \
    | while read ID; do
        URL="${OLD_SERVER}/api/v1/videos/${ID}"
        curl -s "${URL}" > "${DIR}/info-${ID}.json"
        NAME=$(jq -r .name < "${DIR}/info-${ID}.json")
        CATEGORY=$(jq -r .category.id < "${DIR}/info-${ID}.json")
        LICENCE=$(jq -r .licence.id < "${DIR}/info-${ID}.json")
        LANGUAGE=$(jq -r .language.id < "${DIR}/info-${ID}.json")
        PRIVACY=$(jq -r .privacy.id < "${DIR}/info-${ID}.json")

        OLD_VIDEO="https://peertube.social/videos/watch/${ID}"
	mkdir "${DIR}/dl-${ID}"
	youtube-dl "${OLD_VIDEO}" --output="${DIR}/dl-${ID}/dl.mp4"

        echo "Uploading ${OLD_VIDEO}"
        curl "${API_PATH}/videos/upload" \
	  --silent \
          -H "Authorization: Bearer ${token}" \
          --output "${DIR}/curl-out-${ID}.txt" \
          --max-time 6000 \
	  --form name="${NAME}" \
	  --form videofile=@"${DIR}/dl-${ID}/dl.mp4" \
          --form channelId=${NEW_CHANNEL} \
          --form category=${CATEGORY} \
          --form licence=${LICENCE} \
          --form description="TODO VIDEO DESCRIPTION" \
          --form language=${LANGUAGE} \
          --form privacy=${PRIVACY} \
          --form tags="${TAG1}" \
          --form tags="${TAG2}" \
          --form tags="${TAG3}"
    done

Why write an entire game (including Graphics) in a single, hand-coded JavaScript file?

My new game, Rightwaves, is out now! It’s a tribute to the great classic R-Type.

A pixel-art spaceship battles aliens

The entire implementation, including all the graphics, is contained within a single ~5000-line JavaScript file. Why?

This is a terrible idea

Let me start by saying I do not recommend writing code this way. It’s a terrible idea, and some aspects of the development process have been severely hampered by doing this.

A pixellated spaceship dodges bullets and heads for a narrow passageway

Complexity

One of my goals for Smolpxl is to strip coding back to the simplest possible thing it can be.

Over the last 40 years, programming has changed a huge amount: the number and variety of the tools we have to work with has increased beyond what I can take in, and almost all of these things are utterly brilliant.

I can’t imagine writing a game like Eat Apples Quick using the line-orientated editor I used on the ZX Spectrum. I thoroughly enjoyed the Rust code completion and documentation I get from running rust-language-server inside NeoVim. (I’ve even heard there are editors that work inside a windowed environment, but I am not brave enough to try one.)

A pacman-like game

I can’t imagine writing the simple physics engine for Spring in assembly language, or trying to concentrate on that while I had to unravel complex graphics rendering conventions and timings. Instead, I was able to write code that looked a bit like the maths it represented, and allow the browser to handle the graphics card.

A spring with a cheery face bounces off a platform

But, along with all these powerful tools like high-level programming languages, fully cross-platform user interface components, and libraries that make writing a game loop simple, comes a new problem: complexity.

We must choose which tools, libraries and paradigms we’re going to use, and to make that choice we need to understand them.

The Smolpxl library makes some of these choices for you, by enforcing an Elm-style model/update/view split, providing a pixellated canvas of known size, and handling a game loop with fixed framerate. This is particularly suited to writing little retro-style games, similar to what I used to write in AMOS Basic on my Amiga.

I want to make everything simple, but I want our game to work on people’s computers. The only platform we can realistically choose is the browser: the exact same code works almost everywhere.

None of this explains why I would write a game inside a single file of source code, but hopefully it begins to demonstrate where I am coming from.

Simplicity

Another goal for Smolpxl is to teach programming, and a huge barrier I see to getting started is the build process.

No build

You can write Smolpxl games in Rust and compile them to WASM (which is how Eat Apples Quick is done), and you could also package the JavaScript with webpack or similar, but I want the first-class way of using Smolpxl to involve no build at all: I want you to be able to copy in a standard index.html and smolpxl.js and then write your code in game.js and have it just work when you double-click on index.html.

A goal for Rightwaves is to demonstrate that you can write a full game in Smolpxl, without stepping outside that simple story: just double-click on index.html.

It’s all inside game.js

So, Rightwaves is a single* JavaScript file containing the code, the level descriptions, and, probably most unusually, the graphics.

* Note: I cheated – the “action-replay” data is in a separate file – it was much bigger than the source code, and would have made it too hard to deal with. If I wanted to stay pure, I would have had to remove the default action replay from the game, but I just liked it too much.

Level design

Many times while I was writing Rightwaves, I wished I had written a level editor. In fact, I am often quoted as saying

“A game without a level editor is only half a game.”
— Andy Balaam, quite often

But instead, the level design is code like this:

const LEVELS = [
    {
        scenery: [
            { x:   0, y: 80, image: "machinery-20x16-01"},
            { x:  20, y: 88, image: "machinery-20x08-01"},
// ... etc.
        ],
        width: 1526,
        aliens: [
            newRedFlat(130, 20),
            newRedFlat(145, 25),
// ... etc.

The full code is at gitlab.com/smolpxl/rightwaves/-/blob/main/public/game.js#L3478.

It’s not a lot of fun to edit, but on the other hand, every time you make a change a simple refresh in the browser lets you see what it really looks like. No build process; no waiting.

Graphics

I drew the graphics for Rightwaves using GIMP, exported the images as PNG files, and converted them to text using a little Rust utility I wrote.

When I wanted to change an image, it was a nightmare, and I would recommend this approach to no-one whatsoever.

The reason why Smolpxl supports creating images in the source code is so you can hand-craft them right there, using an ASCII-art style.

Here’s the spaceship: An ASCII-art spaceship

and here is the code for it:

const IMAGES = {
    "ship": {
        pixels: [
            "..www.......",
            "dddddww.bbb.",
            "rllldddcccwb",
            "dddhlllccccb",
            "rddddddaccb.",
            "..aaa......."
        ],
        key: {
            "w": [255, 255, 255],
            "d": [88, 88, 88],
            "b": [77, 111, 249],
            "r": [141, 0, 0],
            "l": [126, 125, 125],
            "h": [192, 192, 192],
            "c": [3, 157, 157],
            "a": [42, 42, 42]
        }
    },
// ... lots, and I do mean lots, more here for the other graphics ...

The full code is at gitlab.com/smolpxl/rightwaves/-/blob/main/public/game.js#L41.

Note that I painstakingly coloured that code in for you, for this blog post. In the code there is no such luxury!

Openness

All the Smolpxl games are Free/Open Source software, of course. By avoiding a build, anyone can see the source code, just as I wrote, it in their browser. Anyone (including me) can debug problems without any extra steps. Putting everything in one file makes it easier to find the code (but probably makes it harder to understand).

Optimisation

Rightwaves loads really fast. On my machine, the first time you visit the page, it is playing within 1.4 seconds, after downloading 5 files at a total of 149kB (gzipped). If we exclude the included action replay, it is much smaller.

All of this, with no build process.

Maybe it wasn’t such a bad idea after all.

This is a bad idea

Don’t try this.

For fun

Except, of course, if you like fun. I tried this because it was fun. I would heartily recommend trying things because they might be fun.

Also, if you like fun, try playing the games on Smolpxl games, or write your own!

A puzzle game, a cross-the-road game, a tunnel game, a snake game, a Heli game, and a game-of-life thingy

Matrix is the only (chat) game in town

On my phone and computer I use WhatsApp, Signal, Slack, Keybase, Discord, IRC, XMPP/Jabber and Element/Matrix. In addition, I occasionally use the messaging features of Mastodon, Twitter and even LinkedIn. I’ve never used Telegram, Line, WeChat, Session, Wire or Status chat, but they exist too, along with many others.

It would be better if I could chat with people using the app I prefer, rather than the one I am forced to use.

Of course, the only useful chat app is the one your friends and family are on, so it’s pointless to debate the finer points in each service’s favour, but here I go anyway.

Only Matrix is:

The importance of decentralisation has been re-emphasised for me this week after the freenode IRC debacle. A single controlling entity, even when it is currently benign (as some people believe Signal is) is not a guarantee that things will stay this way. Thank goodness you can connect your usual IRC network to libera.chat: imagine what would happen to Signal users if they realised someone unscrupulous had acquired control.

Matrix does not solve all our problems. Notably:

  • Its security is probably not good enough for people threatened by powerful interests – at the moment it’s quite easy to see who’s talking to whom, and when.
  • Not all clients support end-to-end encryption, and not all turn it on by default (but the most-used ones do).

Despite these limitations, Matrix is the only chat network that is even attempting to provide what users need, and it seems to be doing a pretty good job of it.

I think we should work together to address its weaknesses, and adopt it wherever we can.

So, I recommend Matrix (specifically element.io) for group and individual chat.

Suspending the computer using Kupfer

I have recently started using Kupfer again as my application launcher in Ubuntu MATE, and I found it lacked the ability to suspend the computer.

Here is the plugin I wrote to support this.

To install it, quit Kupfer, create a directory in your home dir called .local/share/kupfer/plugins, and create this file suspend.py inside:

__kupfer_name__ = _("Power management")
__kupfer_sources__ = ("PowerManagementItemsSource", )
__description__ = _("Actions to suspend the computer")
__version__ = "2021-05-05"
__author__ = "Andy Balaam "


from kupfer.plugin import session_support as support


class Suspend (support.CommandLeaf):
    def __init__(self, commands):
        support.CommandLeaf.__init__(self, commands, "Suspend")
    def get_description(self):
        return _("Suspend the computer")
    def get_icon_name(self):
        return "system-suspend"


class PowerManagementItemsSource (support.CommonSource):
	def __init__(self):
		support.CommonSource.__init__(self, _("Power management"))
	def get_items(self):
		return (Suspend((["systemctl", "suspend"],)),)

# Copyright 2021 Andy Balaam, released under the MIT license.

Now restart Kupfer, go to Preferences, Plugins, and tick “Power management”.

You should now see a “Suspend” item if you search for it in the Kupfer interface.

Inspired by: Mate Session Management – Kupfer Plugin.

Reference docs: Kupfer Plugin API

Uploading to PeerTube from the command line

PeerTube’s API documentation gives an example of how to upload a video, but it is missing a couple of important aspects, most notably how to provide multiple tags use form-encoded input, so my more complete script is below. Use it like this:

# First, make sure jq is installed
echo "myusername" > username
echo "mypassword" > password
./upload "video_file.mp4"

Downsides:

  1. Your username and password are visible via ps to users on the same machine (tips to avoid this are welcome)
  2. I can’t work out how to include newlines in the video description (again, tips welcome)

You will need to edit the script to provide your own PeerTube server URL, channel ID (a number), video description, tags etc. Output and errors from the script will be placed in curl-out.txt. Read the API docs to see what numbers you need to use for category, license etc.

Here is upload:

#!/bin/bash

set -e
set -u

USERNAME="$(cat username)"
PASSWORD="$(cat password)"
FILE_PATH="$1"
CHANNEL_ID=MY_CHANNEL_ID_EG_1234
NAME="${FILE_PATH%.*}"
NAME="${NAME#*/}"

API_PATH="https://MY_PEERTUBE_SERVER_URL/api/v1"
## AUTH
client_id=$(curl -s "${API_PATH}/oauth-clients/local" | jq -r ".client_id")
client_secret=$(curl -s "${API_PATH}/oauth-clients/local" | jq -r ".client_secret")
token=$(curl -s "${API_PATH}/users/token" \
  --data client_id="${client_id}" \
  --data client_secret="${client_secret}" \
  --data grant_type=password \
  --data response_type=code \
  --data username="${USERNAME}" \
  --data password="${PASSWORD}" \
  | jq -r ".access_token")

echo "Uploading ${FILE_PATH}"
curl "${API_PATH}/videos/upload" \
  -H "Authorization: Bearer ${token}" \
  --output curl-out.txt \
  --max-time 6000 \
  --form videofile=@"${FILE_PATH}" \
  --form channelId=${CHANNEL_ID} \
  --form name="$NAME" \
  --form category=15 \
  --form licence=7 \
  --form description="MY_VIDEO_DESCRIPTION" \
  --form language=en \
  --form privacy=1 \
  --form tags="TAG1" \
  --form tags="TAG2" \
  --form tags="TAG3" \
  --form tags="TAG4"

Republishing Bartosz Milewski’s Category Theory lectures

Category Theory is an incredibly exciting and challenging area of Maths, that (among other things) can really help us understand what programming is on a fundamental level, and make us better programmers.

By far the best explanation of Category Theory that I have ever seen is a series of videos by Bartosz Milewski.

The videos on YouTube have quite a bit of background noise, and they were not available on PeerTube, so I asked for permission to edit and repost them, and Bartosz generously agreed! The conversation was in the comments section of Category Theory 1.1: Motivation and Philosophy and I reproduce it below.

So, I present these awesome videos, with background noise removed using Audacity, for your enjoyment:

Category Theory by Bartosz Milewski

Permission details:

Andy Balaam: Utterly brilliant lecture series.  Is it available under a free license?  I'd like to try and clean up audio and repost it to PeerTube, if that is permitted. Bartosz Milewski: You have my permission. I consider my lectures public domain.

Andy Balaam: Utterly brilliant lecture series. Is it available under a free license? I’d like to try and clean up audio and repost it to PeerTube, if that is permitted.
Bartosz Milewski: You have my permission. I consider my lectures public domain.

Announcing I-DUNNO 1.0 and web-i-dunno

It’s hard to believe it’s already a year since the release of RFC 8771 (The Internationalized Deliberately Unreadable Network NOtation), which for me at least made me think about IP addresses in a whole new way.

So, it seems fitting for the anniversary to be able to release proper support for this standard in the Rust universe, with Rust I-DUNNO version 1.0 released. You can find it on Rust’s crates.io at crates.io/crates/i-dunno and the API documentation is at docs.rs/i-dunno.

Also, because for a standard like this to receive the wide adoption it deserves, it’s important that young people have a chance to interact with it, playing with encodings to get a real feel for what it’s like to use in practice, I’m proud to announce the I-DUNNO Creator. On that page you can enter an IP address (IPv4 or IPv6) and see it transformed immediately into a candidate I-DUNNO, with live information about the Confusion Level of the I-DUNNO, as specified in the standard. You can find the source code for the I-DUNNO Creator in the web-i-dunno repo.

The I-DUNNO Creator is built on the Rust package, making use of Rust’s highly-developed WASM support to compile the code into a form that works naturally in a web browser.

I hope that by offering both systems programmers and the young people of today and their new-fangled web sites the opportunity to create I-DUNNOs, I can contribute a little to spreading the word about deliberately unreadable notations to new audiences.

Note: the current implementation is limited to generate only I-DUNNOs with no padding bits. As specified in the standard, I-DUNNOs may end with arbitrary padding, and adding this functionality to rust-i-dunno is left as an exercise for the reader: merge requests welcome!

Automatically filling in the UK COVID test results page with Selenium IDE

Lots of people are filling in the extremely detailed UK government COVID test result page twice every week.

It asks you to fill in a very large list of details, most of which are the same every time, but it doesn’t remember what you typed last time.

I didn’t want to write a Python script or similar to enter my results, because I wanted to check I’d done it right, and because there is a captcha at the end that is clearly intended to prevent automation like that.

However, with a Selenium IDE script, I can drive my browser, watching what it does and checking the input, and manually filling in the captcha and final double-check page.

In case it’s helpful, here is the script I created: report-covid-test.side.

You can create one for each child if you have several, filling in school name, NHS number, names, date of birth etc. in the script and re-using it, modifying it each time to enter the bar code number for the test itself.

To use it you’ll need the Selenium IDE plugin for firefox, or Selenium IDE plugin for another browser.

I’d recommend loading this script into the Selenium IDE plugin in Firefox, looking through it and editing the values that say “ENTER…HERE”, then clicking Run Script and watching it fill in values.

It doesn’t actually submit the result, so you can always check and modify it manually if something doesn’t work out, before clicking the last couple of buttons to submit.

Toggle window decorations on Linux GTK3 with Python3

The Internet is full of outdated Python code for doing things with windows, so here is what I got working today in a Python 3, GTK 3 environment.

This script toggles the window decorations on the active window on and off. I have it bound to Ctrl+NumPadMinus for easy access.

#!/usr/bin/env python3

import gi
gi.require_version('Gdk', '3.0')
gi.require_version('GdkX11', '3.0')
gi.require_version('Wnck', '3.0')
from gi.repository import Gdk
from gi.repository import GdkX11
from gi.repository import Wnck


def active_window(screen):
    for window in screen.get_windows():
       if window.is_active() == True:
            return window


def toggle_decorations(w):
    if w.get_decorations().decorations == 0:
        w.set_decorations(Gdk.WMDecoration.ALL)
    else:
        w.set_decorations(0)


screen = Wnck.Screen.get_default()
screen.force_update()
display = GdkX11.X11Display.get_default()
window = active_window(screen)
window_id = window.get_xid()

w = GdkX11.X11Window.foreign_new_for_display(display, window_id)
toggle_decorations(w)


window = None
screen = None
Wnck.shutdown()

Questions about RFC 8771

During my work on RFC 8771 The Internationalized Deliberately Unreadable Network NOtation (I-DUNNO) I have come across a number of questions. I am documenting them here so I can send them to the authors and try to improve my understanding of the intention.

This is an excellent RFC, and I thank the authors for their efforts in creating it.

1. Non-printable characters

In 4.2. Satisfactory Confusion Level, the RFC states that encodings may be deemed Satisfactory if they contain ‘At least one non-printable character’ (as well as one other condition in this section).

Both of the existing implementations that I know of ( and ) interpret “printable” to mean the same as Python’s isprintable() function: that is Nonprintable characters are those characters defined in the Unicode character database as “Other” or “Separator”, excepting the ASCII space (0x20) which is considered printable.

However, the definition of this function may be rather Python-specific, since its intention appears to be related to language internals like the repr function.

It would be helpful to find out exactly what is meant by “non-printable character” in the RFC.

2. Are Modifier Symbols, Symbols?

Also in section 4.2, the RFC mentions ‘”A character from the “Symbol” category’.

The Python implementation excludes Modifier Symbols from its definition. I believe this is incorrect, and have logged an issue on the topic: Some Symbol characters not recognised as such.

It would be helpful to have clarification on this point.

3. What does “different directionalities” mean?

Unicode classifies characters into several Bidi_Classes (for example, U+CED6E is Left_To_Right). In section 4.3. Delightful Confusion Level, the RFC refers to ‘Characters from scripts with different directionalities’.

As far as I can see there are two possible interpretations of this phrase:

  1. The encoding should contain characters from at least two different Bidi_Classes, or
  2. The encoding should contain characters that are both left-to-right and right-to-left in direction, either weakly or strongly.

Both current implementations interpret this statement like number 1, but I suspect the intention may actually be something more like number 2.

If number 2 was meant, I think that would mean ignoring characters with Neutral directionality, and treating weakly directional characters as the same directionality as strongly directional ones.

4. What is a Confusable character?

Section 4.3 mentions ‘Character classified as “Confusables”‘. Both implementations interpret this quite loosely, as meaning something like ‘the encoding contains any character or substring which might be confused with any other character or substring’.

This means a lot of “normal” characters are included: all of the ASCII digits, and many of the Latin letters.

Was this is intention?

That’s all my questions. It has been great fun working on this RFC.

Announcing Rust I-DUNNO

At the ACCU Conference last week I learned about RFC 8771 The Internationalized Deliberately Unreadable Network NOtation (I-DUNNO) from Jim Hague, and thought it would be fun to knock up a Rust implementation.

The project is here: gitlab.com/andybalaam/rust-i-dunno and the docs are published at https://docs.rs/i-dunno.

It’s not done yet, but encoding an IP address as I-DUNNO appears to be working:

$ i-dunno 216.58.205.46
lYԮ

$ i-dunno 216.58.205.46 | hexdump -C
00000000  db 81 6b 1a 2e 0a                                 |..k...|

Decoding is still to be done.

The implementation is seriously slow at the moment, so I am looking forward to improving it.

I am hoping it is reasonably correct – I based it on the existing Python I-DUNNO implementation and in the process found several potential bugs in that, and created some merge requests to fix bugs and help with testability.

Speaking of testability, I am building up a collection of test cases that could be a potential resource for other implementors, and would welcome suggestions of how this could be shared between projects. The examples so far were generated using the Python implementation, and then manually corrected where I found bugs in that, so I do not have 100% confidence that they are correct.

Anyway, have a play, and send patches and feedback!

Letter to my MP on Yemen

Sent via WriteToThem on Monday 1 March 2021.

Dear Ben Spencer,

Your experience on the front line in medicine means you are probably much more able than I to imagine the human impact of the situation in Yemen.

On Sunday OCHA stated that 16 million people will go hungry this year, and nearly 50,000 are already living in famine-like conditions.

How can the UK government throw away its position of leadership on this matter by cutting funding at such a desperate time? I share the urgent concern expressed by Andrew Mitchell this morning.

I would be grateful if you would pass my concerns on to the FCDO.

Yours sincerely,

Andy Balaam

References:

Making Smolpxl work on phones and tablets

I’ve added the first features intended to make Smolpxl games work well on touch interfaces like phones and tablets:

Spring game with touch controls

I’ve added a button bar at the bottom (and moved the navigation buttons to the top).

I’m looking for feedback on this:

  • Does it work on your device?
  • Are the buttons the right size?
  • Do they look ok? If not, how could they look better?
  • For games that require arrow keys, do you need them in the normal arrow-keys layout, or is a simple row fine?

Duckmaze game with touch controls in a single row

If you’re writing a game and you want to add buttons like this, you just need to add a single line like this:

game.showControls(["MENU", "SELECT", "BUTTON1", "BUTTON2"]);

or this:

game.showControls(["MENU", "SELECT", "LEFT", "DOWN", "UP", "RIGHT"]);

and they should appear.

Limiting the number of open sockets in a tokio-based TCP listener

I learned quite a bit today about how to think about concurrency in Rust. I was trying to use a Semaphore to limit how many open sockets my TCP listener allowed, and I had real trouble making it work. It either didn’t actually work, allowing any number of clients to connect, or the compiler told me I couldn’t do what I wanted to do, because the lifetime of my Semaphore was not 'static. Here’s the journey I took towards working code that I think is correct (feedback welcome).

Motivation

In the tokio tutorial there is a short section entitled “Backpressure and bounded channels” (partway down the Channels page). It contains this statement:

…take care to ensure total amount of concurrency is bounded. For example, when writing a TCP accept loop, ensure that the total number of open sockets is bounded.

Obviously, when I started work on a TCP accept loop, I wanted to follow this advice.

Like many things in my journey with Rust, it was harder than I expected, and eventually enlightening.

The code

Here is a short Rust program that listens on a TCP port and accepts incoming connections.

Cargo.toml:

[package]
name = "tcp-listener-example"
version = "1.0.0"
edition = "2018"
include = ["src/"]

[dependencies]
tokio = { version = ">=1.0.1", features = ["full"] }

src/main.rs:

use tokio::io::AsyncReadExt;
use tokio::net::TcpListener;

#[tokio::main]
async fn main() {
    let listener = TcpListener::bind("0.0.0.0:8080").await.unwrap();

    loop {
        let (mut tcp_stream, _) = listener.accept().await.unwrap();
        tokio::spawn(async move {
            let mut buf: [u8; 1024] = [0; 1024];
            loop {
                let n = tcp_stream.read(&mut buf).await.unwrap();
                if n == 0 {
                    return;
                }
                print!("{}", String::from_utf8_lossy(&buf[0..n]));
            }
        });
    }
}

This program listens on port 8080, and every time a client connects, it spawns an asynchronous task to deal with it.

If I run it with:

cargo run

It starts, and I can connect to it from multiple other processes like this:

telnet 0.0.0.0 8080

Anything I type into the telnet terminal window gets printed out in the terminal where I ran cargo run. The program works: it listens on TCP port 8080 and prints out all the messages it receives.

So what’s the problem?

The problem is that this program can be overwhelmed: if lots of processes connect to it, it will accept all the connections, and eventually run out of sockets. This might prevent other things working right on the computer, or it might crash our program, or something else. We need some kind of sensible limit, as the tokio tutorial mentions.

So how do we limit the number of people allowed to connect at the same time?

Just use a semaphore, dummy

A semaphore does exactly what we need here – it keeps a count of how many people are doing something, and prevents that number getting too big. So all we need to do is restrict the number of clients that we allow to connect using a semaphore.

Here was my first attempt:

use tokio::io::AsyncReadExt;
use tokio::net::TcpListener;
use tokio::sync::Semaphore;

#[tokio::main]
async fn main() {
    let listener = TcpListener::bind("0.0.0.0:8080").await.unwrap();
    let sem = Semaphore::new(2);

    loop {
        let (mut tcp_stream, _) = listener.accept().await.unwrap();
        // Don't copy this code: it doesn't work
        let aq = sem.try_acquire();
        if let Ok(_guard) = aq {
            tokio::spawn(async move {
                let mut buf: [u8; 1024] = [0; 1024];
                loop {
                    let n = tcp_stream.read(&mut buf).await.unwrap();
                    if n == 0 {
                        return;
                    }
                    print!("{}", String::from_utf8_lossy(&buf[0..n]));
                }
            });
        } else {
            println!("Rejecting client: too many open sockets");
        }
    }
}

This compiles fine, but it doesn’t do anything! Even though we called Semaphore::new with an argument of 2, intending to allow only 2 clients to connect, in fact I can still connect more times than that. It looks like our code changes had no effect at all.

What we were hoping to happen was that every time a client connected, we created _guard, which is a SemaphoreGuard, that occupies one of the slots in the semaphore. We were expecting that guard to live until the client disconnects, at which point the slot will be released.

Why doesn’t it work? It’s easy to understand when you think about what tokio::spawn does. It creates a task and asks for it to be executed in the future, but it doesn’t actually run it. So tokio::spawn returns immediately, and _guard is dropped, before the code that handles the request is executed. So, obviously, our change doesn’t actually restrict how many requests are being handled because the semaphore slot is freed up before the request is processed.

Just hold the guard for longer, dummy

So, let’s hold on to the SemaphoreGuard for longer:

use tokio::io::AsyncReadExt;
use tokio::net::TcpListener;
use tokio::sync::Semaphore;

#[tokio::main]
async fn main() {
    let listener = TcpListener::bind("0.0.0.0:8080").await.unwrap();
    let sem = Semaphore::new(2);

    loop {
        let (mut tcp_stream, _) = listener.accept().await.unwrap();
        let aq = sem.try_acquire();
        if let Ok(guard) = aq {
            tokio::spawn(async move {
                let mut buf: [u8; 1024] = [0; 1024];
                loop {
                    let n = tcp_stream.read(&mut buf).await.unwrap();
                    if n == 0 {
                        drop(guard);
                        return;
                    }
                    print!("{}", String::from_utf8_lossy(&buf[0..n]));
                }
            });
        } else {
            println!("Rejecting client: too many open sockets");
        }
    }
}

The idea is to pass the SemaphoreGuard object into the code that actually deals with the client request. The way I’ve attempted that is by referring to guard somewhere within the async move closure. What I’ve actually done is tell it to drop guard when we are finished with the request, but actually any mention of that variable within the closure would have been enough to tell the compiler we want to move it in, and only drop it when we are done.

It all sounds reasonable, but actually this code doesn’t compile. Here’s the error I get:

error[E0597]: `sem` does not live long enough
  --> src/main.rs:12:18
   |
12 |         let aq = sem.try_acquire();
   |                  ^^^--------------
   |                  |
   |                  borrowed value does not live long enough
   |                  argument requires that `sem` is borrowed for `'static`
...
29 | }
   | - `sem` dropped here while still borrowed

What the compiler is saying is that our SemaphoreGuard is referring to sem (the Semaphore object), but that the guard might live longer than the semaphore.

Why? Surely sem is held within a scope that includes the whole of the client-handling code, so it should live long enough?

No. Actually, the async move closure that we are passing to tokio::spawn is being added to a list of tasks to run in the future, so it could live much longer. The fact that we are inside an infinite loop confused me further here, but the principle still remains: whenever we make a closure like this and pass something into it, the closure must own it, or if we are borrowing it, it must live forever (which is what a 'static lifetime means).

The code above passes ownership of guard to the closure, but guard itself is referring to (borrowing) sem. This is why the compiler says that “sem is borrowed for 'static“.

Wrong things I tried

Because I didn’t understand what I was doing, I tried various other things like making sem an Arc, making guard an Arc, creating guard inside the closure, and even trying to make sem actually have 'static storage by making it a constant. (That last one didn’t work because only very simple types like numbers and strings can be constants.)

Solution: Share the Semaphore in an Arc

After what felt like too much thrashing around, I found what I think is the right answer:

use std::sync::Arc;
use tokio::io::AsyncReadExt;
use tokio::net::TcpListener;
use tokio::sync::Semaphore;

#[tokio::main]
async fn main() {
    let listener = TcpListener::bind("0.0.0.0:8080").await.unwrap();
    let sem = Arc::new(Semaphore::new(2));

    loop {
        let (mut tcp_stream, _) = listener.accept().await.unwrap();
        let sem_clone = Arc::clone(&sem);
        tokio::spawn(async move {
            let aq = sem_clone.try_acquire();
            if let Ok(_guard) = aq {
                let mut buf: [u8; 1024] = [0; 1024];
                loop {
                    let n = tcp_stream.read(&mut buf).await.unwrap();
                    if n == 0 {
                        return;
                    }
                    print!("{}", String::from_utf8_lossy(&buf[0..n]));
                }
            } else {
                println!("Rejecting client: too many open sockets");
            }
        });
    }
}

This code:

  • Creates a Semaphore and stores it inside an Arc, which is a reference-counting pointer that can be shared between tasks. This means it will live as long as someone holds a reference to it.
  • Clones the Arc so we have a copy that can be safely moved into the async move closure. We can’t move sem in to the closure because it’s going to get used again the next time around the loop. We can move sem_clone in to the closure because it’s not used anywhere else. sem and sem_clone both refer to the same Semaphore object, so they agree on the count of clients that are connected, but they are different Arc instances, so one can be moved into the closure.
  • Only aquires the SemaphoreGuard once we’re inside the closure. This way we’re not doing something difficult like borrowing a reference to something that lives outside the closure. Instead, we’re borrowing a reference via sem_clone, which is owned by the closure which we are inside, so we know it will live long enough.

It actually works! After two clients are connected, listener.accept actually opens a socket to any new client, but because we return almost immediately from the closure, we only hold it open very briefly before dropping it. This seemed preferable to refusing to open it at all, which I thought would probably leave clients hanging, waiting for a connection that might never come.

Lifetimes are cool, and tricky

Once again, I have learned a lot about what my code is really doing from the Rust compiler. I find this stuff really confusing, but hopefully by writing down my understanding in this post I have helped my current and future selves, and maybe even you, be clearer about how to share a semaphore between multiple asynchronous tasks.

It’s really fun and empowering to write code that I am reasonably confident is correct, and also works. The sense that “the compiler has my back” is strong, and I like it.

Recommendation against the use of WhatsApp in your company

Here is the email I just sent to the organisation I volunteer for. Feel free to adapt and use in your context.

Dear [organisation leaders],

Much of the tech industry (e.g. [1]) is warning against the use of WhatsApp due to its policy of collecting and sharing user information with third parties and the poor track record of its parent company (Facebook) on ethical issues (see examples [2] and [3], and many more).

The situation was made considerably worse with a recent change to the WhatsApp terms and conditions [4].

So, as your IT person I recommend not using WhatsApp for our work.

We already have an alternative available, and I would be really happy to help anyone who needs help setting it up.

[Details here of the alternative we use (Zulip) and how to use it. The simplest alternative to recommend is Signal.]

Thanks, Andy

[1] What Facebook and WhatsApp’s Data Sharing Plans Really Mean for User Privacy

[2] Facebook experimented with modifying people’s moods

[3] Facebook paid teens for total access to their phone activity

[4] If you’re a WhatsApp user, you’ll have to share your personal data

Streaming video with Owncast on a free Oracle Cloud computer

I just streamed about 40 minutes of me playing Trials Fusion using Owncast. Owncast is a self-hosted alternative to streaming services like Twitch and YouTube live.

Normally, you would need to pay for a computer to self-host it on. Owncast suggest this will cost about $5/month.

But, Oracle Cloud has a “Always Free” tier that includes a “Compute Instance” (a virtual machine running Linux) that is capable of running Owncast.

Here’s how I did it:

Register for Oracle Cloud

This was probably the worst bit.

I went to oraclecloud.com and clicked “Sign up for free cloud tier”. It didn’t work in Firefox(!) so I had to use Chromium.

I had to enter my name, address, email address, phone number and credit card details. The email was verified, the phone number was verified (with a text message), and the credit card was verified (with a real transaction), so there was no getting around any of it.

They promise that they won’t charge my card. I’ll let you know if I discover differently.

Create a Compute Instance

Once I was logged in to the Oracle “console” (web site), I clicked the burger menu in the top left, chose “Compute” and then “Instances” to create a new instance. I followed all the default settings (including using the default “image”, which meant my instance was running Oracle Linux, which I think is similar to Red Hat), and when I got to the ssh keys part, I supplied the public key of my existing SSH key pair. Read the docs there if you don’t have one of these.

As soon as that was done, and I waited for the instance to be created and started, I was able to SSH in to my instance using a username of opc and the Public IP Address listed:

ssh opc@PUBLIC_IP

(Note: here and below, if I say “PUBLIC_IP”, I mean the IP address listed in the information about your compute instance. It should be a list of four numbers separated by dots.)

Allow connecting to the instance on different ports

Owncast listens for HTTP connections on port 8080, and RTMP streams on 1935, so I needed to do two things to make that work.

Modify the Security List to add Ingress Rules

  • On the information about my instance, I clicked on the name of the Subnet (under Primary VNIC).
  • In the subnet, I clicked the name of the Security List (“Default Security List for …”) in the Security Lists list.
  • In the Security List I clicked Add Ingress Rules and entered:
    Stateless: unchecked
    Source Type: CIDR
    Source CIDR: 0.0.0.0/0
    IP Protocol: TCP
    Source Port Range: (blank)
    Destination Port Range: 8080
    Description: (blank)

    and then clicked Add Ingress Rules to create the rule.

  • I then added another Ingress Rule that was identical, except Destination Port Range was 1935.

Allow ports 8080 and 1935 on the instance’s own firewall

It took me a long time to figure out, but it turns out the Oracle Linux running on the Compute Instance has its own firewall. Eventually, thanks to a blog post by meinside: When Oracle Cloud’s Ubuntu instance doesn’t accept connections to ports other than 22, and some Oracle docs on ways to secure resources, I found that I needed to SSH in to the machine (like I showed above) and run these commands:

sudo firewall-cmd --zone=public --permanent --add-port=8080/tcp
sudo firewall-cmd --zone=public --permanent --add-port=1935/tcp
sudo firewall-cmd --reload

Now I was able to connect to the services I ran on the machine on those ports.

Install Owncast

The Owncast install was incredibly easy. I just followed the instructions at Owncast Quickstart. I SSHd in to the instance as before, and ran:

curl -s https://owncast.online/install.sh | bash

and then edited the file owncast/config.yaml to have a custom stream key in it. You can do that by typing:

nano owncast/config.yaml

There is information about this file at: owncast.online/docs/configuration.

Run Owncast

I ran the service like this:

cd owncast
./owncast

In future, if I want to leave it running, I may run it inside screen, or even use systemd or similar.

Open the web site

I could now see the web site by typing this into my browser’s address bar:

http://PUBLIC_IP:8080

(Where PUBLIC_IP is the Public IP copied from the Instance info as before.)

Stream some video

Finally, in OBS‘s Settings I chose the Stream section and entered:

Service: Custom...
Server: rtmp://PUBLIC_IP/live
Stream key: STREAM_KEY

Where “STREAM_KEY” means the stream key I added to config.yaml earlier.

Now, when I clicked “Start Streaming” in OBS, my stream appeared on the web site!

Costs and limits

Oracle stated during sign-up that I would not be charged unless I explicitly chose to use a different tier.

The Compute Instance is part of the “Always Free” tier, so in theory it should stay up and working.

However, if you use lots of resources (which streaming for a long time probably does), I would expect services would be throttled and/or stopped completely. I have no idea whether they will allow enough resources for regular streaming, or whether this is all waste of time. We shall see.

Pinephone update

I got a Pinephone for Christmas!

Here is quick summary of my experience with it. (Originally published on mastodon.)


Update on the pinephone as promised.

I love it, but I would definitely not recommend expecting to use it as your actual phone.

I have the Manjaro Phosh edition. Phosh is GNOME customised for mobile.

It turns on, you can unlock it, and you get a launcher. It has apps, and some of them work.

Firefox works really well. I can use it for Youtube and loads of other sites. I installed uBlock Origin, and it works.

Adding my Nextcloud config to Phosh seamlessly gave me Calendar, Contact and TODO list apps working, with my data in them.

The Maps app found me easily via GPS. I could bring up directions by entering a from and to, but it didn't seem to want to guide me via GPS.

Several apps don't fit properly on screen, and there doesn't seem to be a way to scroll or move the windows.

The camera technically works but the picture looks terrible (squashed, wibbly and blue-coloured).

Scrolling around on the launcher updates at about 5-10 fps, which is fine but would put many people off.

Many of the apps available to install in the Software app don't really work. I assume the list of apps is the standard for GNOME or Manjaro, so many are not adapted for phones.

I _love_ the fact that all the work that has been put into desktop Linux can be re-used on phones. Why wasn't it always this way?

It's great to be able to buy hardware that is specifically designed to run properly free software.

The Terminal app works nicely and presents a keyboard with extra keys that you need in a terminal.

The settings app works nicely.

My biggest frustration was not being able to find software in the Software app that worked nicely.

I was looking for a Youtube app that protected my privacy. On Android I use NewPipe Legacy. On desktop I use Freetube. I couldn't find Freetube in Software. I tried Minitube but it was unusable (window didn't fit).

I haven't tried installing software from the command line. Maybe I can find (or build) Freetube via a Manjaro repo?

Or maybe I should investigate NewPipe Legacy via anbox, although that seems to miss the point a little :-)

Is your program a function or a service?

Maybe everyone knows this already, but for my own clarity, I think there are really two types of computer program:

  • A function: something that you run, and get back a result. Example: a command-line tool like ls
  • A service: something that sits around waiting for things to happen, and responds to them. Example: a web server

How functions work

Programs that are essentially functions should:

  • Validate their input and stop if it is wrong
  • Stop when they have finished their job*
  • Let you know whether they succeeded or failed

*The Halting Problem shows that you can’t prove they stop, so I won’t ask you to do that.

Writing functions is relatively easy.

How services work

Programs that are services should:

  • Start when you tell them to start, even when things are not right
  • Keep running until you tell them to stop, even when bad things happen
  • Tell the user about problems via some communication mechanism

Writing services seems a little harder than writing functions.

What about UIs?

I suggest that programs with UIs are just a special case of services. Do you agree?

What about let-it-crash?

I think that let-it-crash is a good way to build services, but when you build a service that way, I consider the whole system to be the real service: this means the code we are writing, plus the runtime. In this case, the runtime is responsible for keeping the service running (by restarting it), and telling the user about problems.

In effect, let-it-crash allows us to write programs that look like functions (which I claim is easier), and still have them behave like services, because the runtime does the extra work for us. Erlang seems like a good example of this.

What are the implications?

If you are writing a service, your program should start when asked, and keep running until it is asked to stop, even if things are bad.

For example:

  • a service that relies on a data source should keep running when that data source is unavailable, and emit errors saying that it is unable to work. It should start working when the data source becomes available. (Again, if you implement this behaviour by using a runtime that allows you to write in a let-it-crash style, good for you.)
  • a service that relies on the existence of a directory should probably create that directory if it doesn’t exist.
  • a service that needs config might want to start up with sane defaults if the config is not supplied. Or maybe it should complain loudly and poll for the file to be created?

Why not stop when things are wrong?

  • Using this approach, it doesn’t matter the order of starting services. The more services we have, the more painful it is to have an order we must follow.
  • It’s nice when things are predictable. We expect services to keep running under normal circumstances. Using this approach, our expectations are not wrong when things go wrong.

What are the down sides?

  • You must pay attention to the error reporting coming from running services – they may not be working.
  • Services will still stop, due to bugs, or at least due to hardware failures, so you still have to pay attention to whether services are running.

More: 12 Fractured Apps

Shutdown order consistency: how Rust helps

Some Java code with bugs

Here’s my main method (in Java). Can you guess the bug?

Db db = new Db();
Monitoring monitoring = new Monitoring();
Monitoring mon2 = new Monitoring();
Billing billing = new Billing(db, monitoring);
monitoring.setDb(db);

runMainLoop(billing, mon2);

db.stop();
billing.stop();
monitoring.stop();

If you would like to hunt down the 2 bugs manually, try reading the full code here: ShutdownOrder.java

But maybe you have an idea already? Maybe you’ve seen code like this before? If you have, you probably have an instinct that there’s some kind of bug, even if you can’t say for sure what it is. Code like this almost always has bugs!

This code compiles fine, but it contains two bugs.

First, we forgot to setDb() on mon2. This causes a NullPointerException, because Monitoring expects always to have a working Db.

Second, and in general harder to spot, we shut down our services in the wrong order. It turns out that Monitoring uses its Db during shutdown, so we get an exception. Even worse, if some other code needed to run after monitoring.stop(), it won’t, because the exception prevents us getting any further.

Of course, this is toy code, but this kind of problem is common (and much harder to spot) in real-life code. In fact, my team dealt with a similar bug this week.

It’s fundamentally hard to figure out your shutdown order. It’s complicated further if classes have start() methods too, which I have seen in lots of Java code.

Given that this is just a hard problem, maybe there’s no point looking for tools to make it easier?

Some Rust code without those bugs

Let’s try writing this code in Rust. Here’s the main method:

let db = Db::new();
let monitoring = Monitoring::new(&db);
let mon2 = Monitoring::new(&db);
let billing = Billing::new(&db, &monitoring);

run_main_loop(&billing, &mon2);

// drop() is called automatically on all objects here

Here’s the full code: shutdown_order.rs

This code shuts down all the services automatically at the end, and any mistakes we make in the order are compile errors, not things we find later when our code is running.

The code to shut down each service looks like this:

impl Drop for Monitoring<'_> {
    fn drop(&mut self) {
        // [Disconnect from monitoring API]
        self.db.add_record("MonitorShutDown");
    }
}

This is us implementing the Drop trait for the struct Monitoring (traits are a bit like Java Interfaces). The Drop trait is special: it indicates what to do when an instance of this struct is dropped. In Rust, this is guaranteed to happen when the instance goes out of scope, which is why our comment at the end of the main method sounds so confident.

Furthermore, Rust’s compiler shuts down everything in the reverse order in which it was created, and guarantees that nothing gets used after it has been dropped.

Rust’s lovely world gives us two relevant treats: no unexpected nulls, and lifetimes.

Treat number 1: no unexpected nulls

First, in Rust, like in other modern languages like Kotlin, we have to be explicit about items that could be missing. In our example, we were able to re-arrange the code so that db can never be missing (or null), and the compiler encouraged us to do so. If we really needed it to be missing some of the time, we could have used the Option type, and the compiler would have forced us to handle the case when it was missing, instead of unexpectedly getting a NullPointerException like we did in Java. (In fact, if we’d structured our code to use final in as many places as possible, we could have been encouraged towards basically the same solution in Java too.)

Treat number 2: lifetimes

Second, if you look a bit more closely at the full code of shutdown_order.rs you’ll see lots of confusing-looking annotations like <'a> and &'a:

struct Monitoring<'a> {
    db: &'a Db,
}

The approximate meaning of those annotations is: a Monitoring holds a reference to a Db, and that Db must last longer than the Monitoring.

This “lasts longer than” wording is what Rust Lifetimes are for. Lifetimes are a way of saying how long something lasts.

Lifetimes are really confusing when you start with Rust, and have caused me a lot of pain. Code like this is where they are both most painful and most helpful. As I mentioned earlier, the problem of shutdown order is fundamentally hard. Rust gives you that pain at the beginning, and until you understand what’s going on, the pain is very confusing and acute. But, once your code compiles, it is correct, at least as far as problems like this are concerned.

I love the sense of security it gives me to write Rust code and know the compiler has checked my code for this kind of problem, meaning it can’t crop up at 3am on Christmas Day…

Final note/caveat

This Rust code is probably over-simplified, because all the references are immutable (you can’t change the objects they point to). In practice, we may well have mutable references, and if we do we’re going have to deal with the further difficulty that Rust won’t allow two different objects to hold references to an object if any of those references are mutable. So it would object to Billing and Monitoring using the Db object at the same time. We’d need to make it immutable (as we have here), or find a different way of structuring the code: for example, we could hold the Db instance only within the run_main_loop code, and pass it in temporarily to the Billing and Monitoring objects when we called their methods. A large part of the art, fun and pain of learning Rust is finding new patterns for your code that do what you need to do and also keep the compiler happy. When you manage it, you get amazing benefits!

Edge computing providers

I’m looking into Edge computing at work. By Edge computing I mean running WASM programs in lots and lots of smallish computers in places near to actual people (rather than in huge cloud data centres). I think it’s cool because I love Rust, and Rust is the leading language to compile to WASM.

Here are some companies providing Edge computing services:

  • Fastly – good links with WASM community (hired Mozilla devs), and early adopters – custom WASM engine wasmtime.
  • Cloudflare – huge, and early adopters – WASM engine is Google V8.
  • AWS Lambda@Edge – docs are light on detail, but it looks like a real offering, probably.

Also-rans:

Who did I miss?

Schema upgrades should be reversible (also other transformations, actually)

Are you writing schema upgrade code? Then I humbly suggest you take the time to write schema downgrade code too.

“Why would I do that?” you might well ask, “I won’t ever need to downgrade.”

Now, I imagine you’re expecting me to say you actually will need to downgrade, but that isn’t what I’m saying.

Can you please get on with what you are actually saying?

Whevener you write code to transform something, be it a schema upgrade, some serialisation, or something else, I would highly recommend that you write code to transform it in both directions.

Reasons:

  • It makes testing easier. The best kinds of tests for things like this are round-trips, where you transform something in both directions and check it hasn’t changed. It’s really hard to mess up tests like that.
  • It often uncovers bugs, because it enforces clear thinking about what the transformation actually means.
  • It may improve your code, because it gets annoying writing similar-but-different code to transform in both directions, so you are pushed towards some kind of abstraction.

Also:

  • You almost certainly are going to need it. Sometimes things go wrong and you need to back up.
  • It will be incredibly useful for testing other parts of your code.

Bidirectional scheme up/downgrades are not easy in SQL, but probably worth it. If you’re writing transformation code in a normal programming language, it’s really not that difficult, and I predict it will be worth it.

Announcing Smolpxl Scores – a high score table for your game

It’s a very early beta for now, but I’m ready to announce Smolpxl Scores, which provides high-score tables for Free and Open Source games.

Each game can have multiple high-score tables – for example, you might want one for each level.

At the moment it’s deployed in my own web hosting and therefore written using the technologies that are most convenient for me to deploy there, which is PHP+MySQL. If it becomes more widely used and the performance suffers I guess I’ll ask for donations to host it somewhere else, and use more fashionable technologies.

To add a score you make a POST request like this:

curl https://scores.artificialworlds.net/api/v1/myappname/mytablename/ -d \
    '{"appId":"myappid","name":"Megan Tria", "score": 13.5, "notes": ""}'

and to look at some existing scores you can request them by pages:

curl 'https://scores.artificialworlds.net/api/v1/myappname/mytablename/?startRank=11&num=20'

or by name:

curl 'https://scores.artificialworlds.net/api/v1/myappname/mytablename/?startName=David%20Lloyd%20Geo&offset=-5&num=10'

The results are ordered by players’ scores, and are provided as JSON.

Each table stores only one score per player.

Of course, the API will evolve over time, but I hope that what I have now will be good enough to support some real-life games, and provide enough feedback to make it better.

As soon as people are actually using it, I will ensure the current API version (v1) remains stable, and release any incompatible updates as later versions.

If you’d like to use Smolpxl Scores to add a high-score table to your game, please create an issue at gitlab.com/smolpxl/smolpxl-scores/-/issues.

This service is only available to Free and Open Source games. Also, if someone abuses it (accidentally or on purpose) I will talk to them, and may eventually have to remove their access if we can’t fix the problem.

Dovecot not working after upgrade to Ubuntu 20.04.1 (dh key too small)

I upgraded to Ubuntu 20.04.1 and chose to keep my existing config files, and my mail server stopped working. In the log I saw:

Nov 25 09:07:57 machine dovecot: imap-login: Error: Failed to initialize SSL server context: Can't load DH parameters: error:1408518A:SSL routines:ssl3_ctx_ctrl:dh key too small: user=<>, rip=someip, lip=someip, session=<someid>

I was able to fix this by modifying /etc/dovecot/conf.d/10-ssl.conf and adding this line:

ssl_dh = </usr/share/dovecot/dh.pem

Please let me know if I’ve introduced an horrific security bug, won’t you?

Letter to my MP on the overseas aid budget.

Letter I sent to my MP today on the overseas aid budget. Let’s not be foolish.

Dear Ben Spencer,

Please use your influence to persuade the government to maintain our overseas aid budget commitment at 0.7% of national income.
I believe that changing this policy would be a mistake, increasing the risks of extremism and forced migration around the world.

The policy was established when the budget was very tight, and I think the reasons for it remain compelling: to prevent selfishness and short-termism from hurting our own and others’ interests.

Please pass my letter on to the Chancellor.

Yours sincerely,
Andy Balaam

Feel free to re-use some or all of this.

To write to your MP, try writetothem.com.

Profile a Java unit test (very quickly, with no external tools)

I have a unit test that is running slowly, and I want a quick view of what is happening.

I can get a nice overview of where the code spends its time by adding this to the JVM arguments:

-agentlib:hprof=cpu=samples,lineno=y,depth=3,file=hprof.samples.txt

and running the test as normal.

Now I can look at the file that was created, hprof.samples.txt, and looking at the bottom section I can see how much time is spent in each method.

This worked for me within IntelliJ IDEA community edition by clicking “Run” then “Edit Configurations” and adding the above code to “VM options” for my test.

It should also work in Gradle by editing gradle.properties and adding something like this:

org.gradle.jvmargs=-agentlib:hprof=cpu=samples,lineno=y,depth=3,file=hprof.samples.txt

and should also work in Maven. In fact, I found this information in this stackoverflow question: How do you run maven unit tests with hprof?.

Why a Free Software web games site?

Recently I’ve been having a lot of fun working on Smolpxl, which is a web site featuring some little retro web games that are all Free and Open Source Software.

Here’s a sneak preview of the game I am working on:

A pixellated spaceship avoids some walls, then crashes into them

Why do this?

Apart from the fact that it’s fun, I also think there is a need for a site like this: a safe place for kids to play little games without creepy advertising looking over their shoulder, and perverse incentives for the site creators.

Little web games can be a diversion during train journeys, helpful distractions for parents and teachers to provide for kids, and even be a little educational around mouse and keyboard use. I’ve seen the sites that already exist be helpful in all those contexts, but I’ve always felt uncomfortable that these sites are supported by advertising, which always comes with concerns about privacy, and also leads game creators to focus on “engagement”, creating mechanisms like site-wide currencies and gambling-style rewards that drive addictive behaviours.

Wouldn’t it be nice if we in the Free and Open Source community could write some fun games that are free from those unhealthy influences?

Wouldn’t it be even nicer if we took the opportunity to encourage kids to learn how to make games as well as play them?

Well, that’s the idea. Have a look at smolpxl.artificialworlds.net, play a few games, and think about writing a few more…

Also, if you know of existing Free and Open Source web games that might work well on the site, let me know and I’ll have a chat with their creators: I definitely plan to include games by more people than just me.

Code your first game: Snake in JavaScript (on Raspberry Pi)

Welcome! We are going to code a whole snake game. It’s going to look like this:

A finished snake game being played

It doesn’t matter if you have never written any code before: I am going to try and explain everything from scratch.

I’m going to assume you are using a Raspberry Pi. but you don’t need one – all you need is Git, a text editor (like Notepad on Windows) and a web browser (like Firefox or Chrome or Safari or Edge). All laptops and desktop PCs should have the last two. On a tablet or phone it will be more tricky: try to use a Raspberry Pi or other computer if you can. If you are not using a Raspberry Pi, the screenshots will look a bit different, so you’ll have to do a little detective work to follow along.

For the first part where we download some code, your Raspberry Pi needs to be connected to the Internet, but once the “git clone” part is done, you can do everything else disconnected.

Before we start

If you’ve got a Raspberry Pi with Raspberry Pi OS on it, you are ready to go!

If you’re on another computer, make sure you’ve got Git installed. This should be easy on Linux, and possible on Mac or Windows.

Setting up

The first thing we need to do is download the code we are going to use to help us make a game. We’ll get it from my games web site, Smolpxl games.

Start your web browser:

Click the browser launch button

type in the address: smolpxl.artificialworlds.net

Typing the address into the browser address bar

and press Enter.

Scroll to the bottom of the page and click the link “make your own games at gitlab.com/andybalaam/smolpxl“.

Choosing the gitlab link at the bottom of the page

On the GitLab page that loads, click on Clone on the right:

Clicking the Clone button

and, under “Clone with HTTPS” click the “Copy URL” button:

Clicking the Copy URL button under Clone with HTTPS

Now start the Terminal:

Starting the Terminal by clicking its icon

and type in “git clone” followed by a space, but don’t press Enter yet. Right-click in the Terminal and choose “Paste” to paste in the URL that we copied before.

Your Terminal should look like this:

The Terminal showing text: git clone https://gitlab.com/andybalaam/smolpxl.git

Press Enter, and it should change to look like this:

Results of a successful git clone command

Go to your file manager:

Opening file manager by clicking its icon

and double-click on “smolpxl”:

Double-clicking smolpxl folder

and then double-click on “public”:

Double-clicking public folder

We are going to copy the “snake” folder. Right-click on it:

Right-clicking the snake folder

and choose “Copy”:

Choosing "Copy"

then right-click somewhere in the space below, and click “Paste”:

Choosing Paste

The Pi doesn’t want to paste the new one over the old one, so type a new name for it: “minisnake”, and click “Rename”:

Renaming pasted folder to minisnake

Now go into your new minisnake folder by double-clicking on it:

Double-clicking the new minisnake folder

We’re going to edit the file called game.js. To do this, double-click on it:

Double-clicking game.js

This should open the code in a text editor program called Mousepad. (If you’re not on a Raspberry Pi, you might need to right-click the file and choose “Edit”, or similar, to open it in a text editor.)

We want to delete everything in here and start again. Click “Edit”, then “Select all”:

Choosing Edit, then Select All, and hello to you if you are reading this!

Now press the “Delete” key, and all the code should be gone:

Mousepad with all the code gone

We are ready to start!

A game that does nothing

First, we’re going to type some code that gets us ready, but doesn’t actually do anything useful.

Type in this code:

const game = new Smolpxl.Game();
game.setSize(20, 20);
game.setTitle("Minisnake");

function update(runningGame, model) {
}

function view(screen, model) {
}

game.start("minisnake", {}, view, update);

This sets the size of our screen (20 by 20 squares), and a title, and gets things ready for our game. Notice that we start our game by writing game.start.

Once you’ve typed in the code:

Mousepad with the code I just told you to type in

click “File” and then “Save”:

Choosing "File", then "Save"

Now we’re going to try it out! Go back to the minisnake folder:

Switching back to file manager, which is on the minisnake folder still

and double-click on the file called index.html:

Double-clicking index.html

If you typed everything correctly, a title screen should appear. To figure out any problems, we will want the Developer tools open. To do that, click the three dots in the top right, then “More tools”, and “Developer tools”:

Choosing Chromium's burger menu in the top right, then clicking More tools, then Developer Tools

The tool we want is the Console, so click “Console” to see it:

Choosing the Console tab in Developer tools

If everything went well, the Console should be empty:

Our game on the left, and the Console on the right, with no error messages showing

If you see red error messages in the Console, you probably have a typo: double-check everything you typed, and compare it with this version: game-01-nothing.js.

Now we have everything ready to get started, and you can see our game already has a title screen!

The game is going to be made from three things:

  • a model,
  • a view, and
  • an update function.

Let’s start with the model and the view.

The Model and the View

The first thing we’re going to do is add an apple to the game. To do that, we need to do two things:

  • say where the apple is, and
  • draw the apple on the screen.

To say where the apple is, we need a model. We actually already have a model, but it’s just empty.

Switch back to your code:

Switching back to Mousepad by clicking game.js in the top bar

Look at the last line of code we typed, and find where you typed {}. This is the “model”, or the state of out game when we start it off. Now, we’re going to replace that with newModel(), which means call a function called newModel. (A function is a piece of code that we can re-use, and “calling” a function means running that code.)

So we’re going to write a function called newModel, and then call it. Replace the very last line of your code with this:

function newModel() {
    return {
        apple: [5, 5]
    };
}

game.start("minisnake", newModel(), view, update);

The top part makes a function called newModel, and the bottom part is the same as you had before, except {} is replaced by newModel().

Have another look at the newModel function we wrote (the top part). Can you see where we made the apple? After the word return, we have some stuff inside curly brackets ({ and later, }). When we use curly brackets like that, we are making what JavaScript calls an object. Objects have properties. So far, our object only has one property, which is called “apple”. The right-hand part, [5, 5] is how we are say where our apple is – it’s at the position 5 squares in from the left, and 5 squares down from the top of our screen.

Now we have said where our apple is, we also need to draw it on the screen. That happens inside the view function. Scroll up and find the part that says function view() and add a new line between the opening { and the closing one:

function view(screen, model) {
    screen.set(model.apple[0], model.apple[1], [255, 0, 0]);
}

We are calling a function called screen.set, which draws a square on the screen. We are passing in three things to it: two points for the position to draw, and then the colour to use. The position is model.apple[0], which means take the first part of the [5, 5] we typed before, and model.apple[1], which means take the second part. The color is [255, 0, 0] which means lots of red, and no green or blue, because this is a red-green-blue (RGB) colour.

So let’s try this out and see whether the apple is drawn on the screen. First save your code in Mousepad, then switch back to our game (in Chromium), and click the refresh button to reload it.

Switching back to Chromium in the top bar, then clicking the refresh button

You should see your game saying “Enter to start”. Click on it, and the game should start, and look like this:

The game window is black, except there is a red square for the apple

Well done, you have drawn an apple!

If you see red error messages in the Console, you probably have a typo: double-check everything you typed, and compare it with this version: game-02-just-apple.js.

Drawing a snake

Now that we have an apple, let’s follow the exact same pattern to add a snake. We’re going to add information about it to the model, and then use that information to draw it on the screen.

First, change the newModel function to look like this:

function newModel() {
    return {
        apple: [5, 5],
        body: [[10, 10], [10, 11], [10, 12], [10, 13], [10, 14]]
    };
}

Don’t miss the extra comma at end of the “apple” line!

To give the co-ordinates of the apple, we just used one [x, y] pair. Because the snake’s body is made from 5 points, we need 5 similar pairs.

We’ve described the snake’s body position, so now let’s draw it on the screen. Update the view function so it looks like this:

function view(screen, model) {
    // Snake
    for (const [x, y] of model.body) {
        screen.set(x, y, [0, 255, 0]);
    }

    // Apple
    screen.set(model.apple[0], model.apple[1], [255, 0, 0]);
}

We have added a for loop – it runs through all the points in body, and for each one it draws a square on the screen. This time the colour is [0, 255, 0], which means the snake will be green.

(By the way, did you notice the lines we added that start with // – these are “comments” – we can write anything we like after the slashes and it doesn’t do anything. We can use comments to add notes that help us remember what different bits of our program do.

Save the file, go back to the game in Chromium and click the Refresh button again. If all goes well, you should see the snake appear:

A black game screen, with a red dot and green line for the snake

If you see red error messages in the Console, you probably have a typo: double-check everything you typed, and compare it with this version: game-03-snake.js.

So now we have a snake and an apple, but nothing is really happening … let’s fix that next.

Making the snake move

We’ve made a model in newModel and we’ve drawn it on the screen in view, but how do we make things move around? That is where the update function comes in: this is where we change the model based on what is happening.

Let’s start by making the snake move forward forever. Change the update function to look like this:

function update(runningGame, model) {
    // Move the snake
    let newHead = Smolpxl.coordMoved(model.body[0], model.dir);
    let newTail = model.body.slice(0, -1);

    model.body = [newHead, ...newTail];
    return model;
}

and change the newModel function to look like this:

function newModel() {
    return {
        apple: [5, 5],
        body: [[10, 10], [10, 11], [10, 12], [10, 13], [10, 14]],
        dir: Smolpxl.directions.UP
    };
}

Again, notice the comma at the end of the body line!

We added dir to the model, which is the direction the snake is facing.

The update function makes newHead by moving the snake’s head (the first entry in its body, which it gets with model.body[0]) in the direction it is facing (model.dir). Then we create newTail, which is everything in the old model.body except the last entry in the list (this is what .slice(0, -1) means).

Finally we update the body by setting it to [newHead, ...newTail], which just means make a new list by sticking newHead on to the beginning of newTail.

Save, switch to the game in Chromium, and refresh. Because we set dir to Smolpxl.directions.UP inside newModel, the snake moves updards!

The green snake moves up the screen (and off the top)

If you see red error messages in the Console, you probably have a typo: double-check everything you typed, and compare it with this version: game-04-movement.js.

If the snake disappears off the top, click the refresh button to see it again.

The game isn’t too much fun yet. Let’s add some keyboard controls, and allow you to die when you go off-screen.

Controlling the snake

The update function moves the snake, but now it’s time to make it a bit cleverer, by changing direction when you press a key, and stopping you when you go off-screen. Change it so it looks like this:

function update(runningGame, model) {
    if (!model.alive) {
        return;
    }

    if (runningGame.receivedInput("LEFT"))  {
        model.dir = Smolpxl.directions.LEFT;
    } else if (runningGame.receivedInput("RIGHT")) {
        model.dir = Smolpxl.directions.RIGHT;
    } else if (runningGame.receivedInput("UP")) {
        model.dir = Smolpxl.directions.UP;
    } else if (runningGame.receivedInput("DOWN")) {
        model.dir = Smolpxl.directions.DOWN;
    }

    // Move the snake
    let newHead = Smolpxl.coordMoved(model.body[0], model.dir);
    let newTail = model.body.slice(0, -1);

    // Die if we hit the edge
    if (
        newHead[0] === runningGame.screen.minX ||
        newHead[0] === runningGame.screen.maxX ||
        newHead[1] === runningGame.screen.minY ||
        newHead[1] === runningGame.screen.maxY
    ) {
        model.alive = false;
    }

    model.body = [newHead, ...newTail];
    return model;
}

and change the newModel function to look like this:

function newModel() {
    return {
        alive: true,
        apple: [5, 5],
        body: [[10, 10], [10, 11], [10, 12], [10, 13], [10, 14]],
        dir: Smolpxl.directions.UP
    };
}

We keep track of whether the snake is alive in the model, and we immediately return from update if we are dead, meaning the snake stops moving. (The ! in if (!model.alive) means “not”, so we are saying what to do when we are not alive – when we are dead. The return here means immediately stop running the code in this function.)

The next new part of update allows us to check whether an arrow key was pressed (using the runningGame.receivedInput function), and if so, change the direction of the snake (model.dir).

Finally, nearer the end of update, we check whether the position of the snake’s head (newHead) is at one of the screen edges, by comparing its co-ordinates with the maximum and minimum co-ordinates on the screen. If we are off the edge, we set model.alive to false, meaning the snake is now dead.

Save, switch to the game in Chromium, and refresh. With all that, we can control the snake with the arrow keys, and it can die:

A snake moves around, turning because arrow keys were used to control it

Try clicking on your game and then pressing the arrow keys to control the snake.

If you see red error messages in the Console, you probably have a typo: double-check everything you typed, and compare it with this version: game-05-control.js.

This is kind-of a game, but surely it’s time to eat some apples?

Eating apples

If we’re going to be eating apples, they should probably not always be in the same place, right?

Let’s start off by making a brand new function. Make some space immediately above the update function, and type in this code:

function randApple() {
    return [Smolpxl.randomInt(1, 18), Smolpxl.randomInt(1, 18)];
}

This function gives us some random co-ordinates where we can place an apple. Update newModel to place the first apple at a random place, by making it look like this:

function newModel() {
    return {
        alive: true,
        apple: randApple(),
        body: [[10, 10], [10, 11], [10, 12], [10, 13], [10, 14]],
        dir: Smolpxl.directions.UP
    };
}

So instead of writing the exact co-ordinates we want ([5, 5]), now we’re calling our new randApple function, which gives us back some random co-ordinates.

Now we can place apples randomly, let’s change the update function to allow us to eat apples. While we’re there, let’s check whether we crashed into our own body too:

function update(runningGame, model) {
    ... All the same stuff as before ...

    // Die if we hit the edge
    if (
        newHead[0] === runningGame.screen.minX ||
        newHead[0] === runningGame.screen.maxX ||
        newHead[1] === runningGame.screen.minY ||
        newHead[1] === runningGame.screen.maxY
    ) {
        model.alive = false;
    }

    // If we hit the apple we get longer
    if (Smolpxl.equalArrays(newHead, model.apple)) {
        for (let i = 0; i < 5; i++) {
            newTail.push([-1, -1]);
        }
        model.apple = randApple();
    }

    // If we hit our own body, we die
    if (Smolpxl.arrayIncludesArray(newTail, newHead)) {
        model.alive = false;
    }

    model.body = [newHead, ...newTail];
    return model;
}

We use the Smolpxl.equalArrays function* to ask whether the new position of the snake’s head (newHead) is the same as the position of the apple (model.apple).

* The function is called “equalArrays” because both newHead and model.apple are lists of two co-ordinates (x and y), so we are storing them inside JavaScript “arrays”. You can spot an array because it is surrounded by square brackets ([).

If the head is on top of the apple, we do two things: add new items to the end of newTail, by using a for loop that runs 5 times. Each time it runs, it uses push to add another body part on the to the tail. (It actually adds an off-screen point [-1, -1] every time – this means they won’t get drawn at first, but as the snake moves forward, they will gradually get replaced by on-screen points, and we’ll see the snake gradually get longer.)

The second thing we do if the head is on the apple is move the apple by setting model.apple to another random point given to us by randApple.

We also check whether we have hit our own body using the Smolpxl.arrayIncludesArray function (asking whether newHead is the same point as one of the points in newTail) and set model.alive to false, meaning we’re dead, if so.

Save, switch to the game in Chromium, and refresh. Now we can really play a game of snake!

A snake moves around the screen, eating apples and getting longer when it does

If you see red error messages in the Console, you probably have a typo: double-check everything you typed, and compare it with this version: game-06-eating-apples.js.

We are nearly done. The last jobs are to make the game look a little nicer, and handle starting a new game after we crash.

Finishing off

We need to draw the walls around the edge that you crash into, and it would also be nice to show your score at the top of the screen. We can do both of these by adding a bit more at the beginning of the view function:

function view(screen, model) {
    screen.messageTopLeft(`Score: ${model.body.length}`);

    // Walls
    for (const x of screen.xs()) {
        screen.set(x, screen.minY, [150, 150, 150]);
        screen.set(x, screen.maxY, [150, 150, 150]);
    }
    for (const y of screen.ys()) {
        screen.set(screen.minX, y, [150, 150, 150]);
        screen.set(screen.maxX, y, [150, 150, 150]);
    }

    ... All the same stuff as before ...
}

We use the screen.messageTopLeft function to display a message at the top. To format the message we write some text inside backticks (`), which allows us to substitute values in. Here we typed ${model.body.length}, which means our score is how many points there are inside our body (which is the length of the array model.body). The ${} part just means “substitute this value in here”.

We make use of the screen.xs() and screen.ys() functions that give us a list of all the x and y co-ordinates on the screen to draw the wall.

Save, switch back to the game, and refresh to see how this looks. If you want to check what you have typed so far, compare it against game-07-walls-and-score.js.

The very last thing we need to do is handle restarting the game after we crash.

First, let’s display a message on the screen by updating the view function one last time:

function view(screen, model) {
    ... All the same stuff as before ...

    // Snake
    for (const [x, y] of model.body) {
        screen.set(x, y, [0, 255, 0]);
    }

    // Death
    if (!model.alive) {
        screen.set(model.body[0][0], model.body[0][1], [0, 0, 255]);
        screen.dim();
        const score = model.body.length;
        screen.message(["Well done!", `Score: ${score}`, "Press <SELECT>"]);
    }

    // Apple
    screen.set(model.apple[0], model.apple[1], [255, 0, 0]);
}

When the snake is not alive we draw a blue dot ([0, 0, 255]) where its head is, then make the screen dark with the screen.dim function, and write a message on the screen with the screen.message function.

Last, if we press Enter while the death message is visible, we want to go back to the start. We can do that with a small change to the update function:

function update(runningGame, model) {
    if (!model.alive) {
        if (runningGame.receivedInput("SELECT")) {
            runningGame.endGame();
            return newModel();
        } else {
            return;
        }
    }

    if (runningGame.receivedInput("LEFT"))  {
    ... All the same stuff as before ...

Notice that we deleted the line that just said return; and replaced it with the new if code. We check whether the user pressed the SELECT button (which means the Enter key) and if so, we tell Smolpxl to go back to the title screen by calling runningGame.endGame, and then we reset everything in the model by returning newModel().

Save, switch to the game in Chromium, and refresh. Our game is finished!

A finished snake game being played

If you see red error messages in the Console, you probably have a typo: double-check everything you typed, and compare it with this version: game-08-finished.js.

Well done!

If you got that working, you should be extremely pleased with yourself. Typing all that code correctly is a major challenge, and your determination has paid off.

Take a breath, and have a think about whether you can show what you’ve done to someone else. I’m pretty sure they will be impressed that you coded an entire game!

Challenges

When you’ve properly taken the time to enjoy the great work you’ve done, try reading through the code, and reading back through this blog post to try and understand how the program works.

To learn more about JavaScript and making web sites, you can follow this much more comprehensive tutorial: Getting started with the Web.

You’ve done really well. If you want an extra challenge, try improving your game using these challenges. Bear in mind though, working these out on your own will be much harder than what we’ve done so far:

  • Challenge 1: Display a different message if you don’t eat any apples. If you crash before eating any apples, you might need some extra encouragement: change the part of the view function where we display a “Well done!” message, and if our score is 5 (if (model.body.length === 5) then display a different message – maybe “Bad luck!”. Look for ifelse statements we have already written, to see how they work.
  • Challenge 2: Prevent going back on yourself. Update your code so that when snake is going right, and the user presses left, we ignore it. You will need to change the update function, where we use receivedInput to check what key the user pressed. Instead of just setting the direction, you will need to add a new if statement that check what direction we are facing already.
  • Challenge 3: Remember the high score. Remember the best score anyone has got, and display it at the top-right of the screen. At the moment in our code, we return newModel(); in update, when we want to restart the game. That means we have a totally fresh model, forgetting everything else that happened before. If we want to remember a high score, we can’t do that! If you want some extra inspiration, have a look at the more complete version of snake that comes with the Smolpxl games. It’s actually included in the code you downloaded with the git clone command you typed at the very beginning.

What next?

To learn more, try:

Remember to have fun, and be kind to the people you meet on the way.

Play and create little retro games at Smolpxl

I love simple games: playing them and writing them.

But, it can be overwhelming getting started in the complex ecosystems of modern technology.

So, I am writing the Smolpxl library, which is some JavaScript code that makes it quite simple to write simple, pixellated games. It gives you a fixed-size screen to draw on, and it handles your game loop and frames-per-second, so you can focus on two things: updating your game “model” – the world containing all the things that exist in your game, and drawing onto the screen.

I am also writing some games with this library, and publishing them at smolpxl.artificialworlds.net:

I am hoping this site will gradually gain more and more Free and Open Source games, and start to rival some of the advertising-supported sites for the attention of casual gamers, especially kids.

Smolpxl is done for fun, and for its educational value, so it should be a safer place for kids than a world of advertising, loot boxes and site-wide currencies.

As I write games, I want to show how easy and fun it can be, so I will be streaming myself live doing it on twitch.tv/andybalaam, and putting the recordings up on peertube.mastodon.host/accounts/andybalaam and youtube.com/user/ajbalaam.

I am hoping these videos will serve as tutorials that help other people get into writing simple games.

Would you like to help? If so:

shareon.js.org now has a Share to Mastodon button

I was looking for the right way to make a “Share This”-style button for my tiny games site Smolpxl, and I found shareon which worked exactly the way I wanted (load the JavaScript and call a function to display the buttons, with no privacy concerns), and looked really nice.

The only thing that was missing was a Mastodon button.

“Share to Mastodon” is more complicated than something like Share to Twitter, because Mastodon is not one web site, but lots of web sites that all talk to each other.

So, after someone clicks “Share to Mastodon”, you need to ask them which web site (or Mastodon instance) they meant.

I started out by hacking a Mastodon button in after the shareon ones, and prompting the user for their instance. This was a little unfriendly, but it worked:

Click Share, Mastodon, enter instance URL into ugly browser prompt, and end up at Mastodon

But luckily I didn’t stick with that. Because I think shareon is awesome, and because I want more people to use Mastodon, I decided to try adding a Mastodon button to shareon. I wrote the code to work similarly to my original hack, and submitted a Pull Request.

I am really glad I did that, because what followed was a really positive Free and Open Source Software mini-interaction. Nick Karamov responded with lots of improvements and bug fixes to my original change, and as we discussed the problem more, I expressed the desire for a proper page to choose Mastodon instance, that would be more friendly than a simple prompt. I also said that it would be difficult.

In retrospect, maybe suggesting it would be difficult was a clever trick, because the next thing I knew, Nick had implemented just such a page: toot.karamoff.dev!

Because toot.karamoff.dev now existed, the “Share to Mastodon” button became much simpler: we can send our post information to toot.karamoff.dev, and it asks which Mastodon instance you want to use, and passes it on the correct place.

So my new Pull Request was much simpler than the original, and with a few more improvements suggested by Nick, it’s merged and I have a usable Share to Mastodon button without hacking it in.

The cake has a little more icing too, because I was also able to improve toot.karamoff.dev by adding code that downloads the up-to-date list of Mastodon instances from joinmastodon.org and provides them as suggestions, which can be really helpful if you can’t remember the exact name of your instance. Again, Nick’s suggestions on my Pull Request were incredibly helpful and made the code way better than what I originally wrote. Now it works really smoothly:

Click Share, Mastodon, choose instance from a friendly list on toot, and end up at Mastodon

In a small way, this was a fantastic example of how effective and fun working on Free and Open Source Software can be.

short – command line tool to truncate lines to fit in the terminal

Sometimes I run grep commands that search files with hugely-long lines. If those lines match, they are printed out and spam my terminal with huge amounts of information, that I probably don’t need.

I couldn’t find a tool that limits the line-length of its output, so I wrote a tiny one.

It’s called short.

You use it like this (my typical usage):

grep foo myfile.txt | short

Or specify the column width like this:

short -w 5 myfile.txt

It’s written in Rust. Feel free to add features, fix bugs and package it for your operating system/distribution at gitlab.com/andybalaam/short.

How to send an SMS using netcat (via SMPP)

SMPP is a binary protocol used by phone companies to send text messages, otherwise known as SMS messages.

It can work over TCP, so we can use netcat on the command line to send messages.

Setting up

[Note: the netcat I am using is Ncat 7.70 on Linux.]

The server that receives messages is called an SMSC. You may have your own one, but if not, you can use the CloudHopper one like this:

sudo apt install make maven  # (or similar on non-Debian-derived distros)
git clone https://github.com/fizzed/cloudhopper-smpp.git
cd cloudhopper-smpp

If you are a little slow, like me, I’d suggest making it wait a bit longer for bind requests before giving up on you. To do that, edit the main() method of src/test/java/com/cloudhopper/smpp/demo/ServerMain.java to add a line like this: configuration.setBindTimeout(500000); on about line 80, near the other similar lines. This will make it wait 500 seconds for you to send a BIND_TRANSCEIVER, instead of giving up after just 5 seconds.

Once you’ve made that change, you can run:

make server

Now you have an SMSC running!

Leave that open, and go into another terminal, and type:

mkfifo tmpfifo
nc 0.0.0.0 2776 < tmpfifo | xxd

The mkfifp part creates a “fifo” – a named pipe through which we will send our SMPP commands.

The nc part starts Ncat, connecting to the SMSC we started.

The xxd part will take any binary data coming out of Ncat and display it in a more human-readable way.

Leave that open too, and in a third terminal type:

exec 3> tmpfifo

This makes everything we send to file descriptor 3 go into the fifo, and therefore into Ncat.

Now we have a way of sending binary data to Ncat, which will send it on to the SMSC and print out any responses.

Note: we will be using SMPP version 3.4 since it is in the widest use, even though it is not the newest.

Terminology

“SMPP” is the protocol we are speaking, which we are using over TCP/IP.

An SMSC is a server (which receives messages intended for phones and sends back responses and receipts).

We will be acting as an ESME or client (which sends messages intended for phones and receives responses and receipts).

The units of information that are passed back and forth in SMPP are called “PDUs” (Protocol Data Units) – these are just bits of binary data that flow over the TCP connection between two computers.

The spec talks about “octets” – this means 8-bit bytes.

ENQUIRE_LINK

First, we’ll check the SMSC is responding, by sending an ENQUIRE_LINK, which is used to ask the SMSC whether it’s there and working.

Go back to the third terminal (where we ran exec) and type this:

LEN16='\x00\x00\x00\x10'
ENQUIRE_LINK='\x00\x00\x00\x15'
NULL='\x00\x00\x00\x00'
SEQ1='\x00\x00\x00\x01'

echo -n -e "${LEN16}${ENQUIRE_LINK}${NULL}${SEQ1}" >&3

Explanation: an ENQUIRE_LINK PDU consists of:

  • 4 bytes to say the length of the whole PDU in bytes. ENQUIRE_LINK PDUs are always 16 bytes, “00000010” in hex. I called this LEN.
  • 4 bytes to say what type of PDU this is. ENQUIRE_LINK is “00000015” in hex. I called this ENQUIRE_LINK.
  • 4 bytes that are always zero in ENQUIRE_LINK. I called this NULL.
  • 4 bytes that identify this request, called a sequence number. The response from the server will include this so we can match responses to requests. I called this SEQ.

Check back in the second terminal (where you ran nc). If everything worked, you should see something like this:

00000000: 0000 0010 8000 0015 0000 0000 0000 0001  ................

Ignoring the first and last parts (which are how xxd formats its output), the response we receive is four 4-byte parts, very similar to what we sent:

  • 4 bytes to say the length of the whole PDU in bytes. Here it is “00000010” hex, or 16 decimal.
  • 4 bytes to say what type of PDU this is. Here is is ENQUIRE_LINK is “80000015” in hex, which is the code for ENQUIRE_LINK_RESP.
  • 4 bytes for the success status of the ENQUIRE_LINK_RESP. This is always “00000000”, which means success and is called ESME_ROK in the spec.
  • 4 bytes that match the sequence number we sent. This is “00000001”, as we expected.

BIND_TRANSCEIVER

Now we can see that the SMSC is working, let’s “bind” to it. That means something like logging in: we convince the SMSC that we are a legitimate client, and tell it what type of connection we want, and, assuming it agrees, it will hold the connection open for us for as long as we need.

We are going to bind as a transceiver, which means both a transmitter and receiver, so we can both send messages and receive responses.

Send the bind request like this:

LEN32='\x00\x00\x00\x20'
BIND_TRANSCEIVER='\x00\x00\x00\x09'
NULL='\x00\x00\x00\x00'
SEQ2='\x00\x00\x00\x02'
SYSTEM_ID="sys\x00"
PASSWORD="pas\x00"
SYSTEM_TYPE='typ\x00'
SYSTEM_ID='sys\x00'
PASSWORD='pas\x00'
INTERFACE_VERSION='\x01'
INTERFACE_VERSION='\x34'
ADDR_TON='\x00'
ADDR_NPI='\x00'
ADDRESS_RANGE='\x00'

echo -n -e "${LEN32}${BIND_TRANSCEIVER}${NULL}${SEQ2}${SYSTEM_ID}${PASSWORD}${SYSTEM_TYPE}${INTERFACE_VERSION}${ADDR_TON}${ADDR_NPI}${ADDRESS_RANGE}" >&3

Explanation: this PDU is 32 bytes long, so the first thing we send is “00000020” hex, which is 32.

Then we send “00000009” for the type: BIND_TRANSCEIVER, 4 bytes of zeros, and a sequence number – this time I used 2.

That was the header. Now the body of the PDU starts with a system id (basically a username), a password, and a system type (extra info about who you are). These are all variable-length null-terminated strings, so I ended each one with \x00.

The rest of the body is some options about the types of phone number we will be sending from and sending to – I made them all “00” hex, which means “we don’t know”.

If it worked, you should see this in the nc output:

00000000: 0000 0021 8000 0009 0000 0000 0000 0002  ...!............
00000010: 636c 6f75 6468 6f70 7065 7200 0210 0001  cloudhopper.....

As before, the first 4 bytes are for how long the PDU is – 33 bytes – and the next 4 bytes are for what type of PDU this is – “80000009” is for BIND_TRANSCEIVER_RESP which is the response to a BIND_TRANSCEIVER.

The next 4 bytes are for the status – these are zeroes which indicates success (ESME_ROK) again. After that is our sequence number (2).

The next 15 bytes are the characters of the word “cloudhopper” followed by a zero – this is the system id of the SMSC.

The next byte (“01”) – the last one we can see – is the beginning of a “TLV”, or optional part of the response. The xxd program actually delayed the last byte of the output, so we can’t see it yet, but it is “34”. Together, “0134” means “the interface version we support is SMPP 3.4”.

SUBMIT_SM

The reason why we’re here is to send a message. To do that, we use a SUBMIT_SM:

LEN61='\x00\x00\x00\x3d'
SUBMIT_SM='\x00\x00\x00\x04'
SEQ3='\x00\x00\x00\x03'
SERVICE_TYPE='\x00'
SOURCE_ADDR_TON='\x00'
SOURCE_ADDR_NPI='\x00'
SOURCE_ADDR='447000123123\x00'
DEST_ADDR_TON='\x00'
DEST_ADDR_NPI='\x00'
DESTINATION_ADDR='447111222222\x00'
ESM_CLASS='\x00'
PROTOCOL_ID='\x01'
PRIORITY_FLAG='\x01'
SCHEDULE_DELIVERY_TIME='\x00'
VALIDITY_PERIOD='\x00'
REGISTERED_DELIVERY='\x01'
REPLACE_IF_PRESENT_FLAG='\x00'
DATA_CODING='\x03'
SM_DEFAULT_MSG_ID='\x00'
SM_LENGTH='\x04'
SHORT_MESSAGE='hihi'
echo -n -e "${LEN61}${SUBMIT_SM}${NULL}${SEQ3}${SERVICE_TYPE}${SOURCE_ADDR_TON}${SOURCE_ADDR_NPI}${SOURCE_ADDR}${DEST_ADDR_TON}${DEST_ADDR_NPI}${DESTINATION_ADDR}${ESM_CLASS}${PROTOCOL_ID}${PRIORITY_FLAG}${SCHEDULE_DELIVERY_TIME}${VALIDITY_PERIOD}${REGISTERED_DELIVERY}${REPLACE_IF_PRESENT_FLAG}${DATA_CODING}${SM_DEFAULT_MSG_ID}${SM_LENGTH}${SHORT_MESSAGE}" >&3

LEN61 is the length in bytes of the PDU, SUBMIT_SM is the type of PDU, and SEQ3 is a sequence number, as before.

SOURCE_ADDR is a null-terminated (i.e. it ends with a zero byte) string of ASCII characters saying who the message is from. This can be a phone number, or a name (but the rules about what names are allowed are complicated and region-specific). SOURCE_ADDR_TON and SOURCE_ADDR_NPI give information about what type of address we are providing – we set them to zero to mean “we don’t know”.

DESTINATION_ADDR, DEST_ADDR_TON and DEST_ADDR_NPI describe the phone number we are sending to.

ESM_CLASS tells the SMSC how to treat your message – we use “store and forward” mode, which means keep it and send it when you can.

PROTOCOL_ID tells it what to do if it finds duplicate messages – we use “Replace Short Message Type 1”, which I guess means use the latest version you received.

PRIORITY_FLAG means how important the message is – we used “interactive”.

SCHEDULE_DELIVERY_TIME is when to send – we say “immediate”.

VALIDITY_PERIOD means how long should this message live before we give up trying to send it (e.g. if the user’s phone is off). We use “default” so the SMSC will do something sensible.

REGISTERED_DELIVERY gives information about whether we want a receipt saying the message arrived on the phone. We say “yes please”.

REPLACE_IF_PRESENT_FLAG is also about duplicate messages (I’m not sure how it interacts with PROTOCOL_ID) – the value we used means “don’t replace”.

DATA_CODING states what character encoding you are using to send the message text – we used “Latin 1”, which means ISO-8859-1.

SM_DEFAULT_MSG_ID allows us to use one of a handful of hard-coded standard messages – we say “no, use a custom one”.

SM_LENGTH is the length in bytes of the “short message” – the actual text that the user will see on the phone screen.

SHORT_MESSAGE is the short message itself – our message is all ASCII characters, but we could use any bytes and they will be interpreted as characters in ISO-8859-1 encoding.

You should see a response in the other terminal like this:

00000020: 3400 0000 1180 0000 0400 0000 0000 0000  4...............

The initial “34” is the left-over byte from the previous message as mentioned above. After that, we have:

“00000011” for the length of this PDU (17 bytes).

“80000004” for the type – SUBMIT_SM_RESP which tells us whether the message was accepted (but not whether it was received).

“00000000” for the status – zero means “OK”.

The last two bytes are chopped off again, but what we actually get back is:

“00000003”, which is the sequence number, and then:

“00” which is a null-terminated ASCII message ID: in this case the SMSC is saying that the ID it has given this message is “”, which is probably not very helpful! If this ID were not empty, it would help us later if we receive a delivery receipt, or if we want to ask about the message, or change or cancel it.

DELIVER_SM

If you stop the SMSC process (the one we started with make server) by pressing Ctrl-C, and start a different one with make server-echo, and then repeat the other commands (note you need to be quick because you only get 5 seconds to bind before it gives up on you – make similar changes to what we did in ServerMain to ServerEchoMain if this causes problems), you will receive a delivery receipt from the server, which looks like this:

“0000003d” for the length of this PDU (59 bytes).

“00000005” for the type (DELIVER_SM).

“00000000” for the unused command status.

“00000001” as a sequence number. Note, this is unrelated the sequence number of the original message: to match with the original message, we must use the message ID provided in the SUBMIT_SM_RESP.

“0000003400” to mean we are using SMPP 3.4. (This is a null-terminated string of bytes.)

“00” and “00” for the TON and NPI of the source address, followed by the source address itself, which is a null-terminated ASCII string: “34343731313132323232323200”. This translates to “447111222222”, which was the destination address of our original message. Note: some SMSCs switch the source and destination addresses like this in their delivery receipts, and some don’t, which makes life interesting.

“00” and “00” for the TON and NPI of the destination address, followed by “34343730303031323331323300” for the address itself, which translates to “447000123123”, as expected.

The DELIVER_SM PDU continues with much of the information repeated from the original message, and the SMSC is allowed to provide a short message as part of the receipt – in our example the cloudhopper SMSC repeats the original message. Some SMSCs use the short message to provide information such as the message ID and the delivery time, but there is no formal standard for how to provide it. Other SMSCs use a TLV to provide the message ID instead.

Somewhere in the DELIVER_SM you should find some indication of whether the message was actually delivered to the phone. Often it’s in a TLV called “message state”, but it could also be in the message body. Bizarrely, a state of “4” is the normal code for “delivered successfully”.

In order to complete the conversation, you should provide a DELIVER_SM_RESP, and then an UNBIND, but hopefully based on what we’ve done and the SMPP 3.4 standard, you should be able to figure it out.

You did it

SMPP is a binary protocol layered directly on top of TCP, which makes it slightly harder to work with by hand than the HTTP protocols with which many of us are more familiar, but I hope I’ve convinced you it’s possible to understand what’s going on without resorting to some kind of heavyweight debugging tool or library.

Happy texting!

Letter to my MP on racism in the US and UK

Here is the text of a letter I wrote to my MP today via writetothem.com:

Dear [],

I am writing to you to express my growing horror at the ongoing police violence and brutality Black protesters are facing in the US, and the UK’s disturbing silence on the matter.

These protests, begun as peaceful demonstrations against the state-enabled murder of George Floyd at the hands of the Minneapolis Police Department, have been escalated by the US police, who are using excessive force against protesters practising their civic rights. If this were a non-white country, Western countries would have denounced this brutality; instead, the UK has elected to remain silent.

I am painfully aware that the UK itself suffers from many racist structures. Thanks to systematic injustice, the recent pandemic has impacted BAME people at a vastly higher rate than white people. The death of Belly Mujinga, caused by a racist attack and by her presence at work despite underlying conditions that should have resulted in paid leave, was horrifying and preventable, and the dropped inquiry utterly disgraceful.

I ask you personally to:
– write to the Foreign Secretary asking for a strong statement of condemnation of police brutality in the US,
– make it a personal priority to address the systematic injustice of unfair outcomes for BAME people in the UK,
– ensure that the government responds materially to the report on BAME COVID-19 deaths,
– insist on a full enquiry into the circumstances surrounding Belly Mujinga’s death.

Yours sincerely,
[]

Inspired by: @anahitrooz.

Example Android project with repeatable tests running inside an emulator

I’ve spent the last couple of days fighting the Android command line to set up a simple project that can run automated tests inside an emulator reliably and repeatably.

To make the tests reliable and independent from anything else on my machine, I wanted to store the Android SDK and AVD files in a local directory.

To do this I had to define a lot of inter-related environment variables, and wrap the tools in scripts that ensure they run with the right flags and settings.

The end result of this work is here: gitlab.com/andybalaam/android-skeleton

You need all the utility scripts included in that repo for it to work, but some highlights include:

The environment variables that I source in every script, scripts/paths:

PROJECT_ROOT=$(dirname $(dirname $(realpath ${BASH_SOURCE[${#BASH_SOURCE[@]} - 1]})))
export ANDROID_SDK_ROOT="${PROJECT_ROOT}/android_sdk"
export ANDROID_SDK_HOME="${ANDROID_SDK_ROOT}"
export ANDROID_EMULATOR_HOME="${ANDROID_SDK_ROOT}/emulator-home"
export ANDROID_AVD_HOME="${ANDROID_EMULATOR_HOME}/avd"

Creation of a local.properties file that tells Gradle and Android Studio where the SDK is, by running something like this:

echo "# File created automatically - changes will be overwritten!" > local.properties
echo "sdk.dir=${ANDROID_SDK_ROOT}" >> local.properties

The wrapper scripts for Android tools e.g. scripts/sdkmanager:

#!/bin/bash

set -e
set -u

source scripts/paths

"${ANDROID_SDK_ROOT}/tools/bin/sdkmanager" \
    "--sdk_root=${ANDROID_SDK_ROOT}" \
    "$@"

The wrapper for avdmanager is particularly interesting since it seems we need to override where it thinks the tools directory is for it to work properly – scripts/avdmanager:

#!/bin/bash

set -e
set -u

source scripts/paths

# Set toolsdir to include "bin/" since avdmanager seems to go 2 dirs up
# from that to find the SDK root?
AVDMANAGER_OPTS="-Dcom.android.sdkmanager.toolsdir=${ANDROID_SDK_ROOT}/tools/bin/" \
    "${ANDROID_SDK_ROOT}/tools/bin/avdmanager" "$@"

An installation script that must be run once before using the project scripts/install-android-tools:

#!/bin/bash

set -e
set -u
set -x

source scripts/paths

mkdir -p "${ANDROID_SDK_ROOT}"
mkdir -p "${ANDROID_AVD_HOME}"
mkdir -p "${ANDROID_EMULATOR_HOME}"

# Download sdkmanager, avdmanager etc.
cd "${ANDROID_SDK_ROOT}"
test -f commandlinetools-*.zip || \
    wget -q 'https://dl.google.com/android/repository/commandlinetools-linux-6200805_latest.zip'
unzip -q -u commandlinetools-*.zip
cd ..

# Ask sdkmanager to update itself
./scripts/sdkmanager --update

# Install the emulator and tools
yes | ./scripts/sdkmanager --install 'emulator' 'platform-tools'

# Platforms
./scripts/sdkmanager --install 'platforms;android-21'
./scripts/sdkmanager --install 'platforms;android-29'

# Install system images for our oldest and newest supported API versions
yes | ./scripts/sdkmanager --install 'system-images;android-21;default;x86_64'
yes | ./scripts/sdkmanager --install 'system-images;android-29;default;x86_64'

# Create AVDs to run the system images
echo no | ./scripts/avdmanager -v \
    create avd \
    -f \
    -n "avd-21" \
    -k "system-images;android-21;default;x86_64" \
    -p ${ANDROID_SDK_ROOT}/avds/avd-21
echo no | ./scripts/avdmanager -v \
    create avd \
    -f \
    -n "avd-29" \
    -k "system-images;android-29;default;x86_64" \
    -p ${ANDROID_SDK_ROOT}/avds/avd-29

Please do contribute to the project if you know easier ways to do this stuff.

Creating a tiny Docker image of a Rust project

I am building a toy project in Rust to help me learn how to deploy things in AWS. I’m considering using Elastic Beanstalk (AWS’s platform-as-a-service) and also Kubernetes. Both of these support deploying via Docker containers, so I am learning how to package a Rust executable as a Docker image.

My program is a small web site that uses Redis as a back end database. It consists of some Rust code and a couple of static files.

Because Rust has good support for building executables with very few dependencies, we can actually build a Docker image with almost nothing in it, except my program and the static files.

Thanks to Alexander Brand’s blog post How to Package Rust Applications Into Minimal Docker Containers I was able to build a Docker image that:

  1. Is very small
  2. Does not take too long to build

The main concern for making the build faster is that we don’t download and build all the dependencies every time. To achieve that we make sure there is a layer in the Docker build process that includes all the dependencies being built, and is not re-built when we only change our source code.

Here is the Dockerfile I ended up with:

# 1: Build the exe
FROM rust:1.42 as builder
WORKDIR /usr/src
Creating a tiny Docker image of a Rust project
# 1a: Prepare for static linking
RUN apt-get update && \
    apt-get dist-upgrade -y && \
    apt-get install -y musl-tools && \
    rustup target add x86_64-unknown-linux-musl

# 1b: Download and compile Rust dependencies (and store as a separate Docker layer)
RUN USER=root cargo new myprogram
WORKDIR /usr/src/myprogram
COPY Cargo.toml Cargo.lock ./
RUN cargo install --target x86_64-unknown-linux-musl --path .

# 1c: Build the exe using the actual source code
COPY src ./src
RUN cargo install --target x86_64-unknown-linux-musl --path .

# 2: Copy the exe and extra files ("static") to an empty Docker image
FROM scratch
COPY --from=builder /usr/local/cargo/bin/myprogram .
COPY static .
USER 1000
CMD ["./myprogram"]

The FROM rust:1.42 as build line uses the newish Docker feature multi-stage builds – we create one Docker image (“builder”) just to build the code, and then copy the resulting executable into the final Docker image.

In order to allow us to build a stand-alone executable that does not depend on the standard libraries in the operating system, we use the “musl” target, which is designed to statically linked.

The final Docker image produced is pretty much the same size as the release build of myprogram, and the build is fast, so long as I don’t change the dependencies in Cargo.toml.

A couple more tips to make the build faster:

1. Use a .dockerignore file. Here is mine:

/target/
/.git/

2. Use Docker BuildKit, by running the build like this:

DOCKER_BUILDKIT=1 docker build  .

Keeping track of podcast times with a simple bash script on Linux

I was recording some podcast audio tonight and wanted to be able to press a single key when I reached a significant moment, so I could add the times to the show notes.

I couldn’t find anything that already did this, so I wrote a tiny bash script. I ran this script and pressed Enter whenever I wanted a time recorded:

T=0
echo
while sleep 1; do
    echo -n -e "\e[1A"
    echo $(($T / 60))m $(($T % 60))s
    T=$(($T + 1))
done

The output looks like this:

$ ./times.bash 
0m 41s
6m 16s
9m 59s
13m 30s

The time ticks along, and when you press Enter that time stamp is recorded. You can copy this text out of your terminal to use in show notes. On most terminals you can copy text by selecting it with the mouse and pressing Ctrl-Shift-v.

Custom Bash tab completion for my program

I love Bash tab completion, and I want it for the command I am writing, so it can automatically complete parts of the command line when I run my program.

Code

Here is the script (install-bash-completion) I wrote to set it up (no need to be root – it installs in ~/.local):

#!/bin/bash

set -u
set -e

DIR=${BASH_COMPLETION_USER_DIR:-${XDG_DATA_HOME:-$HOME/.local/share}/bash-completion}/completions

mkdir -p ${DIR}

cp _myprogram ${DIR}

The actual completion script (_myprogram) it installs looks like this:

_myprogram_commands()
{
    local cur prev opts
    COMPREPLY=()
    cur="${COMP_WORDS[COMP_CWORD]}"
    prev="${COMP_WORDS[COMP_CWORD-1]}"
    opts=$(bash -c "./myprogram --commands")

    COMPREPLY=( $(compgen -W "${opts}" -- ${cur}) )
    return 0
}
complete -F _myprogram_commands ./myprogram

Installing

To install it, run:

./install-bash-completion

Then log out and log in again.

Now when you type ./myprogram and press TAB a couple of times, you should see the possible completions listed.

Notes

The completion script must be named to match the program name, with a leading underscore.

If I wanted it to work when it was installed in my PATH, I would need to change ./myprogram to just myprogram in 2 places.

Notice the line opts=$(bash -c "./myprogram --commands") – it actually runs my program to get the list of completions. This means my program needs to accept a --commands option which prints the valid commands. Alternatively, I could have hard-coded it by replacing that line with just:

opts="cmd1 cmd2 --help --etc"

More info:

Struggling with Rust to figure out the right types for a function signature

I am loving writing code in Rust. So many things about the language and its ecosystem feel so right*.

* For example: ownership of objects, expressive type system, compile to native, offline API docs, immutability, high quality libraries.

One of the things I like about it is that I don’t feel like I need to use an IDE, so I can happily code in Vim with no clever plugins.

One thing an IDE might give me would be an “extract function” refactoring. In most languages I am happy to do that manually, because I can let the compile errors guide me on what my function should look like.

However, in Rust I sometimes find it’s hard to find the right signature for a function I want to extract, and I am struggling to persuade the compiler to help me.

Here is an example from my new listsync project, in listsync-client-rust.rs:

use actix_web::{middleware, App, HttpServer};
use listsync_client_rust;
// ...
#[actix_rt::main]
async fn main() -> std::io::Result<()> {
//...
    HttpServer::new(|| {
        App::new()
            .wrap(listsync_client_rust::cookie_session::new_session())
            .wrap(middleware::Logger::default())
            .configure(listsync_client_rust::config)
    })
//...

I would like to extract the code highlighted above, the creation of an App, into a separate function, like this:

fn new_app() -> ??? {
    App::new()
        .wrap(listsync_client_rust::cookie_session::new_session())
        .wrap(middleware::Logger::default())
        .configure(listsync_client_rust::config)
}
//...
    HttpServer::new(|| {
        new_app()
    })

Simple, right? To find out what the return type of the function should be, I can just make a bad guess, and get the compiler to tell me what I did wrong. In this case, I will guess by changing the question marks above into i32, and run cargo test. I get quite a few errors, one of which is:

error[E0277]: the trait bound `i32: actix_service::IntoServiceFactory<_>` is not satisfied
  --> src/bin/listsync-client-rust.rs:27:5
   |
27 | /     HttpServer::new(|| {
28 | |         new_app()
29 | |     })
   | |______^ the trait `actix_service::IntoServiceFactory<_>` is not implemented for `i32`
   |
   = note: required by `actix_web::server::HttpServer`

So the first problem I see is that the error message I am seeing is about the later code, and there are no errors about my new function.

I obviously went a little too fast. Let’s change the HttpServer::new code back to how it was before, and only make a new function new_app. Now I get an error that should help me:

error[E0308]: mismatched types
  --> src/bin/listsync-client-rust.rs:12:5
   |
11 |   fn new_app() -> i32 {
   |                   --- expected `i32` because of return type
12 | /     App::new()
13 | |         .wrap(listsync_client_rust::cookie_session::new_session())
14 | |         .wrap(middleware::Logger::default())
15 | |         .configure(listsync_client_rust::config)
   | |________________________________________________^ expected i32, found struct `actix_web::app::App`
   |
   = note: expected type `i32`
              found type `actix_web::app::App<impl actix_service::ServiceFactory, actix_web::middleware::logger::StreamLog<actix_http::body::Body>>`

So the compiler has told us what type we are returning! Let’s copy that into the type signature of the function:

use actix_service::ServiceFactory;
use actix_http::body::Body;
// ...
fn new_app() -> App<impl ServiceFactory, middleware::logger::StreamLog<Body>> {
// ...

The first error I get from the compiler is a distraction:

error[E0432]: unresolved import `actix_service`
 --> src/bin/listsync-client-rust.rs:1:5
  |
1 | use actix_service::ServiceFactory;
  |     ^^^^^^^^^^^^^ use of undeclared type or module `actix_service`

I can fix it by adding actix-service = "1.0.5" to Cargo.toml. (I found the version by looking in Cargo.lock, since this dependency was already implicitly used – I just need to make it explicit if I am going to use it directly.)

Once I do that I get the next error:

error[E0603]: module `logger` is private
  --> src/bin/listsync-client-rust.rs:13:54
   |
13 | fn new_app() -> App<impl ServiceFactory, middleware::logger::StreamLog<Body>> {
   |                                                      ^^^^^^

This leaves me a bit stuck: I can’t use StreamLog because it’s in a private module.

More importantly, it makes the point that I don’t actually want to be as specific as I am being: I don’t care what the exact type parameters for App are – I just want to return an App of some kind and have the compiler fill in the blanks. Ideally, if I change the body of new_app later, for example to add another wrap call that changes the type of App we are returning, I’d like to leave the return type the same and have it just work.

With that in mind, I took a look at the type that HttpServer::new takes in. Here is impl HttpServer:

impl<F, I, S, B> HttpServer<F, I, S, B> where
    F: Fn() -> I + Send + Clone + 'static,
    I: IntoServiceFactory<S>,
    S: ServiceFactory<Config = AppConfig, Request = Request>,
    S::Error: Into<Error> + 'static,
    S::InitError: Debug,
    S::Response: Into<Response<B>> + 'static,
    <S::Service as Service>::Future: 'static,
    B: MessageBody + 'static, 

and HttpServer::new looks like:

pub fn new(factory: F) -> Self

So it takes in a function which actually makes the App, and the type of that function is F, which is a Fn which returns a I + Send + Clone + 'static. From the declaration of impl HttpServer we can see that the type of I depends on S and B, which have quite complex types. Let’s paste the whole thing in:

use actix_http::{Error, Request, Response};
use actix_service::IntoServiceFactory;
use actix_web::body::MessageBody;
use actix_web::dev::{AppConfig, Service};
use core::fmt::Debug;
// ...
fn new_app<I, S, B>() -> I
where
    I: IntoServiceFactory<S> + Send + Clone + 'static,
    S: ServiceFactory<Config = AppConfig, Request = Request>,
    S::Error: Into<Error> + 'static,
    S::InitError: Debug,
    S::Response: Into<Response<B>> + 'static,
    <S::Service as Service>::Future: 'static,
    B: MessageBody + 'static,
{
    App::new()
        .wrap(listsync_client_rust::cookie_session::new_session())
        .wrap(middleware::Logger::default())
        .configure(listsync_client_rust::config)
}

Note that I had to modify I to include the extra requirements on the return type of F from the definition of HttpServer. (I think I did the right thing, but I’m not sure. If I just remove the + Send + Clone + 'static it seems to behave similarly.)

Now I get this error from the compiler:

error[E0308]: mismatched types
  --> src/bin/listsync-client-rust.rs:27:5
   |
17 |   fn new_app<I, S, B>() -> I
   |                            - expected `I` because of return type
...
27 | /     App::new()
28 | |         .wrap(listsync_client_rust::cookie_session::new_session())
29 | |         .wrap(middleware::Logger::default())
30 | |         .configure(listsync_client_rust::config)
   | |________________________________________________^ expected type parameter, found struct `actix_web::app::App`
   |
   = note: expected type `I`
              found type `actix_web::app::App<impl actix_service::ServiceFactory, actix_web::middleware::logger::StreamLog<actix_http::body::Body>>`
   = help: type parameters must be constrained to match other types
   = note: for more information, visit https://doc.rust-lang.org/book/ch10-02-traits.html#traits-as-parameters

The compiler really tries to help here, suggesting I read a chapter of the Rust Book, but even after reading it I could not figure out how to do what I am trying to do.

Can anyone help me?

Wouldn’t it be amazing if there were some way the compiler could give me easier-to-understand help to figure this out?

Converting HTML slides to a PDF with Firefox

I have not found an automated way to generate a nice PDF from some slides written in HTML – if you know of one please add a comment!

In the meantime, if you create slides using my HTML Slides template, then you can make a decent-ish-looking PDF like this:

  1. View your slides in Firefox and open the Print dialog (press Ctrl-p).
  2. Select Print to File and choose a filename to save to.
  3. Under Options deselect “Ignore Scaling…” and select “–blank–” for all the headers and footers. Ensure “Print Background Colours” and “Print Background Images” are selected.
  4. Under Page Setup set Scale to 70.0 and Orientation to Landscape.
  5. Select the dropdown next to Paper size and choose Manage Custom Sizes…
  6. Create a new custom size called “Screen 16:9” with Width 157.5 mm and Height 280.0 mm. Set all the Paper Margins to 5.0 mm. You can modify the name of the custom size by clicking on it in the list on the left.
    Click Close.
  7. Back in Page Setup, make sure your new custom size is selected next to Paper Size.
  8. Click Print.

If that does not work well for you, try experimenting with different Scale settings.

Support the Software Freedom Conservancy

The Software Freedom Conservancy helps Free/Open Source software projects by providing infrastructure, financial structures, and legal help. It is a not-for-profit organisation that is dedicated to software freedom, something that I think is an important prerequisite for a decent world in the future.

Conservancy looks after lots of projects, including these ones that I personally use: Boost, BusyBox, Etherpad, Git, Godot Engine, Inkscape, phpMyAdmin, QEMU, Racket, Samba, Selenium, Squeak and Wine.

It provides an easy way for projects to accept donations, hold assets and negotiate contracts, as well as mentoring and legal advice. It also leads the difficult and thankless work to persuade (and force) companies to comply with the GPL (a Free Software license). Without this work a great deal more Free Software would be distributed without the required access to source code, meaning it is not free at all.

Further, it is a key supporter of Outreachy, which does important work to address the under-representation, systemic bias, and discrimination in the technology industry.

I encourage you to join me and support the Software Freedom Conservancy.

You might also like to listen to the Free As In Freedom podcast.

Coding workshop example worksheets

This week we did a coding workshop exercise: 2 teams implemented the different sides of the SMPP protocol (without speaking to each other) and this morning we tried out connecting them together.

We successfully sent a message and received an acknowledgement!

It was a lot of fun and I we learned a surprising amount about SMPP (and quite how … interesting … the standard is).

In case they’re useful to anyone, here are the worksheets I made up: Team 1 ODT, Team 1 PDF, Team 2 ODT, Team 2 PDF.

Idea for a team who are less interested in SMPP (!) – try a similar exercise implementing FTP, which is a nice simple text-based protocol. I did this once and found it extremely rewarding.

Building an all-in-one Jar in Gradle with the Kotlin DSL

To build a “fat” Jar of your Java or Kotlin project that contains all the dependencies within a single file, you can use the shadow Gradle plugin.

I found it hard to find clear documentation on how it works using the Gradle Kotlin DSL (with a build.gradle.kts instead of build.gradle) so here is how I did it:

$ cat build.gradle.kts 
import com.github.jengelman.gradle.plugins.shadow.tasks.ShadowJar

plugins {
    kotlin("jvm") version "1.3.41"
    id("com.github.johnrengelman.shadow") version "5.1.0"
}

repositories {
    mavenCentral()
}

dependencies {
    implementation(kotlin("stdlib"))
}

tasks.withType<ShadowJar>() {
    manifest {
        attributes["Main-Class"] = "HelloKt"
    }
}

$ cat src/main/kotlin/Hello.kt 
fun main() {
    println("Hello!")
}

$ gradle wrapper --gradle-version 5.5
BUILD SUCCESSFUL in 0s
1 actionable task: 1 executed

$ ./gradlew shadowJar
BUILD SUCCESSFUL in 1s
2 actionable tasks: 2 executed

$ java -jar build/libs/hello-all.jar 
Hello!

Creating a self-signed certificate for Apache and connecting to it from Java

Our mission: to create a self-signed certificate for an Apache web server that allows us to connect to it over HTTPS (SSL/TLS) from a Java program.

The tricky bit for me was generating a certificate that contains Subject Alternative Names for my server, which is needed to connect to it from Java.

We will use the openssl command.

Creating a self-signed certificate for Apache HTTPD

First create a config file cert.conf:

[ req ]
distinguished_name  = subject
x509_extensions     = x509_ext
prompt = no

[ subject ]
commonName = Example Company

[ x509_ext ]
subjectAltName = @alternate_names

[ alternate_names ]
DNS.1 = example.com

In the above, replace “example.com” with the name you will use for the host when you connect from Java. This is important, because Java requires the name in the certificate to match the name it is using to connect to the server. If you’re connecting to it as localhost, just put “localhost”. Note: do not include “https://” or any port or path after the hostname, so “example.com:8080/mypath” is wrong – it should be just “example.com”.

The alternate_names section above gives the “Subject Alternative Names” for this certificate. You can add more as “DNS.2”, “DNS.3”, etc.

Next, generate the server key and self-signed certificate:

openssl genrsa 2048 > server.key
chmod 400 server.key
openssl req -new -x509 -config cert.conf -nodes -sha256 -days 365 -key server.key -out server.crt

Now you have two new files: server.key and server.crt. These are the files that will be used by Apache HTTPD, so put them somewhere useful (e.g. inside /usr/local/apache2/conf/) and refer to them in the Apache config file using keys “SSLCertificateKeyFile” and “SSLCertificateFile” respectively. For more info see the SSL/TLS How-To.

Checking the certificate is being used

Start up your Apache and ensure you can connect to it over HTTPS using curl:

curl -v --insecure https://example.com:8080

Replace “https://example.com:8080” above with the full URL (this time, include “https://” and the port and path.

To examine the certificate that is being returned, run:

openssl s_client -showcerts -connect example.com:8080

Replace “example.com:8080” above with hostname and port (no “https:// this time!).

Connecting from Java

To be able to connect from Java, we need a Trust Store. We can create one in PKCS#12 format with:

openssl pkcs12 -export -passout pass:000000 -out trust.pkcs12 -inkey server.key -in server.crt

Note: Java 8 onwards is able to use .pkcs12 (PKCS#12) files for its trust store. The old .jks (Java Key Store) format can also be used, but is deprecated.

Now you have a file we can use as a trust store, follow my other article to connect from Java over HTTPS with a self-signed certificate.

Build with a different Java version (e.g. 11) using Docker

To spin up a temporary environment with a different Java version without touching your real environment, try this Docker command:

docker run -i -t --mount "type=bind,src=$PWD,dst=/code" openjdk:11-jdk bash

(Change “11-jdk” to the version you want as listed on the README.)

Then you can build the code inside the current directory something like this:

cd code
./gradlew test

Or similar for other build tools, although you may need to install them first.

Scheduling a task in Java within a CompletableFuture

When we want to do something later in our Java code, we often turn to the ScheduledExecutorService. This class has a method called schedule(), and we can pass it some code to be run later like this:

ScheduledExecutorService executor =
    Executors.newScheduledThreadPool(4);
executor.schedule(
    () -> {System.out.println("..later");},
    1,
    TimeUnit.SECONDS
);
System.out.println("do...");
// (Don't forget to shut down the executor later...)

The above code prints “do…” and then one second later it prints “…later”.

We can even write code that does some work and returns a result in a similar way:

// (Make the executor as above.)
ScheduledFuture future = executor.schedule(
    () -> 10 + 25, 1, TimeUnit.SECONDS);
System.out.println("answer=" + future.get())

The above code prints “answer=35”. When we call get() it blocks waiting for the scheduler to run the task and mark the ScheduledFuture as complete, and then returns the answer to the sum (10 + 25) when it is ready.

This is all very well, but you may note that the Future returned from schedule() is a ScheduledFuture, and a ScheduledFuture is not a CompletableFuture.

Why do you care? Well, you might care if you want to do something after the scheduled task is completed. Of course, you can call get(), and block, and then do something, but if you want to react asynchronously without blocking, this won’t work.

The normal way to run some code after a Future has completed is to call one of the “then*” or “when*” methods on the Future, but these methods are only available on CompletableFuture, not ScheduledFuture.

Never fear, we have figured this out for you. We present a small wrapper for schedule that transforms your ScheduledFuture into a CompletableFuture. Here’s how to use it:

CompletableFuture future =
    ScheduledCompletable.schedule(
        executor,
        () -> 10 + 25,
        1,
        TimeUnit.SECONDS
     );
future.thenAccept(
    answer -> {System.out.println(answer);}
);
System.out.println("Answer coming...")

The above code prints “Answer coming…” and then “35”, so we can see that we don’t block the main thread waiting for the answer to come back.

So far, we have scheduled a synchronous task to run in the background after a delay, and wrapped its result in a CompletableFuture to allow us to chain more tasks after it.

In fact, what we often want to do is schedule a delayed task that is itself asynchronous, and already returns a CompletableFuture. In this case it feels particularly natural to get the result back as a CompletableFuture, but with the current ScheduledExecutorService interface we can’t easily do it.

It’s OK, we’ve figured that out too:

Supplier> asyncTask = () ->
    CompletableFuture.completedFuture(10 + 25);
CompletableFuture future =
    ScheduledCompletable.scheduleAsync(
        executor, asyncTask, 1, TimeUnit.SECONDS);
future.thenAccept(
    answer -> {System.out.println(answer);}
);
System.out.println("Answer coming...")

The above code prints “Answer coming…” and then “35”, so we can see that our existing asynchronous task was scheduled in the background, and we didn’t have to block the main thread waiting for it. Also, under the hood, we are not blocking the ScheduledExecutorService‘s thread (from its pool) while the async task is running – that task just runs in whatever thread it was assigned when it was created. (Note: in our example we don’t really run an async task at all, but just immediately return a completed Future, but this does work for real async tasks.)

I know you’re wondering how we achieved all this. First, here’s how we run a simple blocking task in the background and wrap it in a CompletableFuture:

public static  CompletableFuture schedule(
    ScheduledExecutorService executor,
    Supplier command,
    long delay,
    TimeUnit unit
) {
    CompletableFuture completableFuture = new CompletableFuture<>();
    executor.schedule(
        (() -> {
            try {
                return completableFuture.complete(command.get());
            } catch (Throwable t) {
                return completableFuture.completeExceptionally(t);
            }
        }),
        delay,
        unit
    );
    return completableFuture;
}

And here’s how we delay execution of an async task but still return its result in a CompletableFuture:

public static  CompletableFuture scheduleAsync(
    ScheduledExecutorService executor,
    Supplier> command,
    long delay,
    TimeUnit unit
) {
    CompletableFuture completableFuture = new CompletableFuture<>();
    executor.schedule(
        (() -> {
            command.get().thenAccept(
                t -> {completableFuture.complete(t);}
            )
            .exceptionally(
                t -> {completableFuture.completeExceptionally(t);return null;}
            );
        }),
        delay,
        unit
    );
    return completableFuture;
}

Note that this should all work to run methods like exceptionally(), thenAccept(), whenComplete() etc.

Feedback and improvements welcome!

Convert a video to a GIF with reasonable colours

Here’s a little script I wrote to avoid copy-pasting the ffmpeg command from superuser every time I needed it.

It converts a video to a GIF file by pre-calculating a good palette, then using that palette.

Usage:

./to_gif input.mp4 output.gif

The file to_gif (which should be executable):

#!/bin/bash

set -e
set -u

# Credit: https://superuser.com/questions/556029/how-do-i-convert-a-video-to-gif-using-ffmpeg-with-reasonable-quality

INPUT="$1"
OUTPUT="$2"

PALETTE=$(tempfile --suffix=.png)

ffmpeg -y -i "${INPUT}" -vf palettegen "${PALETTE}"
ffmpeg -y -i "${INPUT}" -i "${PALETTE}" \
    -filter_complex "fps=15,paletteuse" "${OUTPUT}"

rm -f "${PALETTE}"

Note: you might want to modify the number after fps= to adjust how fast the video plays.

Gradle: what is a task, and how can I make a task depend on another task?

In an insane world, Gradle sometimes seems like the sanest choice for building a Java or Kotlin project.

But what on Earth does all the stuff inside build.gradle actually mean?

And when does my code run?

And how do you make a task?

And how do you persuade a task to depend on another task?

[Related: Clever things people do in Groovy so you have to know about them]

Setting up

To use Gradle, get hold of any version of it for long enough to create a local gradlew file, and the use that.

$ mkdir gradle-experiments
$ cd gradle-experiments
$ sudo apt install gradle  # Briefly install the system version of gradle
...
$ gradle wrapper --gradle-version=5.2.1
$ sudo apt remove gradle   # Optional - uninstalls the system version
$ ./gradlew tasks
... If all is good, this should ...
... print a list of available tasks. ...

It is normal for gradlew and the whole gradle/ directory it creates to be checked into source control. This means everyone who fetches the code from source control will have a predictable Gradle version.

What is build.gradle?

build.gradle is a Groovy program that Gradle runs within a context that it has set up for you. That context means that you are actually calling methods of a Project object, and modifying its properties. The fact that Groovy lets you miss out a lot of punctuation makes that harder to see, but it’s true.

The first thing to get your head around is that Gradle actually runs your code immediately, so if your build.gradle looks like this (and only this):

println("Hello")

when you run Gradle your code runs:

$ ./gradlew -q 
Hello
... more guff ...

So that code runs even if you don’t ask Gradle to run a task containing that code. It runs at “configuration time” – i.e. when Gradle is understanding your build.gradle file. Actually, “understanding” it means executing it.

Remember when I said this code runs in the context of a Project? What that means is that if you have something like this in your build.gradle:

repositories {
    jcenter()
}

what it really means is something like this:

project.repositories(
    {
        it.jcenter()
    }
)

You are calling the repositories method on the project object. The argument to the repositories method is a Groovy closure, which is a blob of code that will get run later. I’ve used the magic it name above to demonstrate that jcenter is just a method being called on the object that is the context for the closure when it is run.

When does it run? Let’s find out:

println("before")
project.repositories( {
    println("within")
    jcenter()
})
println("after")
$ ./gradlew -q
before
within
after
... more guff ...

This surprised me – it means the closure you pass in to repositories is actually run immediately, as part of running repositories, before execution gets to the line after that call.

As we’ll see later, some closures you create do not run immediately like this one.

Once you know that build.gradle is actually modifying a Project object, you have starting point for understanding the Gradle reference documentation.

How do you make a task?

You probably shouldn’t do it very often, but it was instructive for me to understand how to make my own custom task. Here’s an example:

tasks.register("mytask") {
    doLast {
        println("running mytask")
    }
}

This creates a new task by calling the register method on the tasks property of the Project object. Register takes two arguments: a name for the task (“mytask” here), and a closure with some code in it to run when we decide we need this task. That closure gets run in a context that can’t see the Project object, but instead can see a Task object which it is helping to make. That Task object has a doLast method that we call, passing it a closure that will be run when the task is actually executed (not immediately).

If we remove some of the syntactic sugar the above build.gradle looks like this:

tasks.register(
    "mytask",
    {
        it.doLast(
            {
                println("running mytask")
            }
        )
    }
)

Above we can see that register really does take two arguments as I said above – the first version uses a Groovy feature where if you miss out the last argument and write a closure immediately afterwards the closure is passed as the last argument. Confusing, eh?

Again, notice that doLast is a method on the Task object that is implicitly available when the closure is run.

So we have created a task that we can run:

 ./gradlew -q mytask
running mytask

How do you make a task depend on another task?

If I want to run my code formatting before my compile (for example) I sometimes need to modify a task to make it depend on another one. This can done for tasks you create or for pre-existing ones. Here’s an example:

plugins {
    id "java"
}
tasks.register("mytask") {
    doLast {
        println("running mytask")
    }
}
compileJava {
    dependsOn tasks.named("mytask")
}

So, calling the plugins method on the Project at the top with a closure that ran the id method on something modified the Project so that it had a new method called compileJava which we called at the bottom, passing it a closure to run. That closure ran in the context of a Task object (similar to when we created a task, but now allow us to modify a pre-existing one). We called the dependsOn method of the Task object, passing in another Task object which we had got by calling the named method on the tasks object.

[Side note: the register method actually returns a Task object that we could have passed to dependsOn without looking it up again using named, but Groovy doesn’t provide a very convenient way of holding on to that reference, so we didn’t do it. The Kotlin example below shows that this is quite simple in Kotlin.]

How do I do all this in Kotlin?

Because one DSL that hides what’s really going on wasn’t enough for you, Gradle now provides a second DSL that hides what’s going on in subtly different ways, which is a program written in Kotlin instead of Groovy. This is marginally better, because Kotlin doesn’t let you do quite so many stupid tricks as Groovy does.

Below are all our examples in Kotlin. You get started exactly the same way, by following “Setting up” above. Remember to name your build file build.gradle.kts.

Say hello in Gradle Kotlin

println("Hello")

This is identical to the Groovy version.

Use jcenter repo in Gradle Kotlin

repositories {
    jcenter()
}

This is identical to the Groovy version, and with the same meaning: repositories is a method on the implicitly-available Project object.

The “unsugared” version looks like this in Kotlin:

this.repositories(
    {
        this.jcenter()
    }
)

[Note that the word this is used to access the implicit context. The word it has a different meaning in Kotlin from in Groovy. In Groovy it means the implicit context, but in Kotlin it means the first argument. We didn’t pass any arguments to jcenter when we called it, so we can’t use it, but we were being run in a context, which we can refer to using this. Simple. huh?]

Execution order in Gradle Kotlin

We this build.gradle.kts:

println("before")
project.repositories( {
    println("within")
    jcenter()
})
println("after")

We see this behaviour:

$ ./gradlew -q
before
within
after

which is all identical to the Groovy version.

Making a new task in Gradle Kotlin

tasks.register("mytask") {
    doLast {
        println("running mytask")
    }
}

Notice that Kotlin lets you do the same trick as Groovy: providing an extra argument to a function that is a closure by writing it immediately after it looks like you’ve finished calling it. It’s good for people who dislike closing brackets hanging around longer than they’re welcome. As someone who likes Lisp, I’m OK with closing brackets, but what do I know?

The above is identical to the Groovy version, but slightly different when unsugared:

tasks.register(
    "mytask",
    {
        this.doLast(
            {
                println("running mytask")
            }
        )
    }
)

One task depending on another in Gradle Kotlin

plugins {
    java
}
val mytask = tasks.register("mytask") {
    doLast {
        println("running mytask")
    }
}
tasks.compileJava {
    dependsOn(mytask)
}

This differs slightly from the Groovy version, even though the meaning is the same: we start off in the context of a Project object that we call methods on.

The code to make one task depend on another gets hold of the Task object called compileJava from inside the tasks property of the Project, and calls it (because it’s a callable object). We pass in a closure that runs in the context of this Task object, calling its dependsOn method, and passing in a reference to the mytask object, which is a Task and was created in the code above.

Corrections and clarifications welcome

The above is what I have worked out by experimentation and trying to read the Gradle documentation. Please add comments that clear up confusions and correct mistakes.

Keybase chat bot in 10 lines of bash

I’ve been getting very excited about keybase.io recently, not least because it offers secure conversation, and you can have bots.

I wrote a quick bot to simulate Arnold Schwarzenegger which I thought I’d share to demonstrate how easy it is. It is based on the keybase command line tool (which comes with the desktop client as standard) and jq, the brilliant command-line JSON manipulator.

For this to work, you need to have the keybase command installed and working, and you need jq.

Here’s the bot:

#!/bin/bash
CHANNEL=mychannel
keybase chat api-listen | while read L; do
{
    OUT=$(jq --raw-output 'select(.type == "chat")|select(.msg.content.text.body|startswith("!arnie "))| .msg.content.text.body | "*" + ltrimstr("!arnie ") + "*"' <<< "$L")
    if [ "${OUT}" != "" ]; then
    {
        keybase chat send "${CHANNEL}" "${OUT}"
    }; fi
}; done

and here's it working:

andy> !arnie Do eet do eet now!!!
andy> Do eet do eet now!!!

Note: here the bot is pretending to be me. To do this nicely, you will want a different account for the bot, but you get the idea.

Obviously, I am now working on a comprehensive bot framework in Rust. Watch this space.

Performance of Java 2D drawing operations (part 3: image opacity)

Series: operations, images, opacity

Not because I was enjoying it, I seemed compelled to continue my quest to understand the performance of various Java 2D drawing operations. I’m hoping to make my game Rabbit Escape faster, especially on the Raspberry Pi, so you may see another post sometime actually trying this stuff out on a Pi.

But for now, here are the results of my investigation into how different patterns of opacity in images affects rendering performance.

You can find the code here: gitlab.com/andybalaam/java-2d-performance.

Results

  • Images with partially-opaque pixels are no slower than those with fully-opaque pixels
  • Large transparent areas in images are drawn quite quickly, but transparent pixels mixed with non-transparent are slow

Advice

  • Still avoid any transparency whenever possible
  • It’s relatively OK to use large transparent areas on images (e.g. a fixed-size animation where a character moves through the image)
  • Don’t bother restricting pixels to be either fully transparent or fully opaque – partially-opaque is fine

Opacity patterns in images

Non-transparent images drew at 76 FPS, and transparent ones dropped to 45 FPS.

I went further into investigating transparency by creating images that were:

  • All pixels 50% opacity (34 FPS)
  • Half pixels 0% opacity, half 100%, mixed up (34 FPS)
  • Double the size of the original image, but the extra area is fully transparent, and the original area is non-transparent (41 FPS)

I concluded that partial-opacity is not important to performance compared with full-opacity, but that large areas of transparency are relatively fast compared with images with complex patterns of transparency and opacity.

Numbers

Transparency and opacity

Test FPS
large nothing 90
large images20 largeimages 76
large images20 largeimages transparentimages 45
large images20 largeimages transparent50pcimages 34
large images20 largeimages transparent0pc100pcimages 34
large images20 largeimages transparentareaimages 41

Feedback please

Please do get back to me with tips about how to improve the performance of my experimental code.

Feel free to log issues, make merge requests or add comments to the blog post.

Performance of Java 2D drawing operations (part 2: image issues)

Series: operations, images

In my previous post I examined the performance of various drawing operations in Java 2D rendering. Here I look at some specifics around rendering images, with an eye to finding optimisations I can apply to my game Rabbit Escape.

You can find the code here: gitlab.com/andybalaam/java-2d-performance.

Results

  • Drawing images with transparent sections is very slow
  • Drawing one large image is slower than drawing many small images covering the same area(!)
  • Drawing images outside the screen is slower than not drawing them at all (but faster than drawing them onto a visible area)

Advice

  • Avoid transparent images where possible
  • Don’t bother pre-rendering your background tiles onto a single image
  • Don’t draw images that are off-screen

Images with transparency

All the images I used were PNG files with a transparency layer, but in most of my experiments there were no transparent pixels. When I used images with transparent pixels the frame rate was much slower, dropping from 78 to 46 FPS. So using images with transparent pixels causes a significant performance hit.

I’d be grateful if someone who knows more about it can recommend how to improve my program to reduce this impact – I suspect there may be tricks I can do around setComposite or setRenderingHint or enabling/encouraging hardware acceleration.

Composite images

I assumed that drawing a single image would be much faster than covering the same area of the screen by drawing lots of small images. In fact, the result was the opposite: drawing lots of small images was much faster than drawing a single image covering the same area.

The code for a single image is:

g2d.drawImage(
    singleLargeImage,
    10,
    10,
    null
)

and for the small images it is:

for (y in 0 until 40)
{
    for (x in 0 until 60)
    {
        g2d.drawImage(
            compositeImages[(y*20 + x) % compositeImages.size],
            10 + (20 * x),
            10 + (20 * y),
            null
        )
    }
}

The single large image was rendered at 74 FPS, whereas covering the same area using repeated copies of 100 images was rendered at 80 FPS. I ran this test several times because I found the result surprising, and it was consistent every time.

I have to assume some caching (possibly via accelerated graphics) of the small images is the explanation.

Drawing images off the side of the screen

Drawing images off the side of the screen was faster than drawing them in a visible area, but slower than not drawing them at all. I tested this by adding 10,000 to the x and y positions of the images being drawn (I also tested subtracting 10,000 with similar results). Not drawing any images ran at 93 FPS, drawing images on-screen at 80 FPS, and drawing them off-screen only 83 FPS, meaning drawing images off the side takes significant time.

Advice: check whether images are on-screen, and avoid drawing them if not.

Numbers

Transparency

Test FPS
large nothing 95
large images20 largeimages 78
large images20 largeimages transparentimages 46

Composite images

(Lots of small images covering an area, or a single larger image.)

Test FPS
large nothing 87
large largesingleimage 74
large compositeimage 80

Offscreen images

Test FPS
large nothing 93
large images20 largeimages 80
large images20 largeimages offscreenimages 83

Feedback please

Please do get back to me with tips about how to improve the performance of my experimental code.

Feel free to log issues, make merge requests or add comments to the blog post.

Performance of Java 2D drawing operations

I want to remodel the desktop UI of my game Rabbit Escape to be more
convenient and nicer looking, so I took a new look at game-loop-style graphics rendering onto a canvas in a Java 2D (Swing) UI.

Specifically, how fast can it be, and what pitfalls should I avoid when I’m doing it?

Results

  • Larger windows are (much) slower
  • Resizing images on-the-fly is very slow, even if they are the same size every time
  • Drawing small images is fast, but drawing large images is slow
  • Drawing rectangles is fast
  • Drawing text is fast
  • Drawing Swing widgets in front of a canvas is fast
  • Creating fonts on-the-fly is a tiny bit slow

Code

You can find the full code (written in Kotlin) at gitlab.com/andybalaam/java-2d-performance.

Basically, we make a JFrame and a Canvas and tell them not to listen to repaints (i.e. we control their drawing).

val app = JFrame()
app.ignoreRepaint = true
val canvas = Canvas()
canvas.ignoreRepaint = true

Then we add any buttons to the JFrame, and the canvas last (so it displays behind):

app.add(button)
app.add(canvas)

Now we make the canvas double-buffered and get hold of a buffer image for it:

app.isVisible = true
canvas.createBufferStrategy(2)
val bufferStrategy = canvas.bufferStrategy
val bufferedImage = GraphicsEnvironment
    .getLocalGraphicsEnvironment()
    .defaultScreenDevice
    .defaultConfiguration
    .createCompatibleImage(config.width, config.height)

Then inside a tight loop we draw onto the buffer image:

val g2d = bufferedImage.createGraphics()
try
{
    g2d.color = backgroundColor
    g2d.fillRect(0, 0, config.width, config.height)

    ... the different drawing operations go here ...

and then swap the buffers:

    val graphics = bufferStrategy.drawGraphics
    try {
        graphics.drawImage(bufferedImage, 0, 0, null)
        if (!bufferStrategy.contentsLost()) {
            bufferStrategy.show()
        }
    } finally {
        graphics.dispose()
    }
} finally {
    g2d.dispose()
}

Results

Baseline: some rectangles

I decided to compare everything against drawing 20 rectangles at random points on the screen, since that seems like a minimal requirement for a game.

My test machine is an Intel Core 2 Duo E6550 2.33GHz with 6GB RAM and a GeForce GT 740 graphics card (I have no idea whether it is being used here – I assume not). I am running Ubuntu 18.04.1 Linux, OpenJDK Java 1.8.0_191, and Kotlin 1.3.20-release-116. (I expect the results would be identical if I were using Java rather than Kotlin.)

I ran all the tests in two window sizes: 1600×900 and 640×480. 640.×480 was embarrassingly fast for all tests, but 1600×900 struggled with some of the tasks.

Drawing rectangles looks like this:

g2d.color = Color(
    rand.nextInt(256),
    rand.nextInt(256),
    rand.nextInt(256)
)
g2d.fillRect(
    rand.nextInt(config.width / 2),
    rand.nextInt(config.height / 2),
    rand.nextInt(config.width / 2),
    rand.nextInt(config.height / 2)
)

In the small window, the baseline (20 rectangles) ran at 553 FPS. In the large window it ran at 87 FPS.

I didn’t do any statistics on these numbers because I am too lazy. Feel free to do it properly and let me know the results – I will happily update the article.

Fewer rectangles

When I reduced the number of rectangles to do less drawing work, I saw small improvements in performance. In the small window, drawing 2 rectangles instead of 20 increased the frame rate from 553 to 639, but there is a lot of noise in those results, and other runs were much closer. In the large window, the same reduction improved the frame rate from 87 to 92. This is not a huge speed-up, showing that drawing rectangles is pretty fast.

Adding fixed-size images

Drawing pre-scaled images looks like this:

g2d.drawImage(
    image,
    rand.nextInt(config.width),
    rand.nextInt(config.height),
    null
)

When I added 20 small images (40×40 pixels) to be drawn in each frame, the performance was almost unchanged. In the small window, the run showing 20 images per frame (as well as rectangle) actually ran faster than the one without (561 FPS versus 553), suggesting the difference is negligible and I should do some statistics. In the large window, the 20 images version ran at exactly the same speed (87 FPS).

So, it looks like drawing small images costs almost nothing.

When I moved to large images (400×400 pixels), the small window slowed down from 553 to 446 FPS, and the large window slowed from 87 to 73 FPS, so larger images clearly have an impact, and we will need to limit the number and size of images to keep the frame rate acceptable.

Scaling images on the fly

You can scale an image on the fly as you draw onto a Canvas. (Spoiler: don’t do this!)

My code looks like:

val s = config.imageSize
val x1 = rand.nextInt(config.width)
val y1 = rand.nextInt(config.height)
val x2 = x1 + s
val y2 = y1 + s
g2d.drawImage(
    unscaledImage,
    x1, y1, x2, y2,
    0, 0, unscaledImageWidth, unscaledImageHeight,
    null
)

Note the 10-argument form of drawImage is being used. You can be sure you have avoided this situation if you use the 4-argument form from the previous section.

Note: the resulting image is the same size every time, and the Java documentation implies that scaled images may be cached by the system, but I saw a huge slow-down when using the 10-argument form of drawImage above.

On-the-fly scaled images slowed the small window from 446 to 67 FPS(!), and the large window from 73 to 31 FPS, meaning the exact same rendering took over twice as long.

Advice: check you are not using one of the drawImage overloads that scales images! Pre-scale them yourself (e.g. with getScaledInstance as I did here).

Displaying text

Drawing text on the canvas like this:

g2d.font = Font("Courier New", Font.PLAIN, 12)
g2d.color = Color.GREEN
g2d.drawString("FPS: $fpsLastSecond", 20, 20 + i * 14)

had a similar impact to drawing small images – i.e. it only affected the performance very slightly and is generally quite fast. The small window slowed from 553 to 581 FPS, and the large window from 87 to 88.

Creating the font every time (as shown above) slowed the process a little more, so it is worth moving the font creating out of the game loop and only doing it once. The slowdown just for creating the font was 581 to 572 FPS in the small window, and 88 to 86 FPS in the large.

Swing widgets

By adding Button widgets to the JFrame before the Canvas, I was able to display them in front. Their rendering and focus worked as expected, and they had no impact at all on performance.

The same was true when I tried adding these widgets in front of images rendered on the canvas (instead of rectangles).

Turning everything up to 11

When I added everything I had tested all at the same time: rectangles, text with a new font every time, large unscaled images, and large window, the frame rate reduced to 30 FPS. This is a little slow for a game already, and if we had more images to draw it could get even worse. However, when I pre-scaled the images the frame rate went up to 72 FPS, showing that Java is capable of running a game at an acceptable frame rate on my machine, so long as we are careful how we use it.

Numbers

Small window (640×480)

Test FPS
nothing 661
rectangles2 639
rectangles20 553
rectangles20 images2 538
rectangles20 images20 561
rectangles20 images20 largeimages 446
rectangles20 images20 unscaledimages 343
rectangles20 images20 largeimages unscaledimages 67
rectangles20 text2 582
rectangles20 text20 581
rectangles20 text20 newfont 572
rectangles20 buttons2 598
rectangles20 buttons20 612

Large window (1200×900)

Test FPS
large nothing 93
large rectangles2 92
large rectangles20 87
large rectangles20 images2 87
large rectangles20 images20 87
large rectangles20 images20 largeimages 73
large rectangles20 images20 unscaledimages 82
large rectangles20 images20 largeimages unscaledimages 31
large rectangles20 text2 89
large rectangles20 text20 88
large rectangles20 text20 newfont 86
large rectangles20 buttons2 88
large rectangles20 buttons20 87
large images20 buttons20 largeimages 74
large rectangles20 images20 text20 buttons20 largeimages newfont 72
large rectangles20 images20 text20 buttons20 largeimages unscaledimages newfont 30

Feedback please

Please do get back to me with tips about how to improve the performance of my experimental code.

Feel free to log issues, make merge requests or add comments to the blog post.

London Python Meetup January 2019 – Async Python and GeoPandas

It was a pleasure to go to the London Python Meetup organised by @python_london. There were plenty of friendly people and interesting conversations.

I gave a talk “Making 100 million requests with Python aiohttp” (slides, Blog post) explaining the basics of writing async code in Python 3 and how I used that to make a very large number of HTTP requests.

Andy giving the presentation

(Photo by CB Bailey.)

Hopefully it was helpful – there were several good questions, so I am optimistic that people were engaged with it.

After that, there was an excellent talk by Gareth Lloyd called “GeoPandas, the geospatial extension for Pandas” in which he explained how to use the very well-developed geo-spatial data tools available in the Python ecosphere to transform, combine, plot and analyse data which includes location information. I was really impressed with how easy the libraries looked to use, and also with the cool Jupyter notebook Gareth used to explain the ideas using live demos.

London Python Meetups seem like a cool place to meet Pythonistas of all levels of experience in a nice, low-pressure environment!

Meetup link: aiohttp / GeoPandas

Run bash inside any version of Linux using Docker

Docker is useful for some things, and not as useful as you think for others.

Here’s something massively useful: get a throwaway bash prompt inside any version of any Linux distribution in one command:

docker run -i -t --mount "type=bind,src=$HOME/Desktop,dst=/Desktop" ubuntu:18.10 bash

This command downloads a recent Ubuntu 18.10 image, mounts my desktop as /Desktop in the container, and gives me a bash prompt. From here I can install any packages I want and then use them.

For example, today I used it to decrypt a file that was encrypted with a cipher my main OS did not have a package for.

When I exit bash, the container stops and I can find it with docker ps -a then remove it with docker rm. To really clean up I can find the downloaded images with docker image ls and remove them with docker image rm.

New home page design

After years of getting around to it, I have redesigned my home page at artificialworlds.net.

It’s basically intended to make me look clever or productive or interesting or something. Alternatively, it gives you somewhere to find that thing you know I made but can’t find the link:

Screenshot of artificialworlds.net - colourful boxes with rich-coloured picture of drawers behind.

The background image is “Read where you are” by delaram bayat.

I am pleased with the page’s responsive design, clarity, fast page-load, and colourfulness.

What do you think?

Code Like a Girl T-shirts

There are lots of people missing from the programming world: lots of the programmers I meet look and sound a lot like me. I’d really like it if this amazing job were open to a lot more people.

One of the weird things that has happened is that somehow we seem to have the idea that programming is only for boys, and I’d like to fight against that idea by wearing a t-shirt demonstrating how cool I think it is to be a woman coder.

So, I commissioned a design from an amazing artist called Ellie Mars, who I found through her Mastodon.art page @elliemars@mastodon.art. She did an amazing job, sending sketches and ideas back and forth, and finally she came up with this awesome design:

I’ve printed a t-shirt for myself that I will give myself for Christmas, and I’ve made a page on Street Shirts so you can get one too!

I’ve uploaded 2 designs, but if you’d like me to set something different up for you, let me know.  Also, these links will expire, but I can re-set them up for you if you contact me.  They are reasonably cheap:

To ask for different designs or an unexpired link, you are welcome to contact me (via DM or publicly) on twitter @andybalaam or Mastodon @andybalaam@mastodon.social, or by email, via a short test.

Ellie and I agreed to set up these t-shirts sales with no profit for us because we’d like to get the word out.  If they are popular we might add a little, so get in fast for a good deal!

10 points for anyone who can recognise the code in the background.  It’s from one of my favourite programs.

Personally, I think we all spend too much of our time walking around advertising faceless corporations when we could be saying something a bit more useful on our clothes.  What do you think of this idea?  Maybe you could design a similar t-shirt?  Let me know your thoughts in the comments below or on Twitter or Mastodon.

Poor performance in Chrome (especially on mobile) – caused by SVG background images

I have spent the last few hours investigating abysmal performance in my latest little game project Cross The Road. Firefox was fine, but Chromium and Chrome, especially on mobile, was rendering at about three frames per second.

When I stopped using SVGs as background-images for my elements, and used PNGs instead, it improved to about 20-30 FPS.

It seems fine to use SVGs as normal images, but for background-image, it really hurt performance.

You must rewind your incoming buffer when you fail to encode a character in a CharsetEncoder or you’ll get an IllegalArgumentException

I am writing a CharsetEncoder in Java, which is my kind of fun.

I was getting a mysterious error when I identified that I could not encode certain characters:

Exception in thread "main" java.lang.IllegalArgumentException
	at java.nio.Buffer.position(Buffer.java:244)
	at java.nio.charset.CharsetEncoder.encode(CharsetEncoder.java:618)
	at java.nio.charset.CharsetEncoder.encode(CharsetEncoder.java:802)
	at java.nio.charset.Charset.encode(Charset.java:843)
	at java.nio.charset.Charset.encode(Charset.java:863)

After some investigation I realised the library code in Charset.encode was expecting me not to have consumed any characters of my incoming CharBuffer if I rejected the input by returning something like CoderResult.unmappableForLength.

Of course, in order to discover the input was unmappable, I did have to read it, but I made this problem go away by stepping back one char when I found an error like this:

@Override
public CoderResult encodeLoop(CharBuffer in, ByteBuffer out) {
    char ch = in.get();
    if(isUnmappable(ch)) {
        in.position(in.position() - 1);
        return CoderResult.unmappableForLength(2);
    }
    // ... rest of method ...

I hope this helps someone.

Godot: Dragging and dropping physics objects video

Series: 2D Shapes, drag and drop

Continuing to explore the Godot 3 game engine. I want to make a game where you drag blocks around and balance them on each other, but I couldn’t find much documentation on how to drag-and-drop objects (except menu UI elements), and especially I found quite a few wrinkles when doing this with objects that are normally controlled by the physics engine.

This time we actually write some code in Godot’s programming language, GDScript.

Godot version: v3.0.6.stable.official.8314054

Elm JSON decoder examples

I find JSON decoding in Elm confusing, so here are some thoughts and examples.

Setup

$ elm --version
0.19.0
$ mkdir myproj; cd myproj
$ elm init
...
$ elm install elm/json
...

To run the “Demo” parts of the examples below, type them into the interactive Elm interpreter. To try them out, start it like this:

$ elm repl

and import the library you need:

import Json.Decode as D

Scroll to “Concepts” at the bottom for lots of waffling about what is really going on, but if you’re looking to copy and paste concrete examples, here we are:

Examples

JSON object to Record

type alias MyRecord =
    { i : Int
    , s : String
    }

recordDecoder : D.Decoder MyRecord
recordDecoder =
    D.map2
        MyRecord
        (D.field "i" D.int)
        (D.field "s" D.string)

Note: this uses a special trick – it turns out you can construct a record like MyRecord writing e.g. MyRecord 3 "qux" – the arguments have to be in the same order that the properties are declared in the record. The above only works because you can do this.

Demo:

> type alias MyRec = {i: Int, s: String}
> myRecDec = D.map2 MyRec (D.field "i" D.int) (D.field "s" D.string)
<internals> : D.Decoder MyRec
> D.decodeString myRecDec "{\"i\": 3, \"s\": \"bar\"}"
Ok { i = 3, s = "bar" }
    : Result D.Error MyRec

JSON array of ints to List

intArrayDecoder : D.Decoder (List Int)
intArrayDecoder =
    D.list D.int

Demo:

> myArrDec = D.list D.int
<internals> : D.Decoder (List Int)
> D.decodeString myArrDec "[3, 4]"
Ok [3,4] : Result D.Error (List Int)

JSON array of strings to List

stringArrayDecoder : D.Decoder (List String)
stringArrayDecoder =
    D.list D.string

Demo:

> myArrDec2 = D.list D.string
<internals> : D.Decoder (List String)
> D.decodeString myArrDec2 "[\"a\", \"b\"]"
Ok ["a","b"] : Result D.Error (List String)

JSON object to Dict

intDictDecoder : D.Decoder (Dict String Int)
intDictDecoder =
    D.dict D.int

Demo:

> myDictDecoder = D.dict D.int
<internals> : D.Decoder (Dict.Dict String Int)
> D.decodeString myDictDecoder "{\"a\": \"b\"}"
Err (Field "a" (Failure ("Expecting an INT") <internals>))
    : Result D.Error (Dict.Dict String Int)
> D.decodeString myDictDecoder "{\"a\": 3}"
Ok (Dict.fromList [("a",3)])
    : Result D.Error (Dict.Dict String Int)

To build a Dict of String to String, replace D.int above with
D.string.

JSON array of objects to List of Records

type alias MyRecord =
    { i : Int
    , s : String
    }

recordDecoder : D.Decoder MyRecord
recordDecoder =
    D.map2
        MyRecord
        (D.field "i" D.int)
        (D.field "s" D.string)


listOfRecordsDecoder : D.Decoder (List MyRecord)
listOfRecordsDecoder =
    D.list recordDecoder

Demo:

> import Json.Decode as D
> type alias MyRec = {i: Int, s: String}
> myRecDec = D.map2 MyRec (D.field "i" D.int) (D.field "s" D.string)
<internals> : D.Decoder MyRec
> listOfMyRecDec = D.list myRecDec
<internals> : D.Decoder (List MyRec)
> D.decodeString listOfMyRecDec "[{\"i\": 4, \"s\": \"one\"}, {\"i\": 5, \"s\":\"two\"}]"
Ok [{ i = 4, s = "one" },{ i = 5, s = "two" }]
    : Result D.Error (List MyRec)

Concepts

What is a Decoder?

A Decoder is something that describes how to take in JSON and spit out something. The “something” part is written after Decoder, so e.g. Decoder Int describes how to take in JSON and spit out an Int.

The Json.Decode module contains a function that is a Decoder Int. It’s called int:

> D.int
<internals> : D.Decoder Int

In some not-all-all-true way, a Decoder is sort of like a function:

-- This is a lie, but just pretend with me for a sec
Decoder a : SomeJSON -> a
-- That was a lie

To actually run your a Decoder, provide it to a function like decodeString:

> D.decodeString D.int "45"
Ok 45 : Result D.Error Int

So the actually-true way of getting an actual function is to combine decodeString and a decoder like int:

> D.decodeString D.int
<function> : String -> Result D.Error Int

When you apply decodeString to int you get a function that takes in a String and returns either an Int or an error. The error could be because the string you passed was not valid JSON:

> D.decodeString D.int "foo bar"
Err (Failure ("This is not valid JSON! Unexpected token o in JSON at position 1") )
    : Result D.Error Int

or because the parsed JSON does not match what the Decoder you supplied expects:

> D.decodeString D.int "\"45\""
Err (Failure ("Expecting an INT") )
    : Result D.Error Int

(We supplied a String containing a JSON string, but the int Decoder expects to find a JSON int.)

Side note: ints and floats are treated as different, even though the JSON Spec treats them all as just “Numbers”:

> D.decodeString D.int "45.2"
Err (Failure ("Expecting an INT") )
    : Result D.Error Int

What is a Value?

Elm has a type that represents JSON that has been parsed (actually, parsed and stored in a JavaScript object) but not interpreted into a useful Elm type. You can make one using the functions inside Json.Encode:

> import Json.Encode as E
> foo = E.string "foo"
 : E.Value

You can even turn one of these into a String containing JSON using encode:

> E.encode 0 foo
"\"foo\"" : String

or interpret the Value as useful Elm types using decodeValue:

> D.decodeValue D.string foo
Ok "foo" : Result D.Error String

(When JSON values come from JavaScript, e.g. via flags, they actually come as Values, but you don’t usually need to worry about that.)

However, what you can’t do is pull Values apart in any way, other than the standard ways Elm gives you. So any custom Decoder that you write has to be built out of existing Decoders.

How do I write my own Decoder?

If you want to make a Decoder that does custom things, build it from the existing magic Decoders, give it a type that describes the type it outputs, and insert your code using one of the mapN functions.

For example, to decode only ints that are below 100:

> under100 i = if i < 100 then D.succeed i else (D.fail "Not under 100")
<function> : number -> D.Decoder number
> intUnder100 = D.int > D.andThen under100
<internals> : D.Decoder Int
> D.decodeString intUnder100 "50"
Ok 50 : Result D.Error Int
> D.decodeString intUnder100 "500"
Err (Failure ("Not under 100") <internals>)
    : Result D.Error Int

Here, we use the andThen function to transform the Int value coming from calling the int function into a Decoder Int that expresses success or failure in terms of decoding. When we do actual decoding using the decodeString funcion, this is transformed into the more familiar Result values like Ok or Err.

If you want to understand the above, pay close attention to the types of under100 and intUnder100.

If you want to write a Decoder that returns some complex type, you should build it using the mapN functions.

For example, to decode strings into arrays of words split by spaces:

> splitIntoWords = String.split " "
<function> : String -> List String
> words = D.map splitIntoWords D.string
<internals> : D.Decoder (List String)
> D.decodeString words "\"foo bar baz\""
Ok ["foo","bar","baz"]
    : Result D.Error (List String)

Above we used map to transform a Decoder String (the provided string function) into a Decoder (List String) by mapping it over a function (splitIntoWords) that transforms a String into a List String.

Again, to understand this, look carefully at the types of splitIntoWords
and words.

How do I build up complex Decoders?

Complex decoders are built by combining simple ones. Many functions that make decoders take another decoder as an argument. A good example is “JSON array of objects to List of Records” above – there we make a Decoder MyRecord and use it to decode a whole list of records by passing it as an argument to list, so that it returns a Decoder (List MyRecord) which can take in a JSON array of JSON objects, and return a List of MyRecords.

Why is this so confusing?

Because Decoders are not functions, but they feel like functions. In fact they are opaque descriptions of how to interpret JSON that the Elm runtime uses to make Elm objects for you out of Values, which are opaque objects that underneath represent a piece of parsed JSON.