Elm JSON decoder examples

I find JSON decoding in Elm confusing, so here are some thoughts and examples.

Setup

$ elm --version
0.19.0
$ mkdir myproj; cd myproj
$ elm init
...
$ elm install elm/json
...

To run the “Demo” parts of the examples below, type them into the interactive Elm interpreter. To try them out, start it like this:

$ elm repl

and import the library you need:

import Json.Decode as D

Scroll to “Concepts” at the bottom for lots of waffling about what is really going on, but if you’re looking to copy and paste concrete examples, here we are:

Examples

JSON object to Record

type alias MyRecord =
    { i : Int
    , s : String
    }

recordDecoder : D.Decoder MyRecord
recordDecoder =
    D.map2
        MyRecord
        (D.field "i" D.int)
        (D.field "s" D.string)

Note: this uses a special trick – it turns out you can construct a record like MyRecord writing e.g. MyRecord 3 "qux" – the arguments have to be in the same order that the properties are declared in the record. The above only works because you can do this.

Demo:

> type alias MyRec = {i: Int, s: String}
> myRecDec = D.map2 MyRec (D.field "i" D.int) (D.field "s" D.string)
<internals> : D.Decoder MyRec
> D.decodeString myRecDec "{\"i\": 3, \"s\": \"bar\"}"
Ok { i = 3, s = "bar" }
    : Result D.Error MyRec

JSON array of ints to List

intArrayDecoder : D.Decoder (List Int)
intArrayDecoder =
    D.list D.int

Demo:

> myArrDec = D.list D.int
<internals> : D.Decoder (List Int)
> D.decodeString myArrDec "[3, 4]"
Ok [3,4] : Result D.Error (List Int)

JSON array of strings to List

stringArrayDecoder : D.Decoder (List String)
stringArrayDecoder =
    D.list D.string

Demo:

> myArrDec2 = D.list D.string
<internals> : D.Decoder (List String)
> D.decodeString myArrDec2 "[\"a\", \"b\"]"
Ok ["a","b"] : Result D.Error (List String)

JSON object to Dict

intDictDecoder : D.Decoder (Dict String Int)
intDictDecoder =
    D.dict D.int

Demo:

> myDictDecoder = D.dict D.int
<internals> : D.Decoder (Dict.Dict String Int)
> D.decodeString myDictDecoder "{\"a\": \"b\"}"
Err (Field "a" (Failure ("Expecting an INT") <internals>))
    : Result D.Error (Dict.Dict String Int)
> D.decodeString myDictDecoder "{\"a\": 3}"
Ok (Dict.fromList [("a",3)])
    : Result D.Error (Dict.Dict String Int)

To build a Dict of String to String, replace D.int above with
D.string.

JSON array of objects to List of Records

type alias MyRecord =
    { i : Int
    , s : String
    }

recordDecoder : D.Decoder MyRecord
recordDecoder =
    D.map2
        MyRecord
        (D.field "i" D.int)
        (D.field "s" D.string)


listOfRecordsDecoder : D.Decoder (List MyRecord)
listOfRecordsDecoder =
    D.list recordDecoder

Demo:

> import Json.Decode as D
> type alias MyRec = {i: Int, s: String}
> myRecDec = D.map2 MyRec (D.field "i" D.int) (D.field "s" D.string)
<internals> : D.Decoder MyRec
> listOfMyRecDec = D.list myRecDec
<internals> : D.Decoder (List MyRec)
> D.decodeString listOfMyRecDec "[{\"i\": 4, \"s\": \"one\"}, {\"i\": 5, \"s\":\"two\"}]"
Ok [{ i = 4, s = "one" },{ i = 5, s = "two" }]
    : Result D.Error (List MyRec)

Concepts

What is a Decoder?

A Decoder is something that describes how to take in JSON and spit out something. The “something” part is written after Decoder, so e.g. Decoder Int describes how to take in JSON and spit out an Int.

The Json.Decode module contains a function that is a Decoder Int. It’s called int:

> D.int
<internals> : D.Decoder Int

In some not-all-all-true way, a Decoder is sort of like a function:

-- This is a lie, but just pretend with me for a sec
Decoder a : SomeJSON -> a
-- That was a lie

To actually run your a Decoder, provide it to a function like decodeString:

> D.decodeString D.int "45"
Ok 45 : Result D.Error Int

So the actually-true way of getting an actual function is to combine decodeString and a decoder like int:

> D.decodeString D.int
<function> : String -> Result D.Error Int

When you apply decodeString to int you get a function that takes in a String and returns either an Int or an error. The error could be because the string you passed was not valid JSON:

> D.decodeString D.int "foo bar"
Err (Failure ("This is not valid JSON! Unexpected token o in JSON at position 1") )
    : Result D.Error Int

or because the parsed JSON does not match what the Decoder you supplied expects:

> D.decodeString D.int "\"45\""
Err (Failure ("Expecting an INT") )
    : Result D.Error Int

(We supplied a String containing a JSON string, but the int Decoder expects to find a JSON int.)

Side note: ints and floats are treated as different, even though the JSON Spec treats them all as just “Numbers”:

> D.decodeString D.int "45.2"
Err (Failure ("Expecting an INT") )
    : Result D.Error Int

What is a Value?

Elm has a type that represents JSON that has been parsed (actually, parsed and stored in a JavaScript object) but not interpreted into a useful Elm type. You can make one using the functions inside Json.Encode:

> import Json.Encode as E
> foo = E.string "foo"
 : E.Value

You can even turn one of these into a String containing JSON using encode:

> E.encode 0 foo
"\"foo\"" : String

or interpret the Value as useful Elm types using decodeValue:

> D.decodeValue D.string foo
Ok "foo" : Result D.Error String

(When JSON values come from JavaScript, e.g. via flags, they actually come as Values, but you don’t usually need to worry about that.)

However, what you can’t do is pull Values apart in any way, other than the standard ways Elm gives you. So any custom Decoder that you write has to be built out of existing Decoders.

How do I write my own Decoder?

If you want to make a Decoder that does custom things, build it from the existing magic Decoders, give it a type that describes the type it outputs, and insert your code using one of the mapN functions.

For example, to decode only ints that are below 100:

> under100 i = if i < 100 then D.succeed i else (D.fail "Not under 100")
<function> : number -> D.Decoder number
> intUnder100 = D.int > D.andThen under100
<internals> : D.Decoder Int
> D.decodeString intUnder100 "50"
Ok 50 : Result D.Error Int
> D.decodeString intUnder100 "500"
Err (Failure ("Not under 100") <internals>)
    : Result D.Error Int

Here, we use the andThen function to transform the Int value coming from calling the int function into a Decoder Int that expresses success or failure in terms of decoding. When we do actual decoding using the decodeString funcion, this is transformed into the more familiar Result values like Ok or Err.

If you want to understand the above, pay close attention to the types of under100 and intUnder100.

If you want to write a Decoder that returns some complex type, you should build it using the mapN functions.

For example, to decode strings into arrays of words split by spaces:

> splitIntoWords = String.split " "
<function> : String -> List String
> words = D.map splitIntoWords D.string
<internals> : D.Decoder (List String)
> D.decodeString words "\"foo bar baz\""
Ok ["foo","bar","baz"]
    : Result D.Error (List String)

Above we used map to transform a Decoder String (the provided string function) into a Decoder (List String) by mapping it over a function (splitIntoWords) that transforms a String into a List String.

Again, to understand this, look carefully at the types of splitIntoWords
and words.

How do I build up complex Decoders?

Complex decoders are built by combining simple ones. Many functions that make decoders take another decoder as an argument. A good example is “JSON array of objects to List of Records” above – there we make a Decoder MyRecord and use it to decode a whole list of records by passing it as an argument to list, so that it returns a Decoder (List MyRecord) which can take in a JSON array of JSON objects, and return a List of MyRecords.

Why is this so confusing?

Because Decoders are not functions, but they feel like functions. In fact they are opaque descriptions of how to interpret JSON that the Elm runtime uses to make Elm objects for you out of Values, which are opaque objects that underneath represent a piece of parsed JSON.

Worksheet: “Tell a story by making animations with code”

I’m running a workshop at the Egham Raspberry Jam on 21st October. The workshop will introduce my little animation language Graft. We will tell a story using animations that we created ourselves using code.

The worksheet for the workshop is here: PDF or ODP.

I reproduced it as images below.

Any feedback much appreciated.

My experience upgrading to Elm 0.19

Elm is unstable, so upgrading to the next version can be painful. Here’s what I needed to do to upgrade from 0.18 to 0.19.

  • Replace elm-package.json and tests/elm-package.json with elm.json – e06f5a1728
  • Switch to the new elm-testb964b7c7a
  • Re-arrange Main, and how we call it from JavaScript – 0c118c49f
  • Stop using eeue56/elm-all-dict (since it’s not ported to 0.19 and porting it looked hard due to a lack of Debug.crash) – fe100f256
  • Replace toString with String.fromX or Debug.toString – 9e78163d0a3
  • Stop “shadowing” names by making new variables with the same name as another in the scope – 9688a621de
  • Adapt to the changed Html.style function – b991ab4f
  • Stop using Debug.crash – f98a70ad1
  • Adapt to the changes in the Regex module – 856762a4
  • Stop using tuples with more than 3 parts – 472c0bb7

The lack of Debug.crash is really, really painful, especially for a library like eeue56/elm-all-dict that has lots of invariants that are hard or impossible to enforce via the type system. On the other hand, if Elm can give a hard guarantee that there will be no runtime errors, this seems pretty cool. The problem is that some code may well have to return the wrong answer silently, instead of crashing, which could be much worse than crashing in some use-cases.

I was annoyed by the lack of more-than-3-part tuples, but even as I did the work to change my code, I saw it get better, so it’s hard to argue with.

The hardest part to work out was how to run the tests. Fortunately the tests themselves needed almost no changes. I just needed to do this:

rm -r tests/elm-stuff
rm tests/elm-package.json
sudo npm install -g elm-test@0.19.0-beta8
elm-test install
elm-test

My next job is to check out the –optimize compiler flag, and the advice on making the code smaller and faster.

Example of a systemd service file

Here is an almost-minimal example of a systemd service file, that I use to run the Mastodon bot of my generative art playground Graft.

I made a dedicated user just to run this service, and installed Graft into /home/graft/apps/graft under that username. Now, as root, I edited a file called /etc/systemd/service/graft.service and made it look like this:

[Service]
ExecStart=/home/graft/apps/graft/bot-mastodon
User=graft
Group=graft
WorkingDirectory=/home/graft/apps/graft/
[Install]
WantedBy=multi-user.target

Now I can start the graft service like any other service like this:

sudo systemctl start graft

and find out its status with:

sudo systemctl status graft

If I want it to run on startup I can do:

sudo systemctl enable graft

and it will. Easy!

If I want to look at its output, it’s:

sudo journalctl -u graft

As a reward for reading this far, here’s a little animation you can make with Graft:

Installing Flarum on Ubuntu 18.04

I am setting up a forum for sharing levels for my game Rabbit Escape, and I have decided to try and use Flarum, because it looks really usable and responsive, has features we need like liking posts and following authors, and I think it will be reasonably OK to write the custom features we want.

So, I want a dev environment on my local Ubuntu 18.04 machine, and the first step to that is a standard install.

Warning: at the time of writing the Flarum docs say it does not work with PHP 7.2, which is what is included with Ubuntu 18.04, so this may not work. (So far it looks OK for me.)

Here’s how I got it working:

sudo apt install \
    apache2 \
    libapache2-mod-php \
    mariadb-server \
    php-mysql \
    php-json \
    php-gd \
    php-tokenizer \
    php-mbstring \
    php-curl

php -r "copy('https://getcomposer.org/installer', 'composer-setup.php');"

# Get the next line from https://getcomposer.org/download/
# Don't copy it exactly!
php -r "if (hash_file('SHA384', 'composer-setup.php') === '544e09ee996cdf60ece3804abc52599c22b1f40f4323403c44d44fdfdd586475ca9813a858088ffbc1f233e9b180f061') { echo 'Installer verified'; } else { echo 'Installer corrupt'; unlink('composer-setup.php'); } echo PHP_EOL;"

mkdir $HOME/bin
php composer-setup.php --install-dir=$HOME/bin/ --filename=composer
rm composer-setup.php

cd /var/www/html
sudo mkdir flarum
sudo chown $(whoami) flarum

# Log out and in again here to get composer to be in your PATH
cd flarum
composer create-project flarum/flarum . --stability=beta

sudo chgrp -R www-data .
sudo chmod -R 775 .

sudo a2enmod rewrite 

Next I granted the permissions for Flarum’s .htaccess file to change settings by modifying /etc/apache2/apache2.conf and adding this after the existing <Directory entries:

<Directory "/var/www/html/flarum/">
    AllowOverride All
</Directory>

and restarted Apache:

sudo systemctl restart apache2

Now I created a database in MariaDB:

$ sudo mariadb
MariaDB [(none)]> CREATE USER flarumuser IDENTIFIED BY 'flarumpassword';
MariaDB [(none)]> CREATE DATABASE flarumdb;
MariaDB [(none)]> GRANT ALL ON flarumdb.* TO flarumuser;

and went to http://localhost/flarum in my browser, and entered these details:

MySQL host: localhost
MySQL database: flarumdb
MySQL user: flarumuser
MySQL password: flarumpassword
table prefix: flarum_

Admin user: admin
Admin email: <my email address>
Admin password: adminpassword

Once I typed all those correctly and submitted, I had to wait a while, and then after it printed some details I refreshed the page and saw a local Flarum ready to customise.

Next, I looked into how to get an extension development environment up and running.

If you want to find and share levels for Rabbit Escape, check up on our progress setting up the forum at https://artificialworlds.net/rabbit-escape/levels.