Convert a video to a GIF with reasonable colours

Here’s a little script I wrote to avoid copy-pasting the ffmpeg command from superuser every time I needed it.

It converts a video to a GIF file by pre-calculating a good palette, then using that palette.

Usage:

./to_gif input.mp4 output.gif

The file to_gif (which should be executable):

#!/bin/bash

set -e
set -u

# Credit: https://superuser.com/questions/556029/how-do-i-convert-a-video-to-gif-using-ffmpeg-with-reasonable-quality

INPUT="$1"
OUTPUT="$2"

PALETTE=$(tempfile --suffix=.png)

ffmpeg -y -i "${INPUT}" -vf palettegen "${PALETTE}"
ffmpeg -y -i "${INPUT}" -i "${PALETTE}" \
    -filter_complex "fps=15,paletteuse" "${OUTPUT}"

rm -f "${PALETTE}"

Note: you might want to modify the number after fps= to adjust how fast the video plays.

Keybase chat bot in 10 lines of bash

I’ve been getting very excited about keybase.io recently, not least because it offers secure conversation, and you can have bots.

I wrote a quick bot to simulate Arnold Schwarzenegger which I thought I’d share to demonstrate how easy it is. It is based on the keybase command line tool (which comes with the desktop client as standard) and jq, the brilliant command-line JSON manipulator.

For this to work, you need to have the keybase command installed and working, and you need jq.

Here’s the bot:

#!/bin/bash
CHANNEL=mychannel
keybase chat api-listen | while read L; do
{
    OUT=$(jq --raw-output 'select(.type == "chat")|select(.msg.content.text.body|startswith("!arnie "))| .msg.content.text.body | "*" + ltrimstr("!arnie ") + "*"' <<< "$L")
    if [ "${OUT}" != "" ]; then
    {
        keybase chat send "${CHANNEL}" "${OUT}"
    }; fi
}; done

and here's it working:

andy> !arnie Do eet do eet now!!!
andy> Do eet do eet now!!!

Note: here the bot is pretending to be me. To do this nicely, you will want a different account for the bot, but you get the idea.

Obviously, I am now working on a comprehensive bot framework in Rust. Watch this space.

Batch-converting audio files to be louder (on Linux)

My mp3 player is very quiet, so I wanted to make all my podcasts as loud as possible.

First I ran this to get the programs I needed:

sudo apt-get install libav-tools normalize-audio

To convert each file I made a script that makes a “loud” directory, and puts the loud version of a file inside there. It uses the normalize-audio command to do it.

Note that this script encodes your (now louder) podcasts into Ogg Vorbis format at 50kb/s, which is quite low quality.

#!/bin/bash

set -e
set -u

FILE="$1"
DIR=`dirname "$FILE"`

FILENAME=`basename "$FILE"`
WAV_FILENAME="${FILENAME}.wav"

LOUD_DIR="$DIR/loud"
WAV_FILE="$LOUD_DIR/$WAV_FILENAME"
LOUD_FILE="$LOUD_DIR/$FILENAME"

mkdir -p "$LOUD_DIR"

avconv -loglevel quiet -i "$FILE" "$WAV_FILE"
normalize-audio -q -a 1 "$WAV_FILE"
avconv -loglevel quiet -i "$WAV_FILE" -c:a libvorbis -b:a 50k "$LOUD_FILE"
rm "$WAV_FILE"

Finally I placed a Makefile in the directory containing podcasts directories, like this:

MP3S := $(wildcard *.mp3)
LOUD_MP3S := $(MP3S:%.mp3=loud/%.mp3)

OGGS := $(wildcard *.ogg)
LOUD_OGGS := $(OGGS:%.ogg=loud/%.ogg)

all: $(LOUD_OGGS) $(LOUD_MP3S)

loud/%.ogg: %.ogg
	loud "$<"

loud/%.mp3: %.mp3
	loud "$<"

Now I can make loud versions of all podcasts by just cding into the directory containing the Makefile, and typing make. By the power of make, it only converts files that have not already been converted.

Bash arrays

Bash arrays are a lot like Bash Associative Arrays, but with numbers as keys.

Here’s a quick reference.

Basics

$ declare -a MYARR  # Create an array
$ MYARR[3]=foo      # Put a value into an array
$ echo ${MYARR[3]}  # Get a value out of an array
foo
$ echo MYARR[3]     # WRONG
MYARR[0]
$ echo $MYARR[3]]   # WRONG
[3]

Creating, adding

$ declare -a MYARR    # Explicitly declare
$ MYARR[3]=foo        # Or this line implicitly makes it an array
$ MYARR[4]=bar        # Can add values one by one
$ declare -a MYARR=(a b c)   # Initialise all at once
$ echo ${MYARR[0]}
a
$ echo ${MYARR[1]}
b
$ echo ${MYARR[2]}
c
$ declare -a MYARR   # Or declare separately
$ MYARR=(a b c)      # Then initialise
$ echo ${MYARR[0]}
a
$ echo ${MYARR[1]}
b
$ echo ${MYARR[2]}
c
$ declare -a MYARR=(a b c)
$ MYARR=("${MYARR[@]}" d)  # Add an element
$ echo ${MYARR[@]}
a b c d
$ declare -a MYARR2=(e f g)
$ MYARR=("${MYARR[@]}" "${MYARR2[@]}")  # Concatenate arrays
$ echo ${MYARR[@]}
a b c d e f g

Keys/Indices

$ declare -a MYARR
$ MYARR[3]=foo
$ echo ${MYARR[0]}  # Unassigned values are empty

$ echo ${MYARR[4]}  # Unassigned values are empty

$ MYARR[seven]=bar     # A text index is treated as 0
$ echo ${MYARR[0]}
bar
$ echo ${MYARR[seven]} # A text index is treated as 0
bar
$ K=3
$ MYARR[$K]=baz      # Variables containing numbers work like numbers
$ echo ${MYARR[$K]}
baz
$ echo ${MYARR[3]}   # Obviously the value is accessible via the actual index
baz
$ K=foo
$ MYARR[$K]=bash     # Variables containing text are treated as 0
$ echo ${MYARR[0]}
bash

Length

$ declare -a MYARR=(a b c)
$ echo ${#MYARR[@]}  # Length of an array
3
$ echo $#MYARR[@]  # WRONG
0MYARR[@]
$ echo ${#MYARR}   # WRONG
1
$ MYARR[7]=x
$ echo ${#MYARR[@]}  # Only existing indices count in the length
4
$ declare -a MYARR=(a bb ccc)
$ echo ${#MYARR[0]}   # Length of an individual element
1
$ echo ${#MYARR[1]}
2
$ echo ${#MYARR[2]}
3

Looping

$ declare -a MYARR=("a 1" b c)
$ # Loop through array values
$ for V in "${MYARR[@]}"; do echo $V; done
a 1
b
c
$ for V in ${MYARR[@]}; do echo $V; done  #WRONG
a
1
b
c
$ echo "${!MYARR[@]}"  # Print all indices - quoted, but quotes removed by echo
0 1 2
$ echo "${MYARR[@]}"   # Print all values - quoted, but quotes removed by echo
a 1 b c

Clearing

$ declare -a MYARR
$ MYARR[3]=x

$ echo ${MYARR[3]}
x
$ unset MYARR
$ declare -a MYARR
$ echo ${MYARR[3]}

Deleting

$ MYARR[2]=foo
$ echo ${MYARR[2]}
foo
$ unset ${MYARR[2]} # WRONG
$ echo ${MYARR[2]}
foo
$ unset MYARR[2]    # To delete from an array, use "unset" with similar syntax to assigning
$ echo ${MYARR[2]}

$ MYARR[3]=quux
$ echo ${MYARR[3]}
quux
$ K=3
$ unset MYARR[$K]   # Can unset using a variable for the key too
$ echo ${MYARR[3]}

$ declare -a MYARR=(a b c d e f)
$ MYARR=("${MYARR[@]:0:3}" "${MYARR[@]:4}")  # Remove element 3, leaving no gap
$ echo ${MYARR[@]}

Cool stuff

$ declare -a MYARR=(a b c d e f g)
$ echo ${MYARR[@]:2:3}              # Extract a sub-array
c d e
$ declare -a MYARR=(a b c d e f g)
$ echo ${MYARR[@]/d/FOO}            # Replace elements that match
a b c FOO e f g

Scope

$ unset MYARR
$ function createmap() { MYARR[5]=bar; }  # Implicit creation puts it in the global scope
$ echo ${MYARR[5]}

$ createmap
$ echo ${MYARR[5]}
bar
$ unset MYARR
$ function createmaplocal() { declare -a MYARR; MYARR[3]=bar; }  # Explicit creation puts it in the local scope
$ echo ${MYARR[3]}

$ createmaplocal
$ echo ${MYARR[3]}

Links

Passing several values through a pipe in bash

I have been fiddling with some git-related shell scripts, and decided to try and follow the same approach as git in their structure. This means using the Unix system where each piece of functionality is a separate script (or executable) that communicates by using command-line arguments, reading from the standard input stream, and writing output to the standard output stream.

This allows each piece of functionality to be written in any programming or scripting language. In git’s case this has allowed initial versions to be written in bash or perl and later optimised versions (sometimes written in C) to be dropped in, piece by piece. It’s an incredibily flexible way of working and can also be very efficient.

Most of my prototyping has been in bash, and I’ve found sometimes I need to write out multiple values from a script and collect them as input in another script.

Writing the output is simple:

#!/bin/bash

# outputter.bash

# Imagine A, B and C have been created by some complex process:
A="foo bar"
B="  bar"
C="baz   "

# At the end of our script we simply write them out on separate lines in a known order
echo "${A}"
echo "${B}"
echo "${C}"

But reading them in somewhere else gave me some trouble until I learned this recipe:

#!/bin/bash

# inputter.bash

# Read in the values one per line:
IFS=$'\n' read A
IFS=$'\n' read B
IFS=$'\n' read C

# Now we can use them.
echo "A='${A}'"
echo "B='${B}'"
echo "C='${C}'"

And now the values transfer succesfully, preserving whitespace:

$ ./outputter.bash | ./inputter.bash 
A='foo bar'
B='  bar'
C='baz   '

The recipe uses bash’s built-in read command to populate the variables, but sets the IFS variable (Internal Field Separator) to a newline, meaning all the whitespace in the line is treated as part of the value to be read. The $'\n' syntax is a literal newline.