Importing/migrating from one peertube server to another

My Peertube server is shutting down, so I need to move my videos to another one. The official scripts don’t seem to cover this case very well, so here is what I did.

My script fetches videos and their details and uploads them to the new server via the Peertube API.

Contributions welcome: I was not able to copy video descriptions across, and I was too lazy so I hard-coded the number of tags. Also, I didn’t make a git repo for all this because I felt it needs more thought, but feel free to start one and I will happily contribute this.

This script copies all videos in a single Peertube channel to a different server. You must find the numeric ID of the channel on the new server to copy into, which I did by looking at the responses in the Network tab of Firefox’s developer tools when I clicked on its name in the web interface. It requires bash, curl, youtube-dl and jq.

Here’s peertube-migrate-channel.bash:

#!/bin/bash

set -u
set -e

# Modify these for your setup
OLD_SERVER="INSERT OLD SERVER e.g. https://peertube.social"
OLD_CHANNEL="INSERT CHANNEL URL-NAME e.g. trials_fusion"
NEW_SERVER="INSERT NEW SERVER e.g. https://video.hardlimit.com"
NEW_CHANNEL="INSERT NEW CHANNEL ID e.g. 4103"
USERNAME="INSERT_USERNAME for new server e.g. trials"
PASSWORD="INSERT PASSWORD for new server"
TAG1="INSERT_A_TAG e.g trials"
TAG2="INSERT_A_TAG e.g. gaming"
TAG3="INSERT_A_TAG e.g. gameing"

DIR=$(mktemp -d)
API_PATH="${NEW_SERVER}/api/v1"

# Find out how many videos are in the channel
curl -s \
    "${OLD_SERVER}/api/v1/video-channels/${OLD_CHANNEL}/videos/?count=1" \
    > "${DIR}/videos-total.json"

TOTAL=$(jq .total < "${DIR}/videos-total.json")
CURRENT=0
PAGE_SIZE=10

# Get a list of URLS for all the videos

echo -n "" > "${DIR}/urls.txt"

while (( CURRENT < TOTAL )); do
    FILE="${DIR}/videos-page-${CURRENT}.json"

    curl -s \
        "${OLD_SERVER}/api/v1/video-channels/${OLD_CHANNEL}/videos/?start=${CURRENT}&count=${PAGE_SIZE}&skipCount=true" \
        > "${FILE}"

    jq ".data | map(.uuid) | .[]" -r < "${FILE}" >> "${DIR}/urls.txt"

    CURRENT=$((CURRENT + PAGE_SIZE))
done

# Log in to the new server

client_id=$(curl -s "${API_PATH}/oauth-clients/local" | jq -r ".client_id")
client_secret=$(curl -s "${API_PATH}/oauth-clients/local" | jq -r ".client_secret")
token=$(curl -s "${API_PATH}/users/token" \
  --data client_id="${client_id}" \
  --data client_secret="${client_secret}" \
  --data grant_type=password \
  --data response_type=code \
  --data username="${USERNAME}" \
  --data password="${PASSWORD}" \
  | jq -r ".access_token")

# Download and upload each video

tac "${DIR}/urls.txt" \
    | while read ID; do
        URL="${OLD_SERVER}/api/v1/videos/${ID}"
        curl -s "${URL}" > "${DIR}/info-${ID}.json"
        NAME=$(jq -r .name < "${DIR}/info-${ID}.json")
        CATEGORY=$(jq -r .category.id < "${DIR}/info-${ID}.json")
        LICENCE=$(jq -r .licence.id < "${DIR}/info-${ID}.json")
        LANGUAGE=$(jq -r .language.id < "${DIR}/info-${ID}.json")
        PRIVACY=$(jq -r .privacy.id < "${DIR}/info-${ID}.json")

        OLD_VIDEO="https://peertube.social/videos/watch/${ID}"
	mkdir "${DIR}/dl-${ID}"
	youtube-dl "${OLD_VIDEO}" --output="${DIR}/dl-${ID}/dl.mp4"

        echo "Uploading ${OLD_VIDEO}"
        curl "${API_PATH}/videos/upload" \
	  --silent \
          -H "Authorization: Bearer ${token}" \
          --output "${DIR}/curl-out-${ID}.txt" \
          --max-time 6000 \
	  --form name="${NAME}" \
	  --form videofile=@"${DIR}/dl-${ID}/dl.mp4" \
          --form channelId=${NEW_CHANNEL} \
          --form category=${CATEGORY} \
          --form licence=${LICENCE} \
          --form description="TODO VIDEO DESCRIPTION" \
          --form language=${LANGUAGE} \
          --form privacy=${PRIVACY} \
          --form tags="${TAG1}" \
          --form tags="${TAG2}" \
          --form tags="${TAG3}"
    done

Uploading to PeerTube from the command line

PeerTube’s API documentation gives an example of how to upload a video, but it is missing a couple of important aspects, most notably how to provide multiple tags use form-encoded input, so my more complete script is below. Use it like this:

# First, make sure jq is installed
echo "myusername" > username
echo "mypassword" > password
./upload "video_file.mp4"

Downsides:

  1. Your username and password are visible via ps to users on the same machine (tips to avoid this are welcome)
  2. I can’t work out how to include newlines in the video description (again, tips welcome)

You will need to edit the script to provide your own PeerTube server URL, channel ID (a number), video description, tags etc. Output and errors from the script will be placed in curl-out.txt. Read the API docs to see what numbers you need to use for category, license etc.

Here is upload:

#!/bin/bash

set -e
set -u

USERNAME="$(cat username)"
PASSWORD="$(cat password)"
FILE_PATH="$1"
CHANNEL_ID=MY_CHANNEL_ID_EG_1234
NAME="${FILE_PATH%.*}"
NAME="${NAME#*/}"

API_PATH="https://MY_PEERTUBE_SERVER_URL/api/v1"
## AUTH
client_id=$(curl -s "${API_PATH}/oauth-clients/local" | jq -r ".client_id")
client_secret=$(curl -s "${API_PATH}/oauth-clients/local" | jq -r ".client_secret")
token=$(curl -s "${API_PATH}/users/token" \
  --data client_id="${client_id}" \
  --data client_secret="${client_secret}" \
  --data grant_type=password \
  --data response_type=code \
  --data username="${USERNAME}" \
  --data password="${PASSWORD}" \
  | jq -r ".access_token")

echo "Uploading ${FILE_PATH}"
curl "${API_PATH}/videos/upload" \
  -H "Authorization: Bearer ${token}" \
  --output curl-out.txt \
  --max-time 6000 \
  --form videofile=@"${FILE_PATH}" \
  --form channelId=${CHANNEL_ID} \
  --form name="$NAME" \
  --form category=15 \
  --form licence=7 \
  --form description="MY_VIDEO_DESCRIPTION" \
  --form language=en \
  --form privacy=1 \
  --form tags="TAG1" \
  --form tags="TAG2" \
  --form tags="TAG3" \
  --form tags="TAG4"

Migrating videos from YouTube to PeerTube inside a Docker container

I have quite a few videos hosted on YouTube that I would like to upload to my new PeerTube location, but I don’t want to install all the PeerTube dependencies on my machine, so I did it all inside a Docker image.

First I built and started a Docker container:

$ git clone https://github.com/chocobozzz/PeerTube /tmp/peertube
$ cd /tmp/peertube
$ docker build . -f ./support/docker/production/Dockerfile.stretch --tag peertube
$ docker run --tty --interactive peertube bash

Then I ran these commands inside it:

# yarn install --production=false
# node dist/server/tools/import-videos.js -u "https://peertube.mastodon.host" -U "andybalaam" -t "https://www.youtube.com/watch?v=TG0qRDrUPpA"

Of course, it would be better to write this up into its own Dockerfile to make this a one-liner.

References: PeerTube Docker setup, PeerTube video import.