TypeScript – Andy Balaam's Blog

Deleting an Indexed DB store can be incredibly slow on Firefox

We had performance problem in Element Web when upgrading the Indexed DB schema, and it turned out that in Firefox, deleting an object store can be incredibly slow. It can take tens of minutes or even hours.

(In Chromium the same operation can take tens of seconds, but it’s way, way faster.)

I was expecting this to be a near-instant operation, so this was a definite surprise to me.

Note that my analysis is based on widely-available browsers in February 2024, and will probably become out-of-date.

You can see the full graphs here: artificialworlds.net/indexed-db-perf/delete.html. Source code is at codeberg.org/andybalaam/indexed-db-perf.

Here’s what I learned:

Headline 1: Firefox can take a very long time to delete an object store

Graph showing that Firefox took 1.2 million milliseconds to delete an object store containing 200K records.

Firefox took 20 minutes to delete an object store containing 200K records, with one index.

In Chromium, I found that deleting a similar object store took 21 seconds, which is still slow, but rather more acceptable.

Consider not deleting stores you don’t need any more. If you must delete them, you will need to provide feedback to your users, especially if they are using Firefox.

Headline 2: Clearing the store before deleting it really helps(!)

Graph showing that Firefox takes 1.2 million ms to delete a large object store (200K records), or 300 thousand ms if you clear it before deleting it.

Graph showing that Chromium takes 20 thousand ms to delete a large object store (200K records), or 10 thousand ms if you clear it before deleting it.

On both Firefox and Chromium, running objectStore.clear() before upgrading the db and deleting the store made a significant improvement to the total time. On Chromium it more than halved the time to delete the store, and on Firefox (where the numbers were huge) it reduced the time by about 3 times!

Thanks to michael-p on Stack Overflow for giving me the idea.

Clear your store before deleting it.

Headline 3: With no indices, this is fine

Firefox deletes object stores fairly fast if there are no indices (and so does Chromium).

Graph showing that even for 200K records, Firefox can delete an object store in under 500ms if there is no index on it.

Even for 200K records, Firefox can delete an object store in under 500ms if there is no index on it.

If you need fast deletion, don’t use indices.

Observation: It’s not done until the DB is closed

In my timing experiments, I found that objectStore.delete() completed, but the operation was not really done. When I called the close() method on my IDBDatabase I had to wait a whole lot longer. (The close time is included in the measurements above.)

Even when I refreshed the browser, I found I had a long wait to open the Indexed DB database. After the wait, it worked fine and the schema update was complete.

Expect long close times after a deletion.

Keep your Indexed DB keys and values small if you want good performance!

In our work recently on Element Web (specifically attempting to replace our encryption code with our cross-platform Rust implementation) we’ve noticed some strange behaviour with the performance of our Indexed DB queries.

We’ve been aware of some slowdowns, and wondered whether it was related to the locks we take in the Rust code (which is compiled to WASM for this project), but actually we’ve tracked the latest problems down to time spent inside Indexed DB queries.

The first surprising thing we found was that several of our slow queries were just counting the number of records in a store. As we dug deeper, we began to suspect that the size of the values we were storing was affecting performance.

Click the link for interactive graphs

I designed a simple test page and we measured some results. You can look at interactive graphs and and record on your own device here: artificialworlds.net/indexed-db-perf. I am hoping to expand this page to more devices and more operations over time, so do check back and/or contribute results. Source code is at codeberg.org/andybalaam/indexed-db-perf.

Note that my analysis is based on widely-available browsers in January 2024, and may become out-of-date.

Here are my conclusions so far.

Headline 1: counting the records in your store can be slow

If we have large keys or values, and we try find out how many records there are in a store containing 200K of them, it will be very slow (6 seconds on my Chromium).

Try to avoid counting records when you can, and consider caching the count in your own code if you need it often.

The slowness of counting probably also indicates that operations that walk all records will also be slow, so think carefully about when and whether you need to do that, and expect to provide feedback to your users when you do it.

Don’t count or walk all records unless you have to.

Headline 2: long keys hurt performance

On Firefox, as keys get longer, performance with large numbers of records becomes slower.

The time it takes to count records grows linearly with the length of the keys, and becomes large when keys go over 50 bytes (500 bytes in Firefox).

The shorter your keys, the better.

Headline 3: large values hurt performance

Graph showing it can take 6 seconds to count records with large values (50K) and lots of records (200K(

On Chrome, as values get larger, performance with large numbers of records becomes slower.

The time it takes to count records grows linearly with the size of the values you are storing, and becomes large when values go over 1K in size (10K in Firefox).

The smaller your values, the better.

Headline 4: the number of records matters

Graphs showing the performance on Firefox degrades rapidly after around 50K records.

On Firefox over 50K records (with large keys or values), performance degrades rapidly.

If you are storing large values (> 1K) or large keys (> 50 bytes), and you need to walk all records regularly, you should probably try to keep the number of records below 50K if you want interactive performance – over that it rapidly increases to the order of seconds.

In Firefox on Intel, it does seem reasonably stable after 50K, though, so if you exclusively target Firefox on Intel, larger stores are feasible, but you will need to manage the user experience around these ~1 second delays.

In Chromium, performance continues to degrade as the number of records increases. It reaches truly terrible times (6 seconds) with large keys (2000 bytes) and large numbers (200K).

Keep below 50K records if you can.

Headline 5: Indexed DB performance is currently very poor on Apple M1

Graph showing counting records can take 12 seconds on an Apple M1.

Chrome on Apple M1 hardware takes 12 seconds to perform operations that take 2.5 seconds on Intel.

Firefox’s indexed DB performance on Apple M1 is ~10 times slower than on Intel, taking 8 seconds to count 200K large records.

Chrome’s performance with large keys (> 1000 bytes) is fairly catastrophic when the number of rows approaches 50K. It takes 12 seconds to count 200K records with keys of length 2000, ~6x slower than on Intel.

Both browsers on M1 hit a wall at about 50K records, where their performance with either large keys or large values rapidly degrades to almost unusable levels. The only option for decent performance is to keep keys short and values small.

Test on Apple M1 – it has a serious problem right now!

Observation: Firefox faster than Chromium

Graph showing that for counting large numbers of large records, Firefox takes 0.8 seconds and Chromium takes 6 seconds.

Chrome takes 6 seconds to count large numbers of large values, and Firefox takes 0.8 seconds to do the same thing.

At time of writing, Firefox’s Indexed DB appears to be much faster than Chromium’s when dealing with large numbers of records. In some scenarios, over 7 times faster.

Side note: string values are better than arrays of numbers

Update: see Don’t store arrays of numbers in Indexed DB – use base64 instead.

I don’t have the formal evidence for this because I ran out of time for my investigation, but my initial exploration suggests that encoding a string as an array of numbers, one for each character, slows down performance by over 10 times.

(I think) you should use strings instead of arrays of numbers.

Conclusions

Watch out for Apple M1!

Meanwhile, if you need to improve the performance of your Indexed DB-based web site, you have three options:

Store fewer records,
Reduce the size of the records, or
Reduce the length of the keys.

1 and 2 are probably harder than 3. So definitely consider 3 first.

If you keep key length and value length small, you can have lots of records.

Explore the graphs here: artificialworlds.net/indexed-db-perf

Combining two function types with & (ampersand) in TypeScript (intersection)

Combining interfaces/objects with &

When you combine two types in TypeScript with & (ampersand), it is called an Intersection Type.

For example:

interface Particle {
    mass: number;
}

interface Wave {
    wavelength: number;
}

type Both = Particle & Wave;

The new type Both is both a particle and a wave, so it has both properties mass and wavelength.

If you combined the types with |, making a Union Type, like this:

type Either = Particle | Wave;

then the new type Either would be either a particle or a wave, not both. To use it, you would need to find out which it was, before accessing only the property (mass or wavelength) that you now know exists, and not the other one.

Combining functions with &

This is all fine, and relatively easy to understand, but what about when you combine function types in this way?

type TakesString = (s: string) => void;
type TakesNumber = (n: number) => void;
type TakesStringOrNumber = TakesString & TakesNumber;

Now you have a type that is both TakesString AND TakesNumber, which means a function that can do both things, which means it takes a string or a number as an argument. I can create something that has this type like this:

const f: TakesStringOrNumber = (x: string | number) => {console.log(x);};

Notice how that | symbol sneaked in there? In order to satisfy the & on the function types, the argument types use |. When two things have this kind of opposite relationship it’s sometimes known as contravariance.

It has to be this way: for the function to be TakesString AND TakesNumber the argument needs to be string OR number.

Functions with literal types

If your functions take Literal Types this can get even more confusing:

type TakesTrue = (success: true) => void;
type TakesFalse = (success: false) => void;

Any argument you pass to a function that is TakesTrue must be true. Similarly, to call a function that is TakesFalse you must pass in false. TypeScript lets you do this, which is fun.

So now imagine you combine these types:

type TakesEither = TakesTrue & TakesFalse;

Now, TakesEither is both TakesTrue and TakesFalse so it can take in either true or false.

Let’s make a function that can do that:

const f: TakesEither = (success: boolean) => { console.log(`${success}`); };

This works – f takes a boolean, so it does indeed allow you to pass in true or false, as required to be a TakesEither.

What’s weird though, is that you can’t do this:

let x: boolean = new Date().getHours > 12;
f(x); // Compile error

(Ignore the date stuff – x is just a boolean that might be true or false.)

Here is the compile error:

index.ts:9:7 - error TS2769: No overload matches this call.
  Overload 1 of 2, '(success: true): void', gave the following error.
    Argument of type 'boolean' is not assignable to parameter of type 'true'.
  Overload 2 of 2, '(success: false): void', gave the following error.
    Argument of type 'boolean' is not assignable to parameter of type 'false'.

The thing is that the compiler only knows that f is a TakesEither, which means that it is either a function that takes true or a function that takes false. It doesn’t know that f can take a boolean (even though here it can!).

This code does work:

let x: boolean = new Date().getHours() > 12;
if (x) {
    f(x); // OK
} else {
    f(x); // OK
}

Why? Because inside the first part of the if, the compiler knows that x is true, so it can call f with it, because f can take true as an argument. Similarly in the second half, it knows x is false.

In future, the compiler might grow the ability to handle work in the first case, but at time of writing, we see this error.

Cool, huh?

(Thanks to jcalz for this stackoverflow answer that helped me understand this a little: typescript intersection types of function.)