22-Dec

TypeScript Decoders and Where to Find Them

TypeScript is fantastic, but without Json Decoders, your code (probably) isn't type safe.

10 min read

By Tarjei Skjærset

December 22, 2021

Let me introduce you to a library I made to solve a pressing problem in the TypeScript space, and convince you it is something you need in your app.

It is called typescript-json-decoder and can be found over on github here: https://github.com/tskj/typescript-json-decoder. It's MIT licenced, tiny, dependency free, and works in any TypeScript environment where you might read external data, such as a React app, or any frontend app written i TypeScript really, or a backend app on Node or Deno.

What even is TypeScript?

To set the stage, let me first explain briefly what TypeScript is and what problems it tries to solve.

TypeScript is fantastic, working in a fully typed app is a great experience. It is in fact significantly better than traditional languages, and makes Java, C#, and friends, seem like dynamically typed languages in comparison.

Let me give a quick example of the kind of things that TypeScript does. The following code is just a simple example of some code you might see in your application, some functions passing a User object around.

const upsertUser = (x: User | null) => {
  if (x !== null) {
    doToUser(x);
  }
  // ...
};

const doToUser = (x: User) => {
  console.log(x.username);
};

Here TypeScript helps you manage the types of your values, letting you access the username property on the User object - but only if it isn't null. Compared to regular, untyped JavaScript this is a lot of help in reminding you what your functions can and cannot do. But even compared to C# or Java, TypeScript "remembers" that you have checked for null in the first function, and allows you to skip the null check in the second. This is the philosophy of having a fully typed app, doing the right thing with as little friction as possible, and only allowing legal operations to compile. This way you have no runtime errors.

Where TypeScript Falters

Or is it so simple? Let us have a look at some more code. For example, a very standard way you might load the data about the Users is the following.

const getUsers = () =>
  fetch('/users').then(x => x.json()) as Promise<LoggedInUsers>;

Pretty standard way of fetching data from an API in a frontend app. In addition to this code, you might have a separate file with some type definitions that mirror the API. That is, you have manually typed the JSON API with some matching TypeScript types, and that might look a bit like the following.

type User = {
  username: string;
  dateOfBirth: Date;
  password: string;
}

type LoggedInUsers = User[]

Just to flesh out the picture of what we're talking about, the following is what a corresponding HTTP request / response might look like.

GET /users

[
  {
    "username": "AzureDiamond",
    "dateOfBirth": "2004-06-03",
    "password": "hunter2"
  },
  ...
]

Notice how the response JSON corresponds to our type, it is a list of objects which have a "username" key with a string value, a "dateOfBirth" key with a Date value, and a "password" key with a string password in cleartext as a value. And here lies our problem. Our astute readers will have noticed that the "dateOfBirth" key does not in fact have a Date object as its value in the JSON representation, but rather a regular old string, which happens to encode a date. This is because JSON is a serialization format and does not have rich datatypes or objects such as Date, but only simple primitive types such as string, number, boolean and their aggregates, in the form of records and arrays.

This mistake will not be caught at compile time, in fact this program compiles without warnings. Yet we will see a runtime crash, which is exactly what TypeScript is designed to avoid. And even worse, the crash happens far away from all the code we have seen so far! It happens much "later" in the program execution than you might think. For example, consider a piece of code that uses the result of fetching this data, which might be something like the following.

const users = await getUsers();
return users.map(user => user.dateOfBirth.getFullYear());

This code tries to get the year part of the date, by calling the method .getFullYear(), which is a standard method on the Date class. However, strings don't have any such method of course, and here the program will crash with a classic getFullYear is not a function.

The Hotfix

When this happens in production you will go back to your type definitions and fix it to actually match the data. We swap the Date for the string in the type definition:

type User = {
  username: string;
  dateOfBirth: string;
  password: string;
}

type LoggedInUsers = User[]

And now finally the compiler yells at us, and forces us to fix our program where it's wrong. The compiler error will be at the use site, so let's go there as well, and parse the date out of the string where we need it.

const users = await getUsers();
return users.map(user => new Date(user.dateOfBirth).getFullYear());

That's a slight complication, but now it's correct and works, also at runtime.

Let's Fix it with a Decoder

This isn't a full fix though, because what stops it from happening again? The API might change, we might change our type definitions to use more of the API, or our current representation of the API might be wrong right now - we just haven't hit the runtime case where it explodes.

I hope I have made a compelling case that this isn't a good way of doing this. It's full of bugs and you can never be confident that it's actually correct. So let me introduce you to a better world, a world of decoders.

What is a decoder? A decoder is some mechanism that validates that our JSON structure actually matches our idea of it, that is - our types line up with the data we receive. In our case we'll write a function that traverses the structure to check it. This function also needs some kind of mechanism to report errors, which could be returning a "Result" type or throw an exception.

Let's implement a hypothetical User decoder.

const userDecoder = (json: unknown): User => {
  if (typeof json !== 'object' || json === null) {
    throw 'Not an object';
  }
  if (!( 'username' in json )) {
    throw 'Missing key "username"';
  }
  if (typeof json['username'] !== 'string') {
    throw '"username" must be a string';
  }
  // ...
};

Here we simply write some straightforward code to check that the thing is an object, and if it is, we verify that it has the right keys, and that the keys have the expected types. Here I've only showed the first key, "username", but you'd have to repeat this process for all keys in the object. This is incredibly tedious. But it works! Let's revisit our example to see how it makes things better.

const response = await fetch('/users');
const rawData = await response.json();
const users: LoggedInUsers = rawData.map(userDecoder);

We simply map the userDecoder over every element in our list, and if any one of them fails, the decoder throws an exception, which causes the promise we're in to be rejected - just as if the network call itself had failed. This makes a lot of sense, because being unable to parse the data in a way that our app understands is just as bad as the endpoint or API being down.

Notice that even this code assumes that the returned data is an array (which is why we can call .map on it), however to be properly safe we would need to have an array decoder as well. And we should, but a much more pressing problem is that our decoder code is incredibly annoying to write and maintain. Which in fact is why no one does this, everyone just does the first thing and praying, because keeping up the decoder code with your types is very time consuming, it's just downright ugly, and keeping them in sync really is a chore. But it doesn't have to be that way.

Finally, automatic Json Decoders with my library

This problem annoyed me so much, especially the fact that the lack of proper tools being so bad that people don't even bother to do it safely, that I decided to make this library. The idea is as simple as it is powerful: automatic decoders would allow you to write one single definiton of the API types and have a decoder derived automatically. That would be huge, it would be no more difficult than the status quo of writing the API types out in a dedicated file and the decoders would give you actual type safety. Here is how it works:

import { decodeType, string, date, array } from 'typescript-json-decoder';

type User = decodeType<typeof userDecoder>;
const userDecoder = record({
  username: string,
  dateOfBirth: date,
  password: string,
});

type LoggedInUsers = decodeType<typeof usersDecoder>;
const usersDecoder = array(userDecoder);

One of the primary design goals of this library is to be as idiomatic TypeScript as possible, and as low friction, low overhead as it can. It leverages your existing knowledge, is easy to migrate to from classic type definitions (because it looks and behaves like them), and gets out of your way.

This code is supposed to look as similar (and mean the same thing!) as possible to the original type definition, all we do is swap the types with imported string and date decoders, and wrap it up in record. This defines a decoder of this type as you would expect it to work, and then we add a single line on the top, which derives the type from the decoder (that's the line type User = decodeType<typeof userDecoder>).

I know that was a mouthful, but the process of migrating your existing types is simple. The decoders mirror your existing types one to one, and all you have to do in practice is to add the single line above every decoder to have the type derived automatically (you don't actually need this either, it's just there for convenience if you want it).

Notice also that here we have an actual array(userDecoder), which I commented on in the previous (manual) example. This is even more correct, because it not only checks every object in the array, but also makes sure the JSON is in fact an array.

Notice also also that here we actually can use date! This is actually an additional benefit of using a decoder library, you are not necessarily stuck only typing your API using primitve JSON types, but you can actually "pretend" that JSON supports these kinds of rich data types. Here, the library provides a date decoder which decodes a string containing date information, encoded in a way the JavaScript Date class understands. And the result of this is an actual Date object! Which means you can safely (with complete type safety and without runtime errors) call .getFullYear() on it. And this isn't just limited to the decoders this library provides, you can compose and write your own decoders which decode any arbitrary data structure you wish.

Wrapping up, you can use this decoder anywhere you want, but I recommend putting it as close to the "edge" of your program as possible, such as where you fetch the data.

const getUsers = () =>
  fetch('/users').then(x => x.json()).then(usersDecoder);

Here you don't need to cast to a Promise<LoggedInUsers>, in fact you don't need to specify any types by hand at all, it is all inferred automatically for you.

And now the use site also works, with complete type safety, as promised.

const users = await getUsers();
return users.map(user => user.dateOfBirth.getFullYear());

Which is exactly the code we started with!

I hope I have made a good case for why you need decoders in your app, and why this library is the best solution. You can find it on github and npm.

I look forward to hearing your feedback on how it works for you in your projects! I'm sure there are things that can be better and which need improving, and I would like to know about them. If you need any help in using the library, have any questoins or have feedback, please don't hesitate to get in touch. I'm @_tskj_ on Twitter and epost@tarjei.org on the emails. Merry Christmas!