Monday 26 April 2021

The Service Now API and TypeScript Conditional Types

The Service Now REST API is an API which allows you to interact with Service Now. It produces different shaped results based upon the sysparm_display_value query parameter. This post looks at how we can model these API results with TypeScripts conditional types. The aim being to minimise repetition whilst remaining strongly typed. This post is specifically about the Service Now API, but the principles around conditional type usage are generally applicable.

Service Now and TypeScript

The power of a query parameter

There is a query parameter which many endpoints in Service Nows Table API support named sysparm_display_value. The docs describe it thus:

Data retrieval operation for reference and choice fields. Based on this value, retrieves the display value and/or the actual value from the database.

Valid values:

  • true: Returns the display values for all fields.
  • false: Returns the actual values from the database.
  • all: Returns both actual and display value

Let's see what that looks like when it comes to loading a Change Request. Consider the following curls:

# sysparm_display_value=all
curl "https://ourcompanyinstance.service-now.com/api/now/table/change_request?sysparm_query=number=CHG0122585&sysparm_limit=1&sysparm_display_value=all" --request GET --header "Accept:application/json" --user 'API_USERNAME':'API_PASSWORD' | jq '.result[0] | { state, sys_id, number, requested_by, reason }'

# sysparm_display_value=true
curl "https://ourcompanyinstance.service-now.com/api/now/table/change_request?sysparm_query=number=CHG0122585&sysparm_limit=1&sysparm_display_value=true" --request GET --header "Accept:application/json" --user 'API_USERNAME':'API_PASSWORD' | jq '.result[0] | { state, sys_id, number, requested_by, reason }'

# sysparm_display_value=false
curl "https://ourcompanyinstance.service-now.com/api/now/table/change_request?sysparm_query=number=CHG0122585&sysparm_limit=1&sysparm_display_value=false" --request GET --header "Accept:application/json" --user 'API_USERNAME':'API_PASSWORD' | jq '.result[0] | { state, sys_id, number, requested_by, reason }'

When executed, they each load the same Change Request from Service Now with a different value for sysparm_display_value. You'll notice there's some jq in the mix as well. This is because there's a lot of data in a Change Request. Rather than display everything, we're displaying a subset of fields. The first curl has a sysparm_display_value value of all, the second false and the third true. What do the results look like?

sysparm_display_value=all:

{
  "state": {
    "display_value": "Closed",
    "value": "3"
  },
  "sys_id": {
    "display_value": "4d54d7481b37e010d315cbb5464bcb95",
    "value": "4d54d7481b37e010d315cbb5464bcb95"
  },
  "number": {
    "display_value": "CHG0122595",
    "value": "CHG0122595"
  },
  "requested_by": {
    "display_value": "Sally Omer",
    "link": "https://ourcompanyinstance.service-now.com/api/now/table/sys_user/b15cf3ebdbe11300f196f3651d961999",
    "value": "b15cf3ebdbe11300f196f3651d961999"
  },
  "reason": {
    "display_value": null,
    "value": ""
  }
}

sysparm_display_value=true:

{
  "state": "Closed",
  "sys_id": "4d54d7481b37e010d315cbb5464bcb95",
  "number": "CHG0122595",
  "requested_by": {
    "display_value": "Sally Omer",
    "link": "https://ourcompanyinstance.service-now.com/api/now/table/sys_user/b15cf3ebdbe11300f196f3651d961999"
  },
  "reason": null
}

sysparm_display_value=false:

{
  "state": "3",
  "sys_id": "4d54d7481b37e010d315cbb5464bcb95",
  "number": "CHG0122595",
  "requested_by": {
    "link": "https://ourcompanyinstance.service-now.com/api/now/table/sys_user/b15cf3ebdbe11300f196f3651d961999",
    "value": "b15cf3ebdbe11300f196f3651d961999"
  },
  "reason": ""
}

As you can see, we have the same properties being returned each time, but with a different shape. Let's call out some interesting highlights:

  • requested_by is always an object which contains link. It may also contain value and display_value depending upon sysparm_display_value
  • state, sys_id, number and reason are objects containing value and display_value when sysparm_display_value is all. Otherwise, the value of value or display_value is surfaced up directly; not in an object.
  • most values are strings, even if they represent another data type. So state.value is always a stringified number. The only exception to this rule is reason.display_value which can be null

Type Definition time

We want to create type definitions for these API results. We could of course create three different results, but that would involve duplication. Boo! It's worth bearing in mind we're looking at a subset of five properties in this example. In reality, there are many, many properties on a Change Request. Whilst this example is for a subset, if we wanted to go on to create the full type definition the duplication would become very impractical.

What can we do? Well, if all of the underlying properties were of the same type, we could use a generic and be done. But given the underlying types can vary, that's not going to work. We can achieve this though through using a combination of generics and conditional types.

Let's begin by creating a string literal type of the possible values of sysparm_display_value:

export type DisplayValue = 'all' | 'true' | 'false';

Making a PropertyValue type

Next we need to create a type that models the object with display_value and value properties.

:::info a type for state, sys_id, number and reason

  • state, sys_id, number and reason are objects containing value and display_value when sysparm_display_value is 'all'. Otherwise, the value of value or display is surfaced up directly; not in an object.
  • most values are strings, even if they represent another data type. So state.value is always a stringified number. The only exception to this rule is reason.display_value which can be null

:::

export interface ValueAndDisplayValue<TValue = string, TDisplayValue = string> {
    display_value: TDisplayValue;
    value: TValue;
}

Note that this is a generic property with a default type of string for both display_value and value. Most of the time, string is the type in question so it's great that TypeScript allows us to cut down on the amount of syntax we use.

Now we're going to create our first conditional type:

export type PropertyValue<
    TAllTrueFalse extends DisplayValue,
    TValue = string,
    TDisplayValue = string
> = TAllTrueFalse extends 'all'
    ? ValueAndDisplayValue<TValue, TDisplayValue>
    : TAllTrueFalse extends 'true'
    ? TDisplayValue
    : TValue;

The PropertyValue will either be a ValueAndDisplayValue, a TDisplayValue or a TValue, depending upon whether PropertyValue is 'all', 'true' or 'false' respectively. That's hard to grok. Let's look at an example of each of those cases using the reason property, which allows a TValue of string and a TDisplayValue of string | null:

const reasonAll: PropertyValue<'all', string, string | null> = {
    "display_value": null,
    "value": ""
};
const reasonTrue: PropertyValue<'true', string, string | null> = null;
const reasonFalse: PropertyValue<'false', string, string | null> = '';

Consider the type on the left and the value on the right. We're successfully modelling our PropertyValues. I've deliberately picked an edge case example to push our conditional type to its limits.

Service Now Change Request States

Let's look at another usage. We'll create a type that repesents the possible values of a Change Request's state in Service Now. Do take a moment to appreciate these values. Many engineers were lost in the numerous missions to obtain these rare and secret enums. Alas, the Service Now API docs have some significant gaps.

/** represents the possible Change Request "State" values in Service Now */
export const STATE = {
    NEW: '-5',
    ASSESS: '-4',
    SENT_FOR_APPROVAL: '-3',
    SCHEDULED: '-2',
    APPROVED: '-1',
    WAITING: '1',
    IN_PROGRESS: '2',
    COMPLETE: '3',
    ERROR: '4',
    CLOSED: '7',
} as const;

export type State = typeof STATE[keyof typeof STATE];

By combining State and PropertyValue, we can strongly type the state property of Change Requests. Consider:

const stateAll: PropertyValue<'all', State> = {
    "display_value": "Closed",
    "value": "3"
};
const stateTrue: PropertyValue<'true', State> = "Closed";
const stateFalse: PropertyValue<'false', State> = "3";

With that in place, let's turn our attention to our other natural type that the requested_by property demonstrates.

Making a LinkValue type

:::info a type for requested_by

requested_by is always an object which contains link. It may also contain value and display_value depending upon sysparm_display_value

:::

interface Link {
    link: string;
}

/** when TAllTrueFalse is 'false' */
export interface LinkAndValue extends Link {
    value: string;
}

/** when TAllTrueFalse is 'true' */
export interface LinkAndDisplayValue extends Link {
    display_value: string;
}

/** when TAllTrueFalse is 'all' */
export interface LinkValueAndDisplayValue extends LinkAndValue, LinkAndDisplayValue {}

The three types above model the different scenarios. Now we need a conditional type to make use of them:

export type LinkValue<TAllTrueFalse extends DisplayValue> = TAllTrueFalse extends 'all'
    ? LinkValueAndDisplayValue
    : TAllTrueFalse extends 'true'
    ? LinkAndDisplayValue
    : LinkAndValue;

This is hopefully simpler to read than the PropertyValue type, and if you look at the examples below you can see what usage looks like:

const requested_byAll: LinkValue<'all'> = {
    "display_value": "Sally Omer",
    "link": "https://ourcompanyinstance.service-now.com/api/now/table/sys_user/b15cf3ebdbe11300f196f3651d961999",
    "value": "b15cf3ebdbe11300f196f3651d961999"
};
const requested_byTrue: LinkValue<'true'> = {
    "display_value": "Sally Omer",
    "link": "https://ourcompanyinstance.service-now.com/api/now/table/sys_user/b15cf3ebdbe11300f196f3651d961999"
}
const requested_byFalse: LinkValue<'false'> = {
    "link": "https://ourcompanyinstance.service-now.com/api/now/table/sys_user/b15cf3ebdbe11300f196f3651d961999",
    "value": "b15cf3ebdbe11300f196f3651d961999"
};

Making our complete type

With these primitives in place, we can now build ourself a (cut-down) type that models a Change Request:

export interface ServiceNowChangeRequest<TAllTrueFalse extends DisplayValue> {
    state: PropertyValue<TAllTrueFalse, State>;
    sys_id: PropertyValue<TAllTrueFalse>;
    number: PropertyValue<TAllTrueFalse>;
    requested_by: LinkValue<TAllTrueFalse>;
    reason: PropertyValue<TAllTrueFalse, string, string | null>;
    // there are *way* more properties in reality
}

This is a generic type which will accept 'all', 'true' or 'false' and will use that type to drive the type of the properties inside the object. And now we have successfully typed our Service Now Change Request, thanks to TypeScript's conditional types.

To test it out, let's take the JSON responses we got back from our curls at the start, and see if we can make ServiceNowChangeRequests with them.

const changeRequestFalse: ServiceNowChangeRequest<'false'> = {
  "state": "3",
  "sys_id": "4d54d7481b37e010d315cbb5464bcb95",
  "number": "CHG0122595",
  "requested_by": {
    "link": "https://ourcompanyinstance.service-now.com/api/now/table/sys_user/b15cf3ebdbe11300f196f3651d961999",
    "value": "b15cf3ebdbe11300f196f3651d961999"
  },
  "reason": ""
};

const changeRequestTrue: ServiceNowChangeRequest<'true'> = {
  "state": "Closed",
  "sys_id": "4d54d7481b37e010d315cbb5464bcb95",
  "number": "CHG0122595",
  "requested_by": {
    "display_value": "Sally Omer",
    "link": "https://ourcompanyinstance.service-now.com/api/now/table/sys_user/b15cf3ebdbe11300f196f3651d961999"
  },
  "reason": null
}

const changeRequestAll: ServiceNowChangeRequest<'all'> = {
  "state": {
    "display_value": "Closed",
    "value": "3"
  },
  "sys_id": {
    "display_value": "4d54d7481b37e010d315cbb5464bcb95",
    "value": "4d54d7481b37e010d315cbb5464bcb95"
  },
  "number": {
    "display_value": "CHG0122595",
    "value": "CHG0122595"
  },
  "requested_by": {
    "display_value": "Sally Omer",
    "link": "https://ourcompanyinstance.service-now.com/api/now/table/sys_user/b15cf3ebdbe11300f196f3651d961999",
    "value": "b15cf3ebdbe11300f196f3651d961999"
  },
  "reason": {
    "display_value": null,
    "value": ""
  }
}

We can! Do take a look at this in the TypeScript playground.

The Service Now API and TypeScript Conditional Types

The Service Now REST API is an API which allows you to interact with Service Now. It produces different shaped results based upon the sysparm_display_value query parameter. This post looks at how we can model these API results with TypeScripts conditional types. The aim being to minimise repetition whilst remaining strongly typed. This post is specifically about the Service Now API, but the principles around conditional type usage are generally applicable.

Service Now and TypeScript

The power of a query parameter

There is a query parameter which many endpoints in Service Nows Table API support named sysparm_display_value. The docs describe it thus:

Data retrieval operation for reference and choice fields. Based on this value, retrieves the display value and/or the actual value from the database.

Valid values:

  • true: Returns the display values for all fields.
  • false: Returns the actual values from the database.
  • all: Returns both actual and display value

Let's see what that looks like when it comes to loading a Change Request. Consider the following curls:

# sysparm_display_value=all
curl "https://ourcompanyinstance.service-now.com/api/now/table/change_request?sysparm_query=number=CHG0122585&sysparm_limit=1&sysparm_display_value=all" --request GET --header "Accept:application/json" --user 'API_USERNAME':'API_PASSWORD' | jq '.result[0] | { state, sys_id, number, requested_by, reason }'

# sysparm_display_value=true
curl "https://ourcompanyinstance.service-now.com/api/now/table/change_request?sysparm_query=number=CHG0122585&sysparm_limit=1&sysparm_display_value=true" --request GET --header "Accept:application/json" --user 'API_USERNAME':'API_PASSWORD' | jq '.result[0] | { state, sys_id, number, requested_by, reason }'

# sysparm_display_value=false
curl "https://ourcompanyinstance.service-now.com/api/now/table/change_request?sysparm_query=number=CHG0122585&sysparm_limit=1&sysparm_display_value=false" --request GET --header "Accept:application/json" --user 'API_USERNAME':'API_PASSWORD' | jq '.result[0] | { state, sys_id, number, requested_by, reason }'

When executed, they each load the same Change Request from Service Now with a different value for sysparm_display_value. You'll notice there's some jq in the mix as well. This is because there's a lot of data in a Change Request. Rather than display everything, we're displaying a subset of fields. The first curl has a sysparm_display_value value of all, the second false and the third true. What do the results look like?

sysparm_display_value=all:

{
  "state": {
    "display_value": "Closed",
    "value": "3"
  },
  "sys_id": {
    "display_value": "4d54d7481b37e010d315cbb5464bcb95",
    "value": "4d54d7481b37e010d315cbb5464bcb95"
  },
  "number": {
    "display_value": "CHG0122595",
    "value": "CHG0122595"
  },
  "requested_by": {
    "display_value": "Sally Omer",
    "link": "https://ourcompanyinstance.service-now.com/api/now/table/sys_user/b15cf3ebdbe11300f196f3651d961999",
    "value": "b15cf3ebdbe11300f196f3651d961999"
  },
  "reason": {
    "display_value": null,
    "value": ""
  }
}

sysparm_display_value=true:

{
  "state": "Closed",
  "sys_id": "4d54d7481b37e010d315cbb5464bcb95",
  "number": "CHG0122595",
  "requested_by": {
    "display_value": "Sally Omer",
    "link": "https://ourcompanyinstance.service-now.com/api/now/table/sys_user/b15cf3ebdbe11300f196f3651d961999"
  },
  "reason": null
}

sysparm_display_value=false:

{
  "state": "3",
  "sys_id": "4d54d7481b37e010d315cbb5464bcb95",
  "number": "CHG0122595",
  "requested_by": {
    "link": "https://ourcompanyinstance.service-now.com/api/now/table/sys_user/b15cf3ebdbe11300f196f3651d961999",
    "value": "b15cf3ebdbe11300f196f3651d961999"
  },
  "reason": ""
}

As you can see, we have the same properties being returned each time, but with a different shape. Let's call out some interesting highlights:

  • requested_by is always an object which contains link. It may also contain value and display_value depending upon sysparm_display_value
  • state, sys_id, number and reason are objects containing value and display_value when sysparm_display_value is all. Otherwise, the value of value or display_value is surfaced up directly; not in an object.
  • most values are strings, even if they represent another data type. So state.value is always a stringified number. The only exception to this rule is reason.display_value which can be null

Type Definition time

We want to create type definitions for these API results. We could of course create three different results, but that would involve duplication. Boo! It's worth bearing in mind we're looking at a subset of five properties in this example. In reality, there are many, many properties on a Change Request. Whilst this example is for a subset, if we wanted to go on to create the full type definition the duplication would become very impractical.

What can we do? Well, if all of the underlying properties were of the same type, we could use a generic and be done. But given the underlying types can vary, that's not going to work. We can achieve this though through using a combination of generics and conditional types.

Let's begin by creating a string literal type of the possible values of sysparm_display_value:

export type DisplayValue = 'all' | 'true' | 'false';

Making a PropertyValue type

Next we need to create a type that models the object with display_value and value properties.

:::info a type for state, sys_id, number and reason

  • state, sys_id, number and reason are objects containing value and display_value when sysparm_display_value is 'all'. Otherwise, the value of value or display is surfaced up directly; not in an object.
  • most values are strings, even if they represent another data type. So state.value is always a stringified number. The only exception to this rule is reason.display_value which can be null

:::

export interface ValueAndDisplayValue<TValue = string, TDisplayValue = string> {
    display_value: TDisplayValue;
    value: TValue;
}

Note that this is a generic property with a default type of string for both display_value and value. Most of the time, string is the type in question so it's great that TypeScript allows us to cut down on the amount of syntax we use.

Now we're going to create our first conditional type:

export type PropertyValue<
    TAllTrueFalse extends DisplayValue,
    TValue = string,
    TDisplayValue = string
> = TAllTrueFalse extends 'all'
    ? ValueAndDisplayValue<TValue, TDisplayValue>
    : TAllTrueFalse extends 'true'
    ? TDisplayValue
    : TValue;

The PropertyValue will either be a ValueAndDisplayValue, a TDisplayValue or a TValue, depending upon whether PropertyValue is 'all', 'true' or 'false' respectively. That's hard to grok. Let's look at an example of each of those cases using the reason property, which allows a TValue of string and a TDisplayValue of string | null:

const reasonAll: PropertyValue<'all', string, string | null> = {
    "display_value": null,
    "value": ""
};
const reasonTrue: PropertyValue<'true', string, string | null> = null;
const reasonFalse: PropertyValue<'false', string, string | null> = '';

Consider the type on the left and the value on the right. We're successfully modelling our PropertyValues. I've deliberately picked an edge case example to push our conditional type to its limits.

Service Now Change Request States

Let's look at another usage. We'll create a type that repesents the possible values of a Change Request's state in Service Now. Do take a moment to appreciate these values. Many engineers were lost in the numerous missions to obtain these rare and secret enums. Alas, the Service Now API docs have some significant gaps.

/** represents the possible Change Request "State" values in Service Now */
export const STATE = {
    NEW: '-5',
    ASSESS: '-4',
    SENT_FOR_APPROVAL: '-3',
    SCHEDULED: '-2',
    APPROVED: '-1',
    WAITING: '1',
    IN_PROGRESS: '2',
    COMPLETE: '3',
    ERROR: '4',
    CLOSED: '7',
} as const;

export type State = typeof STATE[keyof typeof STATE];

By combining State and PropertyValue, we can strongly type the state property of Change Requests. Consider:

const stateAll: PropertyValue<'all', State> = {
    "display_value": "Closed",
    "value": "3"
};
const stateTrue: PropertyValue<'true', State> = "Closed";
const stateFalse: PropertyValue<'false', State> = "3";

With that in place, let's turn our attention to our other natural type that the requested_by property demonstrates.

Making a LinkValue type

:::info a type for requested_by

requested_by is always an object which contains link. It may also contain value and display_value depending upon sysparm_display_value

:::

interface Link {
    link: string;
}

/** when TAllTrueFalse is 'false' */
export interface LinkAndValue extends Link {
    value: string;
}

/** when TAllTrueFalse is 'true' */
export interface LinkAndDisplayValue extends Link {
    display_value: string;
}

/** when TAllTrueFalse is 'all' */
export interface LinkValueAndDisplayValue extends LinkAndValue, LinkAndDisplayValue {}

The three types above model the different scenarios. Now we need a conditional type to make use of them:

export type LinkValue<TAllTrueFalse extends DisplayValue> = TAllTrueFalse extends 'all'
    ? LinkValueAndDisplayValue
    : TAllTrueFalse extends 'true'
    ? LinkAndDisplayValue
    : LinkAndValue;

This is hopefully simpler to read than the PropertyValue type, and if you look at the examples below you can see what usage looks like:

const requested_byAll: LinkValue<'all'> = {
    "display_value": "Sally Omer",
    "link": "https://ourcompanyinstance.service-now.com/api/now/table/sys_user/b15cf3ebdbe11300f196f3651d961999",
    "value": "b15cf3ebdbe11300f196f3651d961999"
};
const requested_byTrue: LinkValue<'true'> = {
    "display_value": "Sally Omer",
    "link": "https://ourcompanyinstance.service-now.com/api/now/table/sys_user/b15cf3ebdbe11300f196f3651d961999"
}
const requested_byFalse: LinkValue<'false'> = {
    "link": "https://ourcompanyinstance.service-now.com/api/now/table/sys_user/b15cf3ebdbe11300f196f3651d961999",
    "value": "b15cf3ebdbe11300f196f3651d961999"
};

Making our complete type

With these primitives in place, we can now build ourself a (cut-down) type that models a Change Request:

export interface ServiceNowChangeRequest<TAllTrueFalse extends DisplayValue> {
    state: PropertyValue<TAllTrueFalse, State>;
    sys_id: PropertyValue<TAllTrueFalse>;
    number: PropertyValue<TAllTrueFalse>;
    requested_by: LinkValue<TAllTrueFalse>;
    reason: PropertyValue<TAllTrueFalse, string, string | null>;
    // there are *way* more properties in reality
}

This is a generic type which will accept 'all', 'true' or 'false' and will use that type to drive the type of the properties inside the object. And now we have successfully typed our Service Now Change Request, thanks to TypeScript's conditional types.

To test it out, let's take the JSON responses we got back from our curls at the start, and see if we can make ServiceNowChangeRequests with them.

const changeRequestFalse: ServiceNowChangeRequest<'false'> = {
  "state": "3",
  "sys_id": "4d54d7481b37e010d315cbb5464bcb95",
  "number": "CHG0122595",
  "requested_by": {
    "link": "https://ourcompanyinstance.service-now.com/api/now/table/sys_user/b15cf3ebdbe11300f196f3651d961999",
    "value": "b15cf3ebdbe11300f196f3651d961999"
  },
  "reason": ""
};

const changeRequestTrue: ServiceNowChangeRequest<'true'> = {
  "state": "Closed",
  "sys_id": "4d54d7481b37e010d315cbb5464bcb95",
  "number": "CHG0122595",
  "requested_by": {
    "display_value": "Sally Omer",
    "link": "https://ourcompanyinstance.service-now.com/api/now/table/sys_user/b15cf3ebdbe11300f196f3651d961999"
  },
  "reason": null
}

const changeRequestAll: ServiceNowChangeRequest<'all'> = {
  "state": {
    "display_value": "Closed",
    "value": "3"
  },
  "sys_id": {
    "display_value": "4d54d7481b37e010d315cbb5464bcb95",
    "value": "4d54d7481b37e010d315cbb5464bcb95"
  },
  "number": {
    "display_value": "CHG0122595",
    "value": "CHG0122595"
  },
  "requested_by": {
    "display_value": "Sally Omer",
    "link": "https://ourcompanyinstance.service-now.com/api/now/table/sys_user/b15cf3ebdbe11300f196f3651d961999",
    "value": "b15cf3ebdbe11300f196f3651d961999"
  },
  "reason": {
    "display_value": null,
    "value": ""
  }
}

We can! Do take a look at this in the TypeScript playground.

Sunday 25 April 2021

The Service Now API and TypeScript Conditional Types

The Service Now REST API is an API which allows you to interact with Service Now. It produces different shaped results based upon the sysparm_display_value query parameter. This post looks at how we can model these API results with TypeScripts conditional types. The aim being to minimise repetition whilst remaining strongly typed. This post is specifically about the Service Now API, but the principles around conditional type usage are generally applicable.

Service Now and TypeScript

The power of a query parameter

There is a query parameter which many endpoints in Service Nows Table API support named sysparm_display_value. The docs describe it thus:

Data retrieval operation for reference and choice fields. Based on this value, retrieves the display value and/or the actual value from the database.

Valid values:

  • true: Returns the display values for all fields.
  • false: Returns the actual values from the database.
  • all: Returns both actual and display value

Let's see what that looks like when it comes to loading a Change Request. Consider the following curls:

# sysparm_display_value=all
curl "https://ourcompanyinstance.service-now.com/api/now/table/change_request?sysparm_query=number=CHG0122585&sysparm_limit=1&sysparm_display_value=all" --request GET --header "Accept:application/json" --user 'API_USERNAME':'API_PASSWORD' | jq '.result[0] | { state, sys_id, number, requested_by, reason }'

# sysparm_display_value=true
curl "https://ourcompanyinstance.service-now.com/api/now/table/change_request?sysparm_query=number=CHG0122585&sysparm_limit=1&sysparm_display_value=true" --request GET --header "Accept:application/json" --user 'API_USERNAME':'API_PASSWORD' | jq '.result[0] | { state, sys_id, number, requested_by, reason }'

# sysparm_display_value=false
curl "https://ourcompanyinstance.service-now.com/api/now/table/change_request?sysparm_query=number=CHG0122585&sysparm_limit=1&sysparm_display_value=false" --request GET --header "Accept:application/json" --user 'API_USERNAME':'API_PASSWORD' | jq '.result[0] | { state, sys_id, number, requested_by, reason }'

When executed, they each load the same Change Request from Service Now with a different value for sysparm_display_value. You'll notice there's some jq in the mix as well. This is because there's a lot of data in a Change Request. Rather than display everything, we're displaying a subset of fields. The first curl has a sysparm_display_value value of all, the second false and the third true. What do the results look like?

sysparm_display_value=all:

{
  "state": {
    "display_value": "Closed",
    "value": "3"
  },
  "sys_id": {
    "display_value": "4d54d7481b37e010d315cbb5464bcb95",
    "value": "4d54d7481b37e010d315cbb5464bcb95"
  },
  "number": {
    "display_value": "CHG0122595",
    "value": "CHG0122595"
  },
  "requested_by": {
    "display_value": "Sally Omer",
    "link": "https://ourcompanyinstance.service-now.com/api/now/table/sys_user/b15cf3ebdbe11300f196f3651d961999",
    "value": "b15cf3ebdbe11300f196f3651d961999"
  },
  "reason": {
    "display_value": null,
    "value": ""
  }
}

sysparm_display_value=true:

{
  "state": "Closed",
  "sys_id": "4d54d7481b37e010d315cbb5464bcb95",
  "number": "CHG0122595",
  "requested_by": {
    "display_value": "Sally Omer",
    "link": "https://ourcompanyinstance.service-now.com/api/now/table/sys_user/b15cf3ebdbe11300f196f3651d961999"
  },
  "reason": null
}

sysparm_display_value=false:

{
  "state": "3",
  "sys_id": "4d54d7481b37e010d315cbb5464bcb95",
  "number": "CHG0122595",
  "requested_by": {
    "link": "https://ourcompanyinstance.service-now.com/api/now/table/sys_user/b15cf3ebdbe11300f196f3651d961999",
    "value": "b15cf3ebdbe11300f196f3651d961999"
  },
  "reason": ""
}

As you can see, we have the same properties being returned each time, but with a different shape. Let's call out some interesting highlights:

  • requested_by is always an object which contains link. It may also contain value and display_value depending upon sysparm_display_value
  • state, sys_id, number and reason are objects containing value and display_value when sysparm_display_value is all. Otherwise, the value of value or display_value is surfaced up directly; not in an object.
  • most values are strings, even if they represent another data type. So state.value is always a stringified number. The only exception to this rule is reason.display_value which can be null

Type Definition time

We want to create a type definitions for these API results. We could of course create three different results, but that would involve duplication. Boo! Also, we're looking at a subset of five properties in this example. In reality, there will be many, many properties and so duplication would become very impractical.

What can we do? Well, if all of the underlying properties were of the same type, we could use a generic and be done. But given the underlying types can vary, that's not going to work. We can achieve this though through using a combination of generics and conditional types.

Let's begin by creating a string literal type of the possible values of sysparm_display_value:

export type DisplayValue = 'all' | 'true' | 'false';

Making a PropertyValue type

Next we need to create a type that models the object with display_value and value properties.

:::info a type for state, sys_id, number and reason

  • state, sys_id, number and reason are objects containing value and display_value when sysparm_display_value is 'all'. Otherwise, the value of value or display is surfaced up directly; not in an object.
  • most values are strings, even if they represent another data type. So state.value is always a stringified number. The only exception to this rule is reason.display_value which can be null

:::

export interface ValueAndDisplayValue<TValue = string, TDisplayValue = string> {
    display_value: TDisplayValue;
    value: TValue;
}

Note that this is a generic property with default values of string for both display_value and value. most of the time, string is the type in question so it's great that TypeScript allows us to cut down on the amount of syntax we use.

Now we're going to create our first conditional type:

export type PropertyValue<
    TAllTrueFalse extends DisplayValue,
    TValue = string,
    TDisplayValue = string
> = TAllTrueFalse extends 'all'
    ? ValueAndDisplayValue<TValue, TDisplayValue>
    : TAllTrueFalse extends 'true'
    ? TDisplayValue
    : TValue;

The PropertyValue will either be a ValueAndDisplayValue, a TDisplayValue or a TValue, depending upon whether PropertyValue is 'all', 'true' or 'false' respectively. That's hard to grok. Let's look at an example of each of those cases using the reason property, which allows a TValue of string and a TDisplayValue of string | null:

const reasonAll: PropertyValue<'all', string, string | null> = {
    "display_value": null,
    "value": ""
};
const reasonTrue: PropertyValue<'true', string, string | null> = null;
const reasonFalse: PropertyValue<'false', string, string | null> = '';

Consider the type on the left and the value on the right. We're successfully modelling our PropertyValues. I've deliberately picked an edge case example to push our conditional type to its limits.

Service Now Change Request States

Let's look at another usage. We'll create a type that repesents the possible values of a Change Request's state in Service Now. Do take a moment to appreciate these values. Many engineers were lost in the numerous missions to obtain these rare and secret enums. Alas, the Service Now API docs have some significant gaps.

/** represents the possible Change Request "State" values in Service Now */
export const STATE = {
    NEW: '-5',
    ASSESS: '-4',
    SENT_FOR_APPROVAL: '-3',
    SCHEDULED: '-2',
    APPROVED: '-1',
    WAITING: '1',
    IN_PROGRESS: '2',
    COMPLETE: '3',
    ERROR: '4',
    CLOSED: '7',
} as const;

export type State = typeof STATE[keyof typeof STATE];

By combining State and PropertyValue, we can strongly type the state property of Change Requests. Consider:

const stateAll: PropertyValue<'all', State> = {
    "display_value": "Closed",
    "value": "3"
};
const stateTrue: PropertyValue<'true', State> = "Closed";
const stateFalse: PropertyValue<'false', State> = "3";

With that in place, let's turn our attention to our other natural type that the requested_by property demonstrates.

Making a LinkValue type

:::info a type for requested_by

requested_by is always an object which contains link. It may also contain value and display_value depending upon sysparm_display_value

:::

interface Link {
    link: string;
}

/** when TAllTrueFalse is 'false' */
export interface LinkAndValue extends Link {
    value: string;
}

/** when TAllTrueFalse is 'true' */
export interface LinkAndDisplayValue extends Link {
    display_value: string;
}

/** when TAllTrueFalse is 'all' */
export interface LinkValueAndDisplayValue extends LinkAndValue, LinkAndDisplayValue {}

The three types above model the different scenarios. Now we need a conditional type to make use of them:

export type LinkValue<TAllTrueFalse extends DisplayValue> = TAllTrueFalse extends 'all'
    ? LinkValueAndDisplayValue
    : TAllTrueFalse extends 'true'
    ? LinkAndDisplayValue
    : LinkAndValue;

This is hopefully simpler to read than the PropertyValue type, and if you look at the examples below you can see what usage looks like:

const requested_byAll: LinkValue<'all'> = {
    "display_value": "Sally Omer",
    "link": "https://ourcompanyinstance.service-now.com/api/now/table/sys_user/b15cf3ebdbe11300f196f3651d961999",
    "value": "b15cf3ebdbe11300f196f3651d961999"
};
const requested_byTrue: LinkValue<'true'> = {
    "display_value": "Sally Omer",
    "link": "https://ourcompanyinstance.service-now.com/api/now/table/sys_user/b15cf3ebdbe11300f196f3651d961999"
}
const requested_byFalse: LinkValue<'false'> = {
    "link": "https://ourcompanyinstance.service-now.com/api/now/table/sys_user/b15cf3ebdbe11300f196f3651d961999",
    "value": "b15cf3ebdbe11300f196f3651d961999"
};

Making our complete type

With these primitives in place, we can now build ourself a (cut-down) type that models a Change Request:

export interface ServiceNowChangeRequest<TAllTrueFalse extends DisplayValue> {
    state: PropertyValue<TAllTrueFalse, State>;
    sys_id: PropertyValue<TAllTrueFalse>;
    number: PropertyValue<TAllTrueFalse>;
    requested_by: LinkValue<TAllTrueFalse>;
    reason: PropertyValue<TAllTrueFalse, string, string | null>;
    // there are *way* more properties in reality
}

This is a generic type which will accept 'all', 'true' or 'false' and will use that type to drive the type of the properties inside the object. And now we have successfully typed our Service Now Change Request, thanks to TypeScript's conditional types.

To test it out, let's take the JSON responses we got back from our curls at the start, and see if we can make ServiceNowChangeRequests with them.

const changeRequestFalse: ServiceNowChangeRequest<'false'> = {
  "state": "3",
  "sys_id": "4d54d7481b37e010d315cbb5464bcb95",
  "number": "CHG0122595",
  "requested_by": {
    "link": "https://ourcompanyinstance.service-now.com/api/now/table/sys_user/b15cf3ebdbe11300f196f3651d961999",
    "value": "b15cf3ebdbe11300f196f3651d961999"
  },
  "reason": ""
};

const changeRequestTrue: ServiceNowChangeRequest<'true'> = {
  "state": "Closed",
  "sys_id": "4d54d7481b37e010d315cbb5464bcb95",
  "number": "CHG0122595",
  "requested_by": {
    "display_value": "Sally Omer",
    "link": "https://ourcompanyinstance.service-now.com/api/now/table/sys_user/b15cf3ebdbe11300f196f3651d961999"
  },
  "reason": null
}

const changeRequestAll: ServiceNowChangeRequest<'all'> = {
  "state": {
    "display_value": "Closed",
    "value": "3"
  },
  "sys_id": {
    "display_value": "4d54d7481b37e010d315cbb5464bcb95",
    "value": "4d54d7481b37e010d315cbb5464bcb95"
  },
  "number": {
    "display_value": "CHG0122595",
    "value": "CHG0122595"
  },
  "requested_by": {
    "display_value": "Sally Omer",
    "link": "https://ourcompanyinstance.service-now.com/api/now/table/sys_user/b15cf3ebdbe11300f196f3651d961999",
    "value": "b15cf3ebdbe11300f196f3651d961999"
  },
  "reason": {
    "display_value": null,
    "value": ""
  }
}

We can! Do take a look at this in the TypeScript playground.

Monday 19 April 2021

ts-loader goes webpack 5

ts-loader has just released v9.0.0. This post goes through what this release is all about, and what it took to ship this version. For intrigue, it includes a brief scamper into my mental health along the way. Some upgrades go smoothly - this one had some hiccups. But we'll get into that.

hello world bicep

One big pull request

As of v8, ts-loader supported webpack 4 and webpack 5. However the webpack 5 support was best efforts, and not protected by any automated tests. ts-loader has two test packs:

  1. A comparison test pack that compares transpilation and webpack compilation output with known outputs.
  2. An execution test pack that executes Karma test packs written in TypeScript using ts-loader.

The test packs were tightly coupled to webpack 4 (and in the case of the comparison test pack, that's unavoidable). The mission was to port ts-loader to be built against (and have an automated test pack that ran against) webpack 5.

This ended up being a very big pull request. Work on it started back in February 2021 and we're shipping now in April of 2021. I'd initially expected it would take a couple of days at most. I had underestimated.

A number of people collaborated on this PR, either with code, feedback, testing or even just responding to questions. So I'd like to say thank you to:

What's changed

Let's go through what's different in v9. There's two breaking changes:

  • The minimum webpack version supported is now webpack 5. This simplifies the codebase, which previously had to if/else the various API registrations based on the version of webpack being used.
  • The minimum node version supported is now node 12. Node 10 reaches end of life status at the end of April 2021.

An interesting aspect of migrating to building against webpack 5 was dropping the dependency upon @types/webpack in favour of the types that now ship with webpack 5 itself. This was a mostly great experience; however we discovered some missing pieces.

Most notably, the LoaderContext wasn't strongly typed. LoaderContext is the value of this in the context of a running loader function. So it is probably the most interesting and important type from the perspective of a loader author.

Historically we used our own definition which had been adapted from the one in @types/webpack. I've looked into the possibility of a type being exposed in webpack itself. However, it turns out, it's complicated - with the LoaderContext type being effectively created across two packages. The type is initially created in webpack and then augmented later in loader-runner, prior to being supplied to loaders. You can read more on that here.

For now we've opted to stick with keeping an interface in ts-loader that models what arrives in the loader when executed. We have freshened it up somewhat, to model the webpack 5 world.

Alongside these changes, a number of dependencies were upgraded.

The hole

By the 19th of February most of the work was done. However, we were experiencing different behaviour between Linux and Windows in our comparison test pack.

As far as I was aware, we were doing all the appropriate work to ensure ts-loader and our test packs worked cross platform. But we were still experiencing problems whenever we ran the test pack on Windows. I'd done no end of tweaking but nothing worked. I couldn't explain it. I couldn't fix it. I was finding that tough to deal with.

I really want to be transparent about the warts and all aspect of open source software development. It is like all other types of software development; sometimes things go wrong and it can be tough to work out why. Right then, I was really quite unhappy. Things weren't working code-wise and I was at a loss to say why. This is not something that I dig.

I also wasn't sleeping amazingly at this point. It was winter and we'd been in lockdown in the UK for three months; as the COVID-19 pandemic ground relentlessly on. I love my family dearly. I really do. With that said, having my children around whilst I attempted to work was remarkably tough. I love those guys but, woah, was it stressful.

I was feeling at a low ebb. And I wasn't sure what to do next. So, feeling tired and pretty fed up, I took a break.

"Anybody down there?"

Time passed. In March Alexander Akait checked in to see how things were going and volunteered to help. He also suggested what turned out to be the fix; namely replacing usage of '\' with '/' in the assets supplied back to webpack. But crucially I implemented this wrong. Observe this commit:

const assetPath = path
  .relative(compilation.compiler.outputPath, outputFile.name)
  // According to @alexander-akait we should always '/' https://github.com/TypeStrong/ts-loader/pull/1251#issuecomment-799606985
  .replace(/\//g, '/');

If you look closely at the replace you'll see that I'm globally replacing '/' with '/' rather than globally replacing '\' with '/'. The wasted time this caused… I could weep.

I generally thrashed around for a bit after this. Going in circles, like a six year old swimming wearing one armband. Then Tobias kindly volunteered to help. This much I've learned from a career in software: if talented people offer their assistance, grab it with both hands!

I'd been trying be as "learn in public" as possible about the issues I was facing on the pull request. The idea being, to surface the problems in a public forum where others can read and advise. And also to attempt a textual kind of rubber duck debugging.

When Tobias pitched in, I wanted to make it as easy as possible for him to help. So I wrote up a full description of what had changed. What the divergent behaviour in test packs looked like. I shared my speculation for what might be causing the issue (I was wrong by the way). Finally I provided a simple way to get up and running with the broken code. The easier I could make it for others to collaborate on this, I figured, the greater the likelihood of an answer. Tobias got to an answer quickly:

The problem is introduced due to some normalization logic in the test case: see #1273

While the PR fixes the problem, I think the paths should be normalized earlier in the pipeline to make this normalization code unnecessary. Note that asset names should have only / as they are filenames and not paths. Only absolute paths have \.

Tobias had raised a PR which introduced a workaround to resolved things in the test pack. This made me happy. More than that, he also identified that the issue lay in ts-loader itself. This caused me to look again at the changes I'd made, including my replace addition. With fresh eyes, I now realised this was a bug, and fixed it.

I found then that I could revert Tobias' workaround and still have passing tests. Result!

Release details

Now that we've got there; we've shipped. You can get the latest version of ts-loader on npm and you can find the release details on GitHub.

Thanks everyone - I couldn't have done it without your help. 🌻❤️

Saturday 10 April 2021

Hello World Bicep

Bicep makes Azure Resource Management a great deal simpler than ARM templates. The selling point here is grokkability. This post takes a look at the "Hello World" example recently added to the Bicep repo to appreciate quite what a difference it makes.

hello world bicep

More than configuration

The "Hello World" added to the Bicep repo by Chris Lewis illustrates the simplest usage of Bicep:

This bicep file takes a yourName parameter and adds that to a hello variable and returns the concatenated string as an ARM output.

This is, when you consider it, the very essence of a computer program. Taking an input, doing some computation and providing an output. When I think about ARM templates, (and because Bicep is transpiled into ARM templates I mentally bracket the two together) I tend to think about resources being deployed. I focus on configuration, not computation

This is an imperfect mental model. ARM templates can do so much more than deploy by slinging strings and numbers. Thanks to the wealth of template functions that exist they have much more power. They can do computation.

The Hello World example focuses just on computation.

From terse to verbose

The Hello World example is made up of two significant files:

  1. main.bicep - the bicep code
  2. main.json - the ARM template compiled from the Bicep file

The main.bicep file amounts to 3 lines of code (I have omitted the comment line):

param yourName string
var hello = 'Hello World! - Hi'

output helloWorld string = '${hello} ${yourName}'
  • the first line takes the input of yourName
  • the second line declares a hello variable
  • the third line computes the new value of helloWorld based upon hello and yourName, then passes it as output

Gosh is it ever simple. It's easy to read and it's simple to understand. Even if you don't know Bicep, if you've experience in another language you can likely guess what's happening.

Let's compare this with the main.json that main.bicep is transpiled into:

{
  "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
  "contentVersion": "1.0.0.0",
  "metadata": {
    "_generator": {
      "name": "bicep",
      "version": "dev",
      "templateHash": "6989941473549654446"
    }
  },
  "parameters": {
    "yourName": {
      "type": "string"
    }
  },
  "functions": [],
  "variables": {
    "hello": "Hello World! - Hi"
  },
  "resources": [],
  "outputs": {
    "helloWorld": {
      "type": "string",
      "value": "[format('{0} {1}', variables('hello'), parameters('yourName'))]"
    }
  }
}

The above ARM template expresses exactly the same thing as the Bicep alternative. But that 3 lines of logic has become 27 lines of JSON. We've lost something in the transition. Intent is no longer clear. We've gone from something easy to reason about, to something that is hard to reason about. You need to think a lot less to write the Bicep alternative and that's a good thing.

I was chatting to someone recently who expressed it well by saying:

ARM is the format that the resource providers understand, so really it’s the Azure equivalent of Assembler – and I don’t know anyone who enjoys coding in Assembler.

This is a great example of the value that Bicep provides. If you'd like to play with the Hello World a little, why not take it for a spin in the Bicep playground.