Under the Duck: An Afternoon in Open Source

Have you ever wondered what happens behind the scenes of open source projects? One that I'm involved with is ts-loader; a TypeScript loader for webpack. Yesterday was an interesting day in the life of ts-loader and webpack; things unexpectedly broke. Oh and don't worry, they're fixed now.

How things panned out reflects well on the webpack community. I thought it might be instructive to take a look at the legs furiously paddling underneath the duck of open source. What follows is a minute by minute account of my life on the afternoon of Wednesday 22nd February 2017:

3:55pm

I'm sat at my desk in the City of London. I have to leave at 4pm to go to the dentist. I'm working away on a project which is built and bundled using ts-loader and webpack. However, having just npm installed and tried to spin up webpack in watch mode, I discover that everything is broken. Watch mode is not working - there's an error being thrown in ts-loader. It's to do with a webpack property called mtimes. ts-loader depends upon it and it looks like it is no longer always passed through. Go figure.

4:01pm

I've got to go. I'm 15 minutes from Bank station. So, I grab my bag and scarper out the door. On my phone I notice an issue has been raised - other people are being affected by the problem too. As I trot down the various alleys that lead to the station I wonder whether I can work around this issue. Using GitHub to fork, edit code and submit a PR on a mobile phone is possible. Just. But it's certainly not easy...

My PR is in, the various test packs are starting to execute somewhere out there in Travis and Appveyor-land. Then I notice Ed Bishop has submitted a near identical PR. Yay Ed! I'm always keen to encourage people to contribute and so I intend to merge that PR rather than my own.

16:12

Rubbish. The Waterloo and City Line is out of action. I need to get across London to reach Waterloo or I'll miss my appointment. It's time to start running....

16:15

It's rather nagging at me that behaviour has changed without warning. This has been reliably in place the entire time I've been involved with ts-loader / webpack. Why now? I don't see any obvious mentions on the webpack GitHub repo. So I head over to the webpack Slack channel and ask: (conversation slightly abridged)

johnny_reilly

Hey all, has something happened to mtimes? Behaviour seems to have changed - now undefined occasionally during watch mode. A PR has been raised against ts-loader to work around this https://github.com/TypeStrong/ts-loader/pull/480#issuecomment-281714600

However I'm wondering if this should actually be merged given behaviour has changed unexpectedly

sokra

ah...

i removed it. I thought it was unused.

johnny_reilly

It's definitely not!

sokra

it's not in the public API^^

Any reason why you are not using getTimes()?

...

johnny_reilly

Okay, I'm on a train and won't be near a computer for a while. ts-loader is presently broken because it depends on mtimes. Would it be possible for you to add this back at least for now. I'm aware many people depend on ts-loader and are now broken.

sokra

sure, I readd it but deprecate it.

...

sean.larkin

@sokra is this the change you just made for that watchpack bug fix? Or unlrelated, just wanted to track if I didn't already have the change/issue

sokra

https://github.com/webpack/watchpack/pull/48

johnny_reilly

This is what the present code does:

const watcher = watching.compiler.watchFileSystem.watcher || 
                watching.compiler.watchFileSystem.wfs.watcher

And then .mtimes

Should I be able to do .getTimes() instead?

sokra

actually you can't rely on watchFileSystem being NodeJsWatchFileSystem. But this is another topic

...

but yes

johnny_reilly

Thanks @sokra - when I get to a keyboard I'll swap mtimes for getTimes() and report back.

17:28

Despite various trains being out of action / missing in action I've made it to the dentists; phew! I go in for my checkup and plan to take a look at the issue later that evening. In the meantime I've hoping that Tobias (Sokra) will get chance to republish so that ts-loader users aren't too impacted.

18:00

Done at the dentist and I'm heading home. Whilst I've been opening wide and squinting at the ceiling, TypeScript 2.2 has shipped. Whilst this is super exciting, according to Greenkeeper, the new version has broken the build. Arrrrghhhh...

I start to look into this and realise we're not broken because of TypeScript 2.2; we were broken because of the mtimes. Tobias has now re-added mtimes and published. With that in place I requeue a build and.... drum roll.... we're green!

The good news just keeps on coming as Luka Zakrajšek has submitted a PR which uses getTimes() in place of mtimes. And the tests pass. Awesome! MERGE. I just need to cut a release and we're done.

18:15

I'm home. My youngest son has been suffering from chicken pox all week and as a result my wife has been in isolation, taking care of him. We chat whilst the boys watch Paw Patrol as the bath runs. I flick open the laptop and start doing the various housekeeping tasks around cutting a release. This is interrupted by various bathtime / bedtime activities and I abandon work for now.

19:30

The boys are down and I get on with the release; updating the changelog, bumping the version number and running the tests. For various reasons this takes longer than it normally does.

20:30

Finally we're there; ts-loader 2.0.1 ships: https://github.com/TypeStrong/ts-loader/releases/tag/v2.0.1.

I'm tremendously grateful to everyone that helped out - thank you all!

ts-loader 2.0.1 has shipped; thanks @wsokra @bancek and @mredbishop https://t.co/I00c7sJyFo #typescript

— John Reilly (@johnny_reilly) February 22, 2017

@types is rogue

Or perhaps I should call this "@types and repeatable builds"....

The other day, on a React / TypeScript project I work on, the nightly CI build started failing. But nothing had changed in the project... What gives? After digging I discovered the reason; spome of the type definitions which my project depends upon had changed. Why did this break my build? Let’s learn some more...

We acquire type definitions via npm. Type definitions from Definitely Typed are published to npm by an automated process and they are all published under the @types namespace on npm. So, the react type definition is published as the @types/react package, the node type definition is published as the @types/node package. The hip bone's connected to the thigh bone. You get the picture.

The npm ecosystem is essentially built on top of semantic versioning and they take it seriously. Essentially, when a package is published it should be categorised as a major release (breaking changes), a minor release (extra functionality which is backwards compatible) or a patch release (backwards compatible bug fixes).

Now we get to the meat of the matter: @types is rogue. You cannot trust the version numbers on @types packages to respect semantic versioning. They don't.

The main reason for this is that when it comes to versioning, the @types type definition essentially looks to mirror the version of the package they are seeking to type. THIS MEANS THE TYPE DEFINITION CANNOT DO ITS OWN SEMANTIC VERSIONING. A simple change in a type definition can lead to breakages in consuming code. That's what happened to me. Let's say an exported interface name changes; all code that relies upon the old name will now break. You see? Pain.

How do we respond to this?

My own take has been to pin the version numbers of @types packages; fixing to specific definitions. No "~" or "^" for my @types devDependencies.

No respect semantic versioning? No problem. You can go much further with repeatable builds and made use of facebook's new npm client yarn and lockfiles (very popular BTW) but I haven't felt the need yet. This should be ample for now.

The other question that may be nagging at your subconscious is this: what’s an easy way to know when new packages are available for my project dependencies? Well, the Get-Package -Updates (nuget hat tip) for npm that I’d recommend is this: npm-check-updates. It does the job wonderfully.

Hands-free HTTPS

I have had a *great* week. You? Take a look at this blog. Can you see what I can see? Here's a clue:

Yup, look at the top left hand corner.... see that beautiful padlock? Yeah - that's what's thrilled me. You see I have a dream; that one day on the red hills of the internet, the sons of former certificates and the sons of former certificate authorities will be able to sit down together at the table of HTTPS. Peace, love and TLS for all.

The world is turning and slowly but surely HTTPS is becoming the default of the web. Search results get ranked higher if they're HTTPS. HTTP/2 is, to all intents and purposes, a HTTPS-only game. Service Workers are HTTPS-only.

I care about all of these. So it's essential that I have HTTPS. But. But. But... Certificates, the administration that goes with them. It's boring. I mean, it just is. I want to be building interesting apps, I don't want to be devoting my time to acquiring certificates and fighting my way through the (never simple) administration of them. I'm dimly aware that there's free certificates to be had thanks to the fine work of LetsEncrypt. I believe that work is being done on reduce the onerous admin burden as well. And that's great. But I'm still avoiding it...

What if I told you you could have HTTPS on your blog, on your Azure websites, on your anywhere.... FOR FREE. IN FIVE MINUTES?. Well, you can thanks to CloudFlare. I did; you should too.

This is where I point you off to a number of resources to help you on your HTTPS way:

1. Read Troy Hunt's "How to get your SSL for free on a Shared Azure website with CloudFlare"
2. Watch Troy Hunt's Pluralsight course "Getting Started with CloudFlare™ Security"
3. Go to Cloudflare's website and sign up

It just works. And that makes me very happy indeed.