Fix Apple's GeoTrust APNS Cert Problem

Apple’s APNs (Apple Push Notification service) servers have started to act up last weekend and there was a lot of confusion about it at first. This is a rare occurrence since APNs and iMessage appear to be Apple’s only rock solid server side services while everything else appears to be regularly operated with a staff count of minus one. By started to act up I specifically mean that the certificate which the service has been using could no longer be verified by many servers after a ca-certificates package update went out removing the root CA (little bit of context here). Lots of servers have probably started to show an error message similar to this:

Feb 10 15:53:55 guardian-example-server service.elf[31376]: sendPushNotification(): APNs request failed: Post "https://api.push.apple.com/3/device/<token>": x509: certificate signed by unknown authority

This happened to us at Guardian as well and I only caught it by accident. This lead to none of our Pro subscribers being able to get real time push notifications, which is a feature I had poured a lot of work into last summer to get right and had to reimplement it a couple of times. All of that work was instantly disabled when the certificate was kicked out the trust store on all of our VPN nodes. I did not want to re-install the certificate system wide again, since GeoTrust appears to be not trustworthy and it would have required me to run commands through a SSH session on way too many servers.

So I resorted to being lazy and disabled the TLS certificate verification for that one HTTP request. Every other outbound network connection would still fail, if they were to try to connect to a host which also served a TLS certificate signed by the same GeoTrust certificate Apple will continue to use until March 29th 2021. This was my lazy initial solution to this problem which never made it into production because @chronic instantly kicking me in the butt about it, and rightfully so. This is a bad practise and should not be used in a production environment in 2021!
He suggested dropping the certificate in the typical .pem format onto the filesystem of all hosts and add it to a temp trust store for that one request instead (code below basically only requires a ioutil.ReadFile() if you wanted to do that). This would mean that come March 30th we’d have a file lingering on all hosts that I did not want to be there. So manual upload now, use it for a couple of weeks and then manual removal. Too much potential for human failure if you ask me.
I ended up with a modified approach of what Will had suggested, but instead of reading a file off the filesystem I decided to embed the GeoTrust certificate into our binary since it really wasn’t a lot of data.

This, in my opinion, is the right way to solve this problem until March 29th 2021 in Go, but I am sure all server side languages offer a similar API. It allows you to establish a verifiable TLS connection now, while not jeopardising the integrity of your entire system’s trust store and enables easy removal once Apple starts using the new certificate.

In order to solve this problem the right way I first downloaded the GeoTrust certificate from their website, to which I was still able to establish a trusted connection since macOS still trusts the certificate.

Link to the GeoTrust certificate
I am not including the entire certificate here for a good reason. Verify it for yourself, you shouldn’t trust me!

var (
	geoTrustRootCA = []byte(`-----BEGIN CERTIFICATE-----
	MIIDVDCCAjygAwIBAgIDAjRWMA0GCSqGSIb3DQEBBQUAMEIxCzAJBgNVBAYTAlVT
	....
	5fEWCRE11azbJHFwLJhWC9kXtNHjUStedejV0NxPNO3CBWaAocvmMw==
	-----END CERTIFICATE-----`)
)

Add the contents of the GeoTrust certificate as a variable any which way you prefer, it just needs to be accessible to x509.AppendCertsFromPEM() as a byte array.
In this case here, I created a global variable by casting a raw literal string to a byte array and assigning all that to geoTrustRootCA.

	certpool := x509.NewCertPool()
	certpool.AppendCertsFromPEM(geoTrustRootCA)

	transport := http.DefaultTransport.(*http.Transport).Clone()
	transport.TLSClientConfig = &tls.Config {
		RootCAs: certpool,
	}

	httpClient := &http.Client {
		Timeout: 15 * time.Second,
		Transport: transport,
	}
	
	resp, reqErr := httpClient.Do(...)

Here I create a new x509 certificate pool, add only the GeoTrust certificate and then include it for use in the HTTP client’s transport object by first creating a copy of the default default HTTP Transport setup. This http client will allow you to safely make a successful API call to the APNs endpoint for now. This may or may not break the second Apple rolls out the certificate on March 29th so be ready and setup and alarm or something…

13.02.2021