Welcome! Plus, an update on ocean currents.

Welcome to the first post of the earth.nullschool.net newsletter! I'm Cameron, the creator of earth.nullschool.net.

This won't be a busy newsletter. Perhaps a couple posts per month. I'll announce new features, answer common questions, and run surveys to understand changes users want. I think it could also be interesting to describe behind-the-scenes details on what it takes to keep the site running smoothly.

About that last point…

Recently, I noticed a problem with ocean currents data. There hadn’t been any update for eight days! This was three days longer than usual. Not a crisis, but it required investigation.

Many people ask how I produce the data shown on the website. Short answer: I don't [*]. Instead, data is produced by teams of scientists and engineers at research organizations and government agencies with supercomputers, satellites, and other fancy machines. Thankfully and fortunately, many of them publish their data for free so people like me can download, process, and convert it into pretty pixels on your screen. And just like any complex system, sometimes things break. Or temporarily slow down. Or change.

The afore-mentioned ocean currents data comes from a model called OSCAR operated by Earth and Space Research, a non-profit research institute. I don’t download the data directly from them. Instead, OSCAR results are published at NASA's Physical Oceanography Distributed Active Archive Center (PO.DAAC), a gigantic repository of Earth data.

A quick email to the admins of PO.DAAC confirmed the problem with OSCAR was already known and temporary, and the data flow would soon resume.

They also said this version of OSCAR would soon be turned off! Uh-oh.

Its replacement, OSCAR v2, is an upgraded model with 30% higher resolution, daily time steps instead of 5-day steps, and an archive backfilled to 1993. Pretty neat. After a few days of coding to support the new version, this was the result:

Hmmm. Clearly something was wrong. The colors were correct and features like the equatorial currents were visible, but everything was badly warped. This implied file decoding was working but the mapping of data to latitude and longitude coordinates on the globe was broken. Further investigation uncovered a decade-old assumption baked into my source code.

Here's a table of numbers:

Let's say someone asked you to read out the numbers in this table over the phone. What would you say? "2, 7, 3, 6, 1, 0, 8, 3, 9"? Or perhaps "2, 6, 8, 7, 1, 3, 3, 0, 9"? Something else?

To accurately communicate the table, both you and the other person on the line would need to first agree on the ordering. If you agreed on row-major order, the numbers would be read out as 2, 7, 3, 6, etc. Conversely, if you agreed on column-major order, the numbers would be 2, 6, 8, 7, etc.

The mapping logic used by the site implicitly assumed row-major order. This worked for all datasets added over the years, hiding the flaw. But OSCAR v2 data is in column-major order! Oops. The fix, naturally, was to add support for column-major ordering.

OSCAR v2 is now live. And you, as newsletter subscribers, are the first to know! Give it a try.

I also thought it would be nice to publish the entire archive of historical data going back to 1993. It took a bit of time to download and process 350 GB of raw data, but it's all there now. (Note that to access data before 2013, you'll need to manually modify the URL.) With this new data, I’m planning to create a multi-decade time lapse animation of ocean currents and post it on the earth.nullschool.net YouTube channel sometime soon.

That's it for this first post. Hope you have a wonderful day. Until next time. 🌏🌍🌎

[*] I compute the Misery Index and Wind Power Density layers, but these are simple combinations of other layers.