New Web (API) Foundations (Part One)

Published: 2021-10-16

Author: Teddi

A foreword: This blog post (and the follow-ups) is a bit more technical than usual and so there won’t be too much in terms of the depths of explaining how they work but instead, terms will be used with at most brief explanations. You should be able to follow the general gist though as some concepts and explanations will be mentioned.

Much like how I’ve rewrote the starting paragraph to this blog post three times now, trying to extract data from our databases in meaningful ways is tricky just like getting the information from my brain to talk about is also tricky.

See I already know this information much like the database knows its’ information but I have to find a medium and a form that allows me to relay this information to you the reader. Additionally I have to process and parse exactly what I can tell you, what might be oversharing and what might not even be enough. I want you to have the exact information you need and no more, no less.

Four to five years ago I wrote the first version of the BB Web API in Python with the goal of having a way to extract data from the database without having to actually have a direct connection to the database. One could argue that there was no direct need for it because ultimately I hold the keys to the kingdom and if I ever needed to generate database credentials for an application, well that’s not exactly difficult for me. But what if I don’t want to do that? What if I want to make it easier for other developers to query the same data in a reliable fashion and that doesn’t require me generating a set of credentials every time. Furthermore what if I only want to give a portion of data as opposed to the entire lump? So off we went.

With reflection, it was an abject failure. Don’t get me wrong; the API works and is still used by a few smaller systems today (with the desire to sunset these endpoints) but the underlying goals that this API set out to solve were never actually solved. There’s no proper authentication or authorization meaning if you know the right incantations and routes to hit you can load the data of any user (minus some personal information such as IPs). As a result what was meant to have eventually become a public API was forever kept internal and very much underutilized. Not only was it underutilized but it couldn’t even be used within our GMod systems on the client for risk of someone going digging and fetching data about players that we felt would otherwise wouldn’t be ideal to have.

Additionally what documentation there was had to be kept internal and required me to manually write and update each and every time which is bit of a pain. You can get tools which help with making REST APIs but in all honesty, they’re not that great and arguably in many cases hinder you rather than help you.

Finally, the toolset used caused additional problems. While I quite like using Python (and still use it for many projects and systems today) I was using bottlepy which is a web framework similar to Flask and Django (but still closer to Flask). Historically I generally picked bottlepy because it’s a very lightweight framework, only the problem is it’s currently halfway between version 0.12 and 0.13 which creates some hilarious inconsistencies at best and downright issues at worst. Going further hosting web applications in Python is… not for the faint-hearted. The tooling available to you outside of managed systems (e.g. Heroku) aren’t exactly great and come with their own set of problems so a very small web API that had less than 20 routes wasn’t always the fastest and was at least 80mb in size. Still smaller than node.js though. The actual management of this got easier over time with some adjustments but it still wasn’t great.This is more of a reflective point overall because my general go-to for web based services is now Golang (more on this in later posts).

We’ve always had the occasional request for exposed web endpoints as people want to do things with the data we have on hand. This might be just to use the data to build a better ban page than what we had, crunch stats about their timetrial data or just even fun little tools or a peak behind the curtain but we could never provide for that. Furthermore I was starting to seek more often a way to pull data from the database and expanding the (now named) version 1 web API always felt like a chore.

And so, the brain-storming and gear-turning process started to answer the question: What would a version 2 of the API look even look like?

Tune in for Part 2 appearing in the next few days (hopefully)!