The Market Exploit – Part 1

Published: 2020-10-31

Author: Santamon

As some players are aware, we were made notified of an exploit in the marketplace, that allows for the duplication of cubes. I wanted to create this blog post to explore, technically, what went wrong. Before we go into that, I should make it clear that the player in question had all of their marketplace transactions reversed. This is why some players got strange deliveries that removed cubes from their inventory, and also saw items they sold re-appear in their inventory. Reversing market transactions is not something we take lightly, as we generally have a zero-refund policy on any transactions via the marketplace.

Anyone, onto the blog!

Preamble

To perform the marketplace exploit, the player in question used a client-side hack (Which itself is against the rules, but anyway) and they crafted messages that replicated the marketplace buy listing request. If there were spammed at very high rates (Hundreds a second), then it would duplicate the cubes they had spent.

So, how did this happen? First, we need to understand a little bit of technical background as it relates to how the server, speaks to the database.

So, the Lua engine in gmod is entirely single-threaded – this means that all Lua code that is due to be invoked in an engine tick must run to completion. Basically, all code paths must terminate at a return (Or a yield if a coroutine). This is a bit of a headache, but entirely manageable – it does however present a problem when we’re dealing with external systems – such as a database.

Let us look at Python as an example, when you code a MySQL query, you have the queries in-line, so something like this:

What this means is that the python code halts until the database returns the results of each query. When you’re coding a web application, this is acceptable, however, for gmod, this would mean that the Lua engine would pause until the database returns the results of the query. Every time that a query was performed, players would feel a slight jolt in the server while it waited for the database to return. The server can run dozens of queries a second, so obviously, this is not acceptable behaviour.

Instead, in gmod, we have an asynchronous model – this means we send the query to an external DLL, which contains its own threading. This thread holds our query external to gmod, to allow the lua engine to continue its execution. Once the database returns the results, the module executes a callback, which then triggers next tick. We say this is asynchronous as the server continues as normal until we have our results.

In summary, you get something like this. This is an example, not the section of code where the exploit existed.

So, we run a query, the database runs the query, returns the results, and then the callback (GetListingCountsResults) is executed (line 444), which contains the data we want as an arg (see below). We then do whatever we need to do to that – in this instance, we send the counts for each category to the player (Using WriteTable is expensive for net messages, I know! But it’s rare and I was lazy).

So, why is all this important? Well, this means that if the same query is running multiple times in the same frame, then it will get the same results back. Unlike in the python example, where we would select, and update our rows in line, so when we next call the select, it would show the updated results. In-line queries also work nicely with transactions – async methods can implement transactions, but it’s a lot more complicated and support was flaky in Gmod SQL modules until recently.

So, how did the exploit work?

The marketplace, when it comes to buying an item, has a pretty simple flow process. When you go to buy an item, we charge you the purchase price (On the server), we then tell the marketplace (Which operates as an external system, hence it’s cross-server) we want to buy an item. The marketplace then gets a lock on the database row (AKA a semaphore), after which, we check if it has been bought or not. If it has not been bought, we create a delivery for the item; otherwise, we create a delivery for the cubes we just charged you as.  But why can’t the marketplace just tell the server to refund the cubes? Rather than creating a delivery? There’s no guarantee that the player will still be connected to the server and we can’t give cubes to a player who is no longer connected on the server outside the delivery system. The underlying reason for this (Which deserves its own blog post) is why you can’t enter the lottery from Discord.

Okay, so you spammed the market place, generated hundreds of transactions for the marketplace to refund you cubes – so what? You were charged these cubes in the first place, so you are owed them. Now, this is where the subtlety exists. When the marketplace tells the server that it’s done what it has asked, then the server asks the delivery system to check a player’s mail in 5 seconds. A deficiency in the code meant that it would start as many checks for every time you tried to buy a listing – rather than just one check.

What this means, is that within the same frame, in 5 seconds time, the server will repeatedly check for deliveries. As this is asynchronous, we will then, next frame, get all these selects back – all containing duplicate copies of the player’s mailbox – including the cube refunds. The delivery system, thinking that these are all legitimate it would then redeem all of these. This would, in effect, duplicate the number of cubes you spent. The delivery system would mark these deliveries as redeemed, but at this point, it’s too late. Next frame, after the database has updated, any queries would, of course, recognise they have been redeemed, however at this point, it’s too late.

So, what should have happened? In an ideal world, a transaction in the delivery system would have locked the rows to stop anyone else selecting them until they had been updated. This means that other queries would be queued until the transaction in progress releases the semaphore. As noted, before, transactions are troublesome in Gmod, so we should have had a semaphore on the server which basically said: “A system is already checking the mail, you have to wait!“. This didn’t exist because the delivery system was designed before the market system, and thus, mailbox checks were only done once when a player joined the server so it was never really needed.

Generally speaking, we’re pretty defensive when it comes to coding. All of our net interfaces are/should be protected by a centralised system to stop them being spammed (See line 439 in the second figure) – this was missing on the market buy item interface. We have now added this to prevent the spamming, but we have also added a semaphore into the delivery system to stop multiple mailbox checks running in parallel.

Join us for Part 2, where we pick up the pieces!