Tokenomics of Solving The Disobey 2027 Hacker Puzzle
Note. This post contains spoilers. All the hacker tickets are gone, so here are some solutions being mentioned. If you are saving it to work through yourself,stop here.
So, I finished the Disobey 2027 hacker puzzle in an attempt to push my harness + LLM.
I wanted to see how far my own harness could get with as little of me in the loop as possible. The puzzle had 8 checkpoints and required interaction with the IRC channel.
While I write this, I also think about Daniel Miessler’s piece called Keep the Robots Out of the Gym. He is right. I did the opposite here, on purpose, and I will come back to whether that was a good idea.
What it cost
Purely the amount of tokens: roughly 1.16 billion.
This is not 100% accurate, as about 98% of it is cache reads - the same conversation being re-read at the top of every turn. The actual number is mostly the output and is close to: 3.24 million tokens.
At current Opus rates that is somewhere between $780 and $858 in API costs. Comparitavely 8 or 9 times cheaper with subscription.
One checkpoint took two thirds of it
| checkpoint | what it was | cost | share |
|---|---|---|---|
| 1 arrivals | a message filed with a 1994 timestamp | $25 | 2.9% |
| 2 kouvostonet | a 1994 BBS with a three-word greeting | $171 | 19.9% |
| 3 intranet | SSL 3.0 only, gated on the client’s clock | $30 | 3.5% |
| 4 securedripper | one character disclosed per minute | $29 | 3.4% |
| 5 mysterymachinecode | an x86 boot sector that plays chess | $34 | 4.0% |
| 6 agent-farm | prompt injection through a hidden PDF layer | $27 | 3.1% |
| 7 and 8 gatekeeper | one ROM, two processors | $543 | 63.3% |
The last checkpoint took 63% of the total, more than the other six combined, and it was the hardest one of them all. You get a 7KB binary and three scanned schematics. The trick is that the schematics describe two processors - an 8051 and an 8085 - sharing one dual- port RAM, and the binary contains the code for both. Disassembling it as only one architecture makes the RE confusing. This is where my LLM and harness struggled and a lot of steerign was required.
Nothing off the shelf runs it, since QEMU has no 8051 target, so at one point the LLM ended up writing two emulators, a disassembler and a bridge between them. :)
All that emulator work was useless, and the flag turns out to be computable in about six lines of Python with no emulation at all: the key is an 11 by 11 linear system mod 256, the payload a repeating-key XOR, the flag another XOR. My harness and LLM decided to build a machine to run the thing when it could have done the arithmetic.
So, the learning was that ‘cost tracks unfamiliarity, not difficulty’. For example, solving the chess-playing boot sector cost $34 because emulating x86 is possible, compared to understanidng the unkown territory and digging in loops. This makes hard problem in a known form cheap and an easy problem in an unknown form expensive. Most tokens were spent in exploring, falling into rabbit holes, pulling itself out and repeating until all options are covered.
Doing it by hand
One way to reduce the token usage is to do most of the setup and exploration and reading yourself - your own IRC client, your own emulator, your own requests - and use the LLM only to answer questions about what you are looking at.
What will the costs be then?:
| how you work | what it costs |
|---|---|
| hands off, harness drives everything | $780 to $858 |
| same, without my personal scaffolding | $735 to $817 |
| you drive, LLM answers questions | $12 to $105 |
The difference comes down to plumbing yourself or write a plumber (your harness) that does the plumbing for you. In my LLM transcript, 80% of all content is tool results: logs, hex dumps, disassembly, HTTP responses, each one re-read on every later turn. When a human runs the tools and pastes in only the interesting part, most of that never enters the context at all.
In this case, I didnt’t work this way, so treat the range as loose direction, not an accurate number.
So, are CTFs dead?
I do not think so, but the thing they measure has moved.
Back to Daniel’s split, job versus gym. At work the point is that the weight gets moved, so let the machine lift it. At the gym the point is that you lift it, and a robot doing your squats is doing nothing for you.
A CTF is a gym for the most part (Or at least these started as one). It is a receipt for having learned something.
But security is also my job, making the line blurrier for me. A good part of my week is similar workflow: unfamiliar target, no documentation, work out what it does. If I use an LLM for that on a Monday and call it leverage, calling it cheating on a Saturday needs a better argument than I have.
And I did learn things. Different things from what solving it by hand would have taught me: how to tell when an LLM is confidently wrong, when to stop trusting a line of reasoning and demand an experiment. That is a different gym, and I am not sure it is a worse one :)
To conclude, If you run a CTF: spend your effort on the format, not on detection. You will not out-filter this.
If you solve one with an LLM: at least be honest with yourself about which gym you were in.