The attacker does not wait for an audit
Shieldify stands between a website and everyone who wants to reach it. If our edge can be crashed, tricked or worn down, so can every website behind it. Attackers test that every day, for free. We would rather find the weak spots first.
So we set two days aside and attacked our own stack: the edge we wrote in Rust, the web application firewall it runs, and Breakwave, our XDP program in the network driver. The rules were simple. Real attacks only, at real speed. Labs that run the same software and settings as production. And a machine outside our network, attacking a test website of ours on the live edge, the way a stranger would.
Hundreds of millions of malformed inputs
Everything the internet sends us is read by code we wrote. A parser that panics on odd input takes the edge down with it; one that takes too long on odd input is a cheap way to slow it down. We fuzzed it in three layers:
- The parsers. TLS ClientHellos, HTTP/1.1 heads, HTTP/2 frames, host names and paths, forms, Range headers, cache dates and TCP options: 352 million mutated inputs in three minutes. No panic and no slow call. To check the fuzzer itself, we planted a bug; it was found within seconds.
- The whole edge. About 133,000 mutated requests, raw heads and raw HTTP/2 frame sequences through a lab edge with the firewall, cache and rules on, in front of an origin that answers oddly on purpose. The edge kept answering, with no error of its own.
- Breakwave. 21.5 million crafted packets through the XDP program, with real connections behind them so the deeper code paths run. Every verdict was valid, every packet it rewrote stayed well-formed, and it kept no state for addresses that sent nothing.
We also compared our firewall's regular expressions with PCRE2, the engine the OWASP Core Rule Set is written for: 114.8 million comparisons across all 335 patterns we run, built from each pattern's own structure. Apart from one difference we already knew of and handle, the live patterns matched. The fuzzer did find three constructs our translation read differently from PCRE2, none of them in a pattern we run. We fixed all three, so a future rule cannot trip over them.
85 ways to hide a request
Request smuggling hides one request inside another, so that two servers in a row disagree about where a request ends. The hidden one skips the checks of the first server. We sent 85 smuggling and desync attempts over HTTP/1.1 and raw HTTP/2 frames to a lab edge, behind an origin that recorded exactly what it received.
The edge itself read every one of them correctly. But one attempt showed a weakness at the origin's side. Some web servers do not read a request body on some paths. Such a server takes the body for the start of the next request, and on a connection the edge reuses, the next request belongs to another visitor. A hidden request then reaches the website past our checks and the firewall, and the other visitor gets its answer.
This is the origin's bug, not ours, but it is our job to keep it harmless. Since 10 November a request that carries a body closes its connection to the origin afterwards, and that connection is never reused. HTTP/2 connections to origins are unaffected: their framing leaves nothing to misread.
Bypassing the firewall from outside
From a machine outside our network, we sent attack payloads to a test website on the live edge, wrapped every way we could think of, with Node.js and Flask applications behind it to see how real frameworks read what got through. Ordinary attacks were refused. Four things were not:
JSON the firewall could not read
Applications read JSON more leniently than the firewall's parser did: a byte-order mark, UTF-16 or UTF-32 without a charset, NaN and Infinity, invalid UTF-8. A body the parser gave up on was neither refused nor inspected, and the application read it fine. A payload placed after many harmless keys was not reached at all. Since 10 November the firewall reads a JSON body the way applications do and checks it to the end, and a JSON or XML body it cannot read is refused.
A request checked too late
Very large bodies are checked under a time limit, so one request cannot hold the firewall. When a body ran out of time, the rules that had not run yet were skipped, and the request passed. Now a request the firewall could not finish checking is refused, and the visitor can try again.
Slow work on large bodies
Profiling those large bodies led us to a quadratic loop in libmodsecurity, the engine under our firewall: with 20,000 arguments, one step took 5.06 seconds. We fixed it (1.69 seconds now, the rest is the rules' own work) and sent the fix upstream, so every ModSecurity user gets it.
Code hidden in an image
Like most firewalls, ours did not read the contents of uploaded files, so a web shell dressed up as a picture went straight to the website. Shieldify Upload Protection now refuses images that carry server code, with a rule per image format that each website can switch off. Documents and text files are never read, because they can quote code. On 207 real images it flagged none.
We repeated the outside test after the fixes went live: every JSON trick was refused, and so were all the ordinary attacks.
Thirteen floods and one visitor
Floods are our main work, so they got the biggest lab. It runs the real edge and the real XDP program with production settings, on a test website with new-site defaults. Two generators send each flood from tens of thousands of distinct source addresses. Next to the flood, a visitor keeps loading the website the way a browser does, solving whatever the edge asks of it. Every flood ran for 20 seconds.
| Flood | Requests that reached the website | Visitor got through |
|---|---|---|
| Scripted browser, fast | 0 | Every time |
| Scripted browser, paced | 0 | Every time |
| HTTP/1.1 GET flood | 0 | Every time |
| Plain HTTP flood | 5 | Every time |
| Cache-busting queries | 0 | Every time |
| POST flood | 0 | Every time |
| HTTP/2, many streams per connection | 0 | Every time |
| HTTP/2 Rapid Reset | 0 | Every time |
| HTTP/2 CONTINUATION flood | 0 | Every time |
| Slowloris | 0 | Every time |
| Slow POST | 0 | Every time |
| Low and slow, many addresses | 0 | Every time |
The plain HTTP requests were the first of the flood, before it was recognised. The thirteenth flood is not in the table because it tests something else: it replayed one solved check's cookie from tens of thousands of addresses. From every address but the two it was issued to, the cookie was refused; those two sent 98 requests that reached the website, as their own cookie allows. A visitor got through that flood too. The visitor's slowest page took about five seconds once, for the invisible check; after that its pages loaded in milliseconds.
No flood kept the visitor out, and not one request of the three HTTP/2 floods reached the website. But the lab showed something we did not like. In the HTTP/1.1 floods, Breakwave ended tens of thousands of connections in the network driver once the edge had judged the tool. In the HTTP/2 floods the edge refused every request itself, and paid for every new connection's TLS handshake to do it. That is work an attacker should not get from us for free.
The reason was in what counted as evidence. Rapid Reset and CONTINUATION floods break HTTP/2 itself, and the edge closed those connections, but a closed connection counted for nothing. Since 10 November a connection that breaks HTTP/2 in a way no browser does is evidence against the client that opened it, and so is any refusal of a client that has not proven itself, while a website is under attack. The tool is judged sooner, and its connections end in the driver. Run again, the same three floods had 98 to 99% of their connections ended at Breakwave. Only the first wave of connections, opened before anything was decided, reached the edge.
We then tried the HTTP/2 attacks once more from outside, against the live edge. Each attack tool was stopped at the network driver after its first few connections, while the test website kept serving everyone else.
The six fixes
| Found | Fix | Live |
|---|---|---|
| Origins that skip a request body could take it for another visitor's request | A request with a body never shares its origin connection | 10 Nov |
| A request the firewall ran out of time on could pass | It is refused | 10 Nov |
| JSON the firewall's parser could not read went uninspected | Read as applications read it, checked to the end, refused when unreadable | 10 Nov |
| A quadratic loop in libmodsecurity | Linear, sent upstream | 10 Nov |
| Server code hidden in uploaded images | Shieldify Upload Protection | 10 Nov |
| HTTP/2 floods never reached the network driver | Protocol abuse and refusals count as evidence | 10 Nov |
Data and method
Every attack went against our own software: labs on our own machines, and a test website of ours on the live edge. No customer website was attacked. The figures come from our test tools and from the edge's and Breakwave's own counters. We leave out how our checks decide, their limits, and the firewall's settings: they would help the next attacker more than any reader.