NEW RESEARCH: WHY SHIELDIFY MOVED TO AWS.Read the report Sign in

We Tried to Break Shieldify.

On 9 and 10 November we spent two days attacking Shieldify the way an attacker would: our edge, its firewall and Breakwave, our program in the network driver. Fuzzers on everything that reads client bytes, request smuggling, firewall bypasses from a machine outside our network, and thirteen Layer 7 floods. Most of it held. Six things did not, and all six are fixed and live.

6weaknesses found and fixed. All six fixes are live.
352 millionmalformed inputs fed to the parsers that read what clients send. No crash.
13Layer 7 floods from tens of thousands of source addresses each. A real visitor got through every one.
0flood requests reached the website in eleven of the thirteen floods, every HTTP/2 flood among them.
Why attack ourselves

The attacker does not wait for an audit

Shieldify stands between a website and everyone who wants to reach it. If our edge can be crashed, tricked or worn down, so can every website behind it. Attackers test that every day, for free. We would rather find the weak spots first.

So we set two days aside and attacked our own stack: the edge we wrote in Rust, the web application firewall it runs, and Breakwave, our XDP program in the network driver. The rules were simple. Real attacks only, at real speed. Labs that run the same software and settings as production. And a machine outside our network, attacking a test website of ours on the live edge, the way a stranger would.

Fuzzing

Hundreds of millions of malformed inputs

Everything the internet sends us is read by code we wrote. A parser that panics on odd input takes the edge down with it; one that takes too long on odd input is a cheap way to slow it down. We fuzzed it in three layers:

  • The parsers. TLS ClientHellos, HTTP/1.1 heads, HTTP/2 frames, host names and paths, forms, Range headers, cache dates and TCP options: 352 million mutated inputs in three minutes. No panic and no slow call. To check the fuzzer itself, we planted a bug; it was found within seconds.
  • The whole edge. About 133,000 mutated requests, raw heads and raw HTTP/2 frame sequences through a lab edge with the firewall, cache and rules on, in front of an origin that answers oddly on purpose. The edge kept answering, with no error of its own.
  • Breakwave. 21.5 million crafted packets through the XDP program, with real connections behind them so the deeper code paths run. Every verdict was valid, every packet it rewrote stayed well-formed, and it kept no state for addresses that sent nothing.

We also compared our firewall's regular expressions with PCRE2, the engine the OWASP Core Rule Set is written for: 114.8 million comparisons across all 335 patterns we run, built from each pattern's own structure. Apart from one difference we already knew of and handle, the live patterns matched. The fuzzer did find three constructs our translation read differently from PCRE2, none of them in a pattern we run. We fixed all three, so a future rule cannot trip over them.

Request smuggling

85 ways to hide a request

Request smuggling hides one request inside another, so that two servers in a row disagree about where a request ends. The hidden one skips the checks of the first server. We sent 85 smuggling and desync attempts over HTTP/1.1 and raw HTTP/2 frames to a lab edge, behind an origin that recorded exactly what it received.

The edge itself read every one of them correctly. But one attempt showed a weakness at the origin's side. Some web servers do not read a request body on some paths. Such a server takes the body for the start of the next request, and on a connection the edge reuses, the next request belongs to another visitor. A hidden request then reaches the website past our checks and the firewall, and the other visitor gets its answer.

This is the origin's bug, not ours, but it is our job to keep it harmless. Since 10 November a request that carries a body closes its connection to the origin afterwards, and that connection is never reused. HTTP/2 connections to origins are unaffected: their framing leaves nothing to misread.

Bypassing the firewall

Bypassing the firewall from outside

From a machine outside our network, we sent attack payloads to a test website on the live edge, wrapped every way we could think of, with Node.js and Flask applications behind it to see how real frameworks read what got through. Ordinary attacks were refused. Four things were not:

JSON the firewall could not read

Applications read JSON more leniently than the firewall's parser did: a byte-order mark, UTF-16 or UTF-32 without a charset, NaN and Infinity, invalid UTF-8. A body the parser gave up on was neither refused nor inspected, and the application read it fine. A payload placed after many harmless keys was not reached at all. Since 10 November the firewall reads a JSON body the way applications do and checks it to the end, and a JSON or XML body it cannot read is refused.

A request checked too late

Very large bodies are checked under a time limit, so one request cannot hold the firewall. When a body ran out of time, the rules that had not run yet were skipped, and the request passed. Now a request the firewall could not finish checking is refused, and the visitor can try again.

Slow work on large bodies

Profiling those large bodies led us to a quadratic loop in libmodsecurity, the engine under our firewall: with 20,000 arguments, one step took 5.06 seconds. We fixed it (1.69 seconds now, the rest is the rules' own work) and sent the fix upstream, so every ModSecurity user gets it.

Code hidden in an image

Like most firewalls, ours did not read the contents of uploaded files, so a web shell dressed up as a picture went straight to the website. Shieldify Upload Protection now refuses images that carry server code, with a rule per image format that each website can switch off. Documents and text files are never read, because they can quote code. On 207 real images it flagged none.

We repeated the outside test after the fixes went live: every JSON trick was refused, and so were all the ordinary attacks.

Thirteen floods

Thirteen floods and one visitor

Floods are our main work, so they got the biggest lab. It runs the real edge and the real XDP program with production settings, on a test website with new-site defaults. Two generators send each flood from tens of thousands of distinct source addresses. Next to the flood, a visitor keeps loading the website the way a browser does, solving whatever the edge asks of it. Every flood ran for 20 seconds.

FloodRequests that reached the websiteVisitor got through
Scripted browser, fast0Every time
Scripted browser, paced0Every time
HTTP/1.1 GET flood0Every time
Plain HTTP flood5Every time
Cache-busting queries0Every time
POST flood0Every time
HTTP/2, many streams per connection0Every time
HTTP/2 Rapid Reset0Every time
HTTP/2 CONTINUATION flood0Every time
Slowloris0Every time
Slow POST0Every time
Low and slow, many addresses0Every time

The plain HTTP requests were the first of the flood, before it was recognised. The thirteenth flood is not in the table because it tests something else: it replayed one solved check's cookie from tens of thousands of addresses. From every address but the two it was issued to, the cookie was refused; those two sent 98 requests that reached the website, as their own cookie allows. A visitor got through that flood too. The visitor's slowest page took about five seconds once, for the invisible check; after that its pages loaded in milliseconds.

No flood kept the visitor out, and not one request of the three HTTP/2 floods reached the website. But the lab showed something we did not like. In the HTTP/1.1 floods, Breakwave ended tens of thousands of connections in the network driver once the edge had judged the tool. In the HTTP/2 floods the edge refused every request itself, and paid for every new connection's TLS handshake to do it. That is work an attacker should not get from us for free.

The reason was in what counted as evidence. Rapid Reset and CONTINUATION floods break HTTP/2 itself, and the edge closed those connections, but a closed connection counted for nothing. Since 10 November a connection that breaks HTTP/2 in a way no browser does is evidence against the client that opened it, and so is any refusal of a client that has not proven itself, while a website is under attack. The tool is judged sooner, and its connections end in the driver. Run again, the same three floods had 98 to 99% of their connections ended at Breakwave. Only the first wave of connections, opened before anything was decided, reached the edge.

We then tried the HTTP/2 attacks once more from outside, against the live edge. Each attack tool was stopped at the network driver after its first few connections, while the test website kept serving everyone else.

What we fixed

The six fixes

FoundFixLive
Origins that skip a request body could take it for another visitor's requestA request with a body never shares its origin connection10 Nov
A request the firewall ran out of time on could passIt is refused10 Nov
JSON the firewall's parser could not read went uninspectedRead as applications read it, checked to the end, refused when unreadable10 Nov
A quadratic loop in libmodsecurityLinear, sent upstream10 Nov
Server code hidden in uploaded imagesShieldify Upload Protection10 Nov
HTTP/2 floods never reached the network driverProtocol abuse and refusals count as evidence10 Nov
Data and method

Data and method

Every attack went against our own software: labs on our own machines, and a test website of ours on the live edge. No customer website was attacked. The figures come from our test tools and from the edge's and Breakwave's own counters. We leave out how our checks decide, their limits, and the firewall's settings: they would help the next attacker more than any reader.

More research

Testedthe Hard Way.

Point one DNS record to Shieldify and these protections cover your website too, also in the middle of an attack.