BGP re-anoouncements and DDOS
By tracking the BGP announcements of large peering providers, we were able to identify what reassembled as a highly congested link in a backbone network, something that normally triggers BGP route flaps and session drops.
Monitoring the sudden increase of BGP route announcements of a given ASN, it has proben to be a good sign of the presence of Denial of Serice Attacks as congested links provoke the drop of BGP sessions.
Same, same ... but not the same
During the morning of the 26th February 2018, we received several alarms from routing monitoring servers that indicated problems of BGP stability in a Tier-1 Swedish ASN.
The time frame was 10:40 AM to 11:40 AM, 60 minutes, a typical indication of attacks powered by booters or stress testing services.
Volumetric attacks of this nature are normally conducted using amplification vectors. UDP-based services are abused to perform these attacks and when a new vulnerable service is widely exploited, big bandwidth is available to the attackers. With time, the number of vulnerable services decrease as they are patched by their owners.
The most common UDP-reflection services include:
- Simple Service Discovery Protocol (SSDP)
- Network Time Protocol (NTP)
- Connection-less Lightweight Directory Access Protocol (CLDAP)
- Character Generator Protocol (CharGEN)
- Simple Network Management Protocol version 2 (SNMPv2)
- Domain Name Service (DNS)
Other less common vectors include: RPC, BitTorrent, QOTD, NetBIOS, RIP etc.
Darknet recordings and a new UDP attack vector
Two weeks before this large attack, we identified the increase of heavy scanning from AS43350 and AS44050 looking for memcache servers UDP/TCP 11211. This was our main suspect as it was later confirm by numerous articles online.
An example of how to obtain TCP scanning for Memcache during the first three weeks of January 2017
# ./dnc_ASN.py -f 2018-01-01 -t 2018-01-20 -n 100 -R memcacheT
2018-01-01, 2018-01-20, memcacheT, 'AS6939 Hurricane Electric, Inc.', United States, 6877
2018-01-01, 2018-01-20, memcacheT, 'AS60781 LeaseWeb Netherlands B.V.', United States, 763
2018-01-01, 2018-01-20, memcacheT, 'AS36351 SoftLayer Technologies Inc.', United States, 499
2018-01-01, 2018-01-20, memcacheT, 'AS29073 Quasi Networks LTD.', Seychelles, 277
2018-01-01, 2018-01-20, memcacheT, 'AS32748 Steadfast', United States, 256
2018-01-01, 2018-01-20, memcacheT, 'AS4134 Chinanet', China, 194
2018-01-01, 2018-01-20, memcacheT, 'AS10439 CariNet, Inc.', United States, 135
2018-01-01, 2018-01-20, memcacheT, 'AS4837 CNCGROUP China169 Backbone', China, 73
2018-01-01, 2018-01-20, memcacheT, 'AS23338 DCS Pacific Star, LLC', United States, 66
2018-01-01, 2018-01-20, memcacheT, 'AS7552 Viettel Corporation', Vietnam, 37
2018-01-01, 2018-01-20, memcacheT, 'AS9808 Guangdong Mobile Communication Co.Ltd.', China, 30
2018-01-01, 2018-01-20, memcacheT, 'AS32475 SingleHop, Inc.', United States, 29
2018-01-01, 2018-01-20, memcacheT, 'AS12695 JSC Digital Network', United States, 28
2018-01-01, 2018-01-20, memcacheT, 'AS50613 Thor Data Center ehf', Iceland, 23
2018-01-01, 2018-01-20, memcacheT, 'AS16276 OVH SAS', Canada, 1
A bit of maths
We reached out to the victim and we could confirm that they received a large UDP attack, that congested a couple of their backbone links.
The numbers: 273 Gbps, 23 million packets per second.
In order to calculate the real bandwidth used by the attacker we did a bit of maths. For every spoofed packet sent to a memcache server the attacker was able to reflect five large packets back aimed to the victim.
The attacker issued the "stats" command, giving a payload response of 7000 bytes for the 15 bytes of the request.
0x0020: 0001 0000 7374 6174 730d 0a00 0000 ....stats.....
In order to really calculate the "amplification" achieved it is important to understand what is the minimum packet size that an attacker can effectively send on the wire. Every packet transmitted on a Ethernet link includes 7 bytes of preamble and 1 byte of SFD (start of frame delimiter) plus 12 byte of IFG (inter-frame gap). So on the wire, 20 extra bytes are always needed to send the smaller Ethernet packet of 64 bytes.
The UDP requests contains 15 bytes \x00\x01\x00\x00\x00\x01\x00\x00stats\r\n
that are 84 bytes on the wire. It is common to read articles making these maths wrong and calculating amplification as UDP payload reflected vs UDP payload sent.
void setup_udp_header(struct udphdr *udph)
{
udph->source = htons(5678);
udph->dest = htons(11211);
udph->check = 0;
memcpy((void *)udph + sizeof(struct udphdr), "\x00\x01\x00\x00\x00\x01\x00\x00stats\r\n", 15);
udph->len=htons(sizeof(struct udphdr) + 15);
}
So how many packets needs an atacker that sends a memcache "stats" request to generate 23 Mpps. It needs to generate 1/5th of the total number of packets received, in our case 4.6 Mpps. How much bandwith is that? 4.6 Mpps * 84 bytes/packet * 8 bits/packet ~= 3 Gbps
This attack achieved a x90 amplification on the wire. Three times the ratio that we see with other attack vectors as DNS Amp.
How 3 Gbps of memcache requests can be generated?
Very few research articles focus on the real capabilities of the attackers and the key enablers for their sucessfull campaigns. Traditional SYN and UDP flooders written in C and using the standard network drivers can achieve a few hundred thousand packet per second per server. Scripts written in other programming languages as python can deliver 1/10 of that if lucky.
The most known stress testing services use these type of traffic generators, so how is possible to push 5 Mpps of spoof traffic into the network?
Attackers able to generate these type of traffic are not using standard drivers and rely on fast packet processing frameworks to achieve faster speeds. The attackers able to generate these type of volumes of traffic run dedicated servers and networking ports to achieve such a performance.
Is this so dangerous?
Yes, it will get worse as most of the attackers are still building up good techniques to achieve bigger amplifications.
When a new attack vector is exploited in wild, dozen of actors start to abuse it. The attackers are often an interesting mix of script kiddies running code available in pastebin and online forums and those that have the real know-how to be faster.
In order to understand how large amplifications are achieved and who is or are behind them we combined our darknet data with a few honeypots that recorded the memcache commands and the TTL values of the spoof traffic sent to them.
What we saw is that the majority of the attackers are using the known "stats" command while others are exploring how to achieve bigger amplification ratios. For example, we have seen a few actors using the command "get Vco0W" instead of stats or stats items. Tracking the stored keys in the memcache server has helped us to track a few attackers already.
0x0020: 0001 0000 7374 6174 7320 6974 656d 730d ....stats.items.
0x0020: 0001 0000 7374 6174 730d 0a73 7461 7473 ....stats..stats
0x0020: 0001 0000 6765 7420 5663 6f30 570d 0a ....get.Vco0W..
This graph is an example of the packets per second received in the memcache honeypot.
In order to get a full picture of what is going on we have modified a memcache server to log more details about the connections. It was a surprise to find out that memcache does not have means to log the IPs of the requests so we patched the source to log them.
The honeypots have also recorded attackers that opt for randomizing the source port vs attacks with fix ports. These clues are a good indication that several implementation of the UDP spoofing are on currently being used.
Preloading memcache to increase amplification
Similar to other amplifications as "NTP Monlist" attack, advance attackers are pre-loading the open memcache servers with "keys" that can be later on requested. In this way, the attackers ensure that the service supports the memcache API and that the memcache server is delivering a large payload.
Attackers are currently pre-loading vulnerable memcaches with large keys. When memcached stores a value, it looks up the "slab" associated with that value. A slab holds values within a particular size range and are composed of 1MB pages. Checking the memcache source, it seems that this value is limited to 1024x1024 bytes or 1,048,576 bytes.
slab class 42: chunk size 1048576 perslab 1
The smart attacker will preload the memcache servers with a 1 MB payload and every spoof request translates in 750 packets back. Running a packet capture during a DDOS provides a clear signature :)
Where is the traffic generated?
The scanning
In the top 10 of providers scanning for memcache servers we find AS43350 (NForce Entertainment B.V.), AS44050 (Petersburg Internet Network), AS4134 (Chinanet), AS134764/38283 (Chinanet), AS32748 (Steadfast) and AS14061 (Digital Ocean)
One interesting prefix involved in the scanning is 77.72.82.0/24, first seen announced by NForce last year and already publicly involved in other attacks.
The Spoofing
Where is the spoof traffic generated? There is a common believe that spoof traffic is difficult to track, that the traffic is generated behind one or multiple transit providers and difficult to track back as will require a coordinated action.
What we see in our memcache honeypots is that several attackers are generating the traffic directly from peers present in an Internet Exchange.