DDoS Attack on Bitotsav '19 Website

This is not a technical write-up. This is a story that I want to share which might be a lesson for several web and app developers out there. Bitotsav is the annual cultural fest of BIT Mesra, and the Tech Team was responsible for developing the website and the Android app. There were 46 events in the fest, and we were responsible for sending out updates (announcements, results, etc.) for each event to the app.

On February 16, 2019, the second day of the fest, the website started experiencing problems. The website got slower and slower. The website was hosted on a Microsoft Azure Virtual Machine. It was evening, around 5:00 PM. I checked the logs for the Virtual Machine. In the past six hours, 700 GB of data was uploaded and downloaded from the server. The Virtual Machine had 4 VCPUs, and all of them were being rigorously used. Due to such huge traffic, all the APIs stopped responding. The app was no longer usable. No updates could be sent to the participants.

What was happening? In the last month, hardly 10 GB of data was transferred, and all of a sudden, in the last six hours, 700 GB of data was transferred. Clearly, the site was receiving way too many requests than it was capable of processing.

The first solution that came to my mind was to increase the capacity of the server. I doubled its size (Cloud lets you scale as per your needs). Now, we had 8 VCPUs and 16 GBs of Memory. I expected that this configuration might easily handle all the requests. I was wrong! The incoming requests were so fast that there were approximately 2000 requests pending in the pool at a time. The server crashed and restarted repeatedly. It was failing to process any request.

I came back from Infocell to meet my team – Ashank and Aakarshit. Ashank was clear that it was one of the APIs that was failing. An API is used to extract information from a database. There were several APIs running on the machine. For example:

/getEventById – This API gives out the details of an event. The requester has to send the ID of the event.
/getTeamDetails – This API gives out the details of each member of a team that is registered for an event. The requester has to send the Event ID and Team Leader ID to get the details.

These are just two of several other APIs. The two APIs above had a specialty. They do not require authentication for their use. Anyone can post a request to these APIs. The second one, i.e., /getTeamDetails, involved heavy operations on the database and required some time to complete the processing on the server. We were aware of the fact that these APIs can be exploited.

Ashank asked me to start logging the API which was being requested. Our guess was correct – it was /getTeamDetails which was being repeatedly requested. There were more than 1000 requests per second. Humans cannot generate such huge requests. They were being generated by bots. They were automatically generated by code to overwhelm the server. But who would do that? We started logging the IPs from which the requests originated. We checked the IPs and realized that all the IPs belonged to the Asia Pacific Network. The IPs were distributed from Brisbane (Australia).

What! Who on earth has so much time to find the API and repeatedly send requests? First of all, the APIs were not publicly available. The app needs to be decompiled to find the API. Second, it's Bitotsav! Who on earth would wish to target such a small application when there are millions of other important websites to target? Whoever he/she was, he/she had invested a lot of time in setting up the attack against the website. We were not convinced that the attack could originate from a foreign location. Maybe, he/she was someone from India who was using a proxy from Australia. 700 GB of data! It should have cost him/her a fortune to use such proxy servers.

We decided to call Paritosh. Paritosh is an expert in handling Nginx (the server that we used). He told us to limit the number of requests by changing the server configuration. We limited the number of acceptable requests to 10 requests/second. That worked, but not for long. We were ignoring too many important queries that must be answered, without which the app would not function properly. We decided to remove the limit.

Next, we started logging the number of unique IPs from which we were receiving the requests. There were over 300 unique IPs from which we were receiving requests. Oh My God! It was a proper DDoS attack; an attack suffered by many popular websites around the world. 300 computers around the world were sending out thousands of requests and killing the entire machine.

Suddenly, we realized that the location of the distribution of IPs was a bad measure for finding the location of the requester. We must instead find the telecommunication carrier of the IP. We checked all the IPs from which we received the requests. It turns out the major carrier was Reliance Jio Infocomm. The circle from which the requests originated was Jharkhand, India. Bang! We were 100 steps closer to the culprit now. He/She was within the state.

"Let us block those IPs," suggested Paritosh.

"Not possible, as the IPs are dynamic, and we might end up blocking some genuine devices."

The next hour went around guessing the attacker who could have initiated this attack. Was it the guy from junior year, whom we rejected in the interview for the Tech Team? Or was it Ayush Gupta (K16) from the Tech Team itself? Apart from us, he was the only one who knew the API, and he was not picking up his phone since morning. Or was it some guy from NIT Jamshedpur (NIT Jamshedpur is our rival college and they do not have a website for their cultural fest)?

The guessing game ended. We had no time to find the culprit. We have to find a remedy to counter the attack. We decided to use Azure DDoS protection. Azure DDoS protection uses machine learning to detect an attack. Unfortunately, the attack was fast enough to kill the server and slow enough to bypass Azure DDoS protection. What an intelligent attack!! It was 10:00 PM at night. All three of us were still struggling to get everything up again. Ayush Raj and Aakarshit suggested we use CloudFlare Protection. You must have seen CAPTCHA verification on some websites. It is provided by CloudFlare, and it provides good protection against DDoS. It took 2 hours to update the site's DNS. We expected a slowdown. But the attack multiplied. We were receiving more and more requests.

Ah! Nothing was working. What was wrong? It is an apocalypse. It is a nightmare for every programmer to see his/her program failing in production. "Calm down!". We had to do something. We couldn't have let things slip from our hands.

We decided to log the results of each request. As I said, the API required Event ID and Team Leader ID to be sent by the requester for processing. We realized that each request was success, i.e., the attacker was sending the correct Event ID and Team Leader ID, and such a team for the given event was always found in the database.

"Since the number of teams is limited, let us find the number of unique combinations of teams being requested!" I suggested.

There were 71 unique requests for the combination of Event ID and Team Leader ID. Why 71? Why would the attacker choose such a number?

Next, we logged the number of unique Event IDs. There were 24 unique Event IDs!

26 events were marked complete on Day 1 and Day 2 of the fest. The results of 2 events were not yet declared. There were 3 winning teams for each event, which makes a total of 24 x 3 = 72 winning teams.

"No, the event ‘Mr. and Ms. Bitotsav’ has no third position," said someone. That makes a total of 71 winning teams. The attacker is requesting the details of all 71 winning teams.

"What did you say?" asked Ashank.

"What?"

"There is no team who came 3rd in ‘Mr. and Ms. Bitotsav’?"

"Yes! It's ‘Mr. and Ms. Bitotsav.’ Only one guy and one girl could have won it."

"The app requests the details of all the winning teams every time it starts up. Whenever some team is not found, it fails and sends all the requests again till eternity!" said Ashank.

We felt the earth slip. There were over one thousand people who were using the Android app. All the apps were requesting the details of 72 teams. One of the team details was missing. The app crashed and requested again. The details were missing again. The app crashed and requested again. We were the creators of our own attack. It is us who programmed the attack against our own server. Each installation of the app was acting as a source of an attack and flooded the server with requests.

"You never told me that there can be no third position!" argued Ashank.

"Why are you requesting again and again if the API told you that the team was not found?" I argued. And we blamed each other for the next hour.

We added a dummy team for the third position of ‘Mr. and Ms. Bitotsav.’ I started a cluster of servers balanced by a load balancer. The app was updated to not request the details of a team that was not found in the database. Finally, everything was under control and working again. It took us 10 long hours to find out the cause of this chaos. The app functioned smoothly on Day 3. We were not satisfied with what we had done. The 700 GB of data transferred to the cloud cost us a lot.

Key takeaways: