Online Wars: Surviving DDoS Attacks

5 Jan, 2009

Recently more than ever, hackers with different agendas are launching different attacks on many famous websites with an attempt to take them down, control the servers to attack other servers, corrupt or destroy the database or even steal valuable data. Those hackers launch their attacks using immature to sophisticated mechanisms. Their agenda might be political one like an Intelligent agency attacking its rivals’ websites during war, or religious groups attacking their opponents.

In the Arab world, these attacks are becoming more prominent. last September around 900 Sunni religious websites, Arabic news agencies and forum have been under intense attacks which disabled them. Later on, Alarabiya domain name was hacked.

Today as “Israel” continues its war against Palestinians in Gaza, Hamas affiliated news website has been under intense attack which led to a temporally shutdown of its Arabic website, and as of now Hamas military wing: Ez ElDeen Elqassam’s website is down.  There is even an Israeli Internet initiative that is recruiting surfers for cyberwarfare to attack Hamas and other Muslim websites– Not Affiliated with Hamas-. On the Israeli online casualties hackers have launched a massive cyberattack, defacing more than 300 Israeli websites.

In light of the ever increasing online threats I asked Ahmed Mekkawy the team leader of System Admins in eSpace to write a guest post about his experience in surviving a DDoS attack on a famous Arabic forum. Mekkaway is also one of the admins team of Egypt Gnu/Linux Users Group. He is well known of being fond of Free Software.

A famous arabic forum has been under really strong attacks that led the forum to go down more than once. In the end it was down for over a week. At that time the site owners contacted eSpace to try to secure the site. This task fell back on me as I’m the SysAdmins’ team leader.

After some investigations, I found out that there were multiple attacks, the main one was a Distributed Denial of Service (DDoS) coming from around 10k different IPs. The first thing that came into my mind is
that it’s a SYN flood but I found out that they are real IPs that are trying to open the home page (i.e http://domainname.com/). So the first defense to do is to split the site into 2 servers. One of them serves only the home page, and all the links on it redirects to another domain name that are served from the other server. Plus minimizing the HTML size of the home page so that the server can handle the load (provided that they were not active clients, meaning they don’t get the images, css, javascripts, .. etc). The attackers followed us to the new domain and they were still opening the home page (which didn’t contain any useful data anyway), and they also increased the number of the attack.

The attack wasn’t continuous all the time, but it was more like an attack waves, and between them there were a relatively a small number of attackers who never stop around the clock.

The forum’s web application was an old one, it acts as an application server and a web server at the same time. It was single threaded so I couldn’t get use of the multi-core server processing power, my bottleneck at that time was the processor usage. So I decided to go for performance enhancement. So I added varnish as a reverse-proxy and nginx to act as a frontend, so the HTTP request is managed by nginx first, then passed to varnish to server it if it’s cached, if not in the cache, then it’s brought from our backend.

Lots of tweaks were made to both varnish and nginx. For those who don’t know those two software, varnish is the best reverse proxy I found that acts at fabulous speed, it even caches in memory to avoid HardDisk latency. But it’s not the thing that works out of the box, it takes time to configure, when you are using cookies, and you’ll have to learn VCL (varnish configuration language) and nginx In my opinion is the black horse in web servers market, that is so light, with high performance and stability and is an easily configured web server. By using varnish & nginx I could make use of the extra processor cores that I had and I already had lots of memory available for caching using Varnish.

After making all that, the load average dropped from around 5 to less than 0.5 which is great. Moreover, I have managed to find the attack pattern and detect it from nginx then closed the connection after serving 0 bytes. The attackers then became very aggressive, the attack waves reached 50k IPs, so I had to increase the nginx workers up to 10 and increase the ulimit alot (made it 512k) and made lots of kernel tweaks in order to be able to handle the TCP load (like increasing TCP buffers, tcp fin_timout, … etc,) I even had to drop the IP_CONNTRACK kernel module and redesign the custom firewall that I made to use the TCP flags. The bottleneck at that phase was handling the TCP connections overhead on the kernel level.

After I made all the above, the site was still performing while the attack waves are active. Adding another stand-by server and automatic failover made the end user feels nothing about the attacks that were happening.

In few weeks an email from the attacker asking for $5,000 per month or else he’ll continue attacking. I Just didn’t answer him and he tried for a couple of more attack waves then everything became very stable, and the attacks were stopped, because they could not affect the website anymore :) .

The attacks started in 2008, and it lasted for about 8 months.

Share and Enjoy

  • Facebook
  • Twitter
  • Delicious
  • LinkedIn
  • StumbleUpon
  • Add to favorites
  • Email
  • RSS

  1. Ayman Khateeb  |  January 5th, 2009 at 10:43 PM #

    This is a great article and story to share.

    the pldf.net site you mentioned is performing greatly after several attacks, i’m not sure what did they do from a technical point of view, but functionally they reduced the web site size and disabled most of the sections and only kept the most important ones, and also prevented unneeded graphics and scripts from appearing…

    what about interviewing them,,, after the victory isa…

  2. Amon  |  January 9th, 2009 at 6:25 PM #

    Why do you have the name Israel in double quotes? isn’t it supposed to be an objective article?

  3. Gaith Saqer  |  January 10th, 2009 at 12:22 AM #

    AC is not a political blog it focus on Arab Tech startups. We do out best to stay out of politics. However many times politics gets mixed, we just note whenever a territory is disputed.

  4. ASD  |  January 10th, 2009 at 3:23 AM #

    Stop saying Israel Say IsraHell , in you writing , chat , talk , conversations and artwork .

    we lunched this campaign , to show the world how illegal israhell is .

    Please write an Article , POST , make an Artwork about this new name of IsraHell .

    Please forward the message and say IsraHell .

  5. "Name"  |  January 10th, 2009 at 9:17 PM #

    Gaith:

    That’s bull and you know it. For a culture and a people who once lead the world in innovation, invention, enlightenment, education, modern thought, and so many other things, its descendants sure are a disappointment.

    Maybe it’s time to calm the rhetoric and get back to the things that made the Arab world so great.

  6. Khalid  |  January 12th, 2009 at 6:11 PM #

    Joly good work my friend :)

  7. Hasbullah Pit  |  January 13th, 2009 at 10:34 PM #

    How do you close connection after serving 0 byte?
    (nginx)

  8. Ahmed Mekkawy  |  January 14th, 2009 at 11:55 AM #

    @Amon @ASD I thought I would have more technical conversation here.
    @Khalid thank :)
    @Hasbullah Pit by returning the http code 444, nginx understands it as closing the connection directly.. see http://wiki.codemongers.com/NginxHttpRewriteModule?highlight=(nonstandard%20code%20444)

  9. Was the Twitter DDoS attack Cyber Warfare?  |  August 7th, 2009 at 3:53 PM #

    [...] our security systems up-to-date. It’s basic but very effective. If you do find yourself the target of a DDoS attack, this article over at ArabCrunch may be of use. I have a feeling that yesterday’s events will trigger the emergence of new technology to [...]

  10. Was Yesterday’s Twitter DDoS Attack Cyber Warfare | Search Engine Optimization & Internet Marketing (SEO & SEM) Blog  |  August 7th, 2009 at 6:54 PM #

    [...] our security systems up-to-date. It’s basic but very effective. If you do find yourself the target of a DDoS attack, this article over at ArabCrunch may be of use. I have a feeling that yesterday’s events will trigger the emergence of new technology to stop [...]

  11. Was Yesterday’s Twitter DDoS Attack Cyber Warfare - Google Live Search  |  August 7th, 2009 at 7:59 PM #

    [...] our security systems up-to-date. It’s basic but very effective. If you do find yourself the target of a DDoS attack, this article over at ArabCrunch may be of use. I have a feeling that yesterday’s events will trigger the emergence of new technology to [...]

  12. Toppositions.org - Was The Twitter Ddos Attack Cyber warfare? « TopPositions.org  |  August 9th, 2009 at 1:52 AM #

    [...] our security systems up-to-date. It’s basic but very effective. If you do find yourself the target of a DDoS attack, this article over at ArabCrunch may be of use. I have a feeling that yesterday’s events will trigger the emergence of new technology to [...]

  13. Top Positions - Was Yesterday’s Twitter Ddos Attack Cyber warfare « TopPositions.org  |  August 9th, 2009 at 2:19 AM #

    [...] our security systems up-to-date. It’s basic but very effective. If you do find yourself the target of a DDoS attack, this article over at ArabCrunch may be of use. I have a feeling that yesterday’s events will trigger the emergence of new technology to [...]

  14. Breaking: Arabic Sites Untiny and Filaty Are Under DoS Attack  |  September 17th, 2009 at 12:45 PM #

    [...] Mekkawy the security expert from Egypt wrote here a nice post on ArabCrunch explaining how he managed to survive a DoSS [...]

  15. “End the Illegal Siege of Gaza” Succeeds: #Gaza A Trending Topic On Twitter  |  December 27th, 2009 at 9:48 PM #

    [...] then, we mentioned the related web events, Online attacks from Israeli hackers, and  the counter attacks from Pro Palestine hackers . We also mentioned Aljazeera’s Crowd sourced Mapping Mashup For The War On Gaza , which put [...]

  16. نجاح مؤقت لحملة “أوقفوا الحصار الجائر على #غزة” على تويتر -#Gaza في صدارة المواضيع على تويتر لعدة ساعات  |  December 28th, 2009 at 3:05 PM #

    [...] ذكرنا الأحداث المتعلقة لهذه الحملة على الشبكة مثل هجمات (المتسللون) هاكرز إسرائليون على المواقع الفلسطين… و الإسلامية و الهجمات المضادة من قبل متسللون (هاكرز) [...]

  17. apaa  |  January 31st, 2010 at 9:01 AM #

    Hello,

    Im going to setup a server using nginx.

    But how you manage the PHP?

    (My php-cgi use too much RAM in my VPS. Im using Lighttpd and eAccelerator, and I dont know the effective way to reduce the RAM usage)

  18. security war  |  February 10th, 2010 at 6:37 PM #

    good topic keep on

    can we full protect from ddos attack

  19. Waleed Alzuhair  |  April 20th, 2010 at 1:59 AM #

    Well done.. A very informative article, thanks for sharing..

Leave a Feedback

  • Become Our Fan On Facebook

    ArabCrunch on Facebook

  • Popular Posts



  • Recent Comments
    • Gaith Saqer: Thank you Ahmed for your analysis I am n...
    • Ahmed: @Gaith Intel acquired SysDSoft early ...
    • ali: Mr Mohammad Abdl Fatah: you are accusati...
    • ali: Iman Gaith is right and you are arguing ...
    • Mohamed Abdallah: Egypt is facing tough time no question b...