Setting up new Network, Security concern on DDOS
Posted by EricTham, 04-15-2009, 11:15 PM I am intending to setup a network as the following: NOC1: Cisco/Dlink Managed Router Firewall with DOSS protection Server1 Server2 Server3 Backup Data Bank Drive NOC2: Cisco/Dlink Managed Router Firewall with DOSS protection Server4 Server5 Server6 NOC2 are backup servers. I will need to have whatever in NOC1 to be written to NOC2, i think is call IP mirroring or RAID , not too sure (please advice too). My domain.com is going to have nameserver1/2 to zoneedit. Zoneedit hosts reliable DNS servers. It also support something call failover. So if NOC1 is down, i will switch to IP to NOC2 IPs. Now, if i face a DDOS attack, i am suppose to switch to a DDOS attack managment company (with big bandwidth and blocking), is it as simple as switching the domain.com to the DDOS attack management company using zoneedit and the company will than link back to my noc1/2 ? How does it works? Is the way i setup the network correct? Please advice.
Posted by CiscoMike, 04-16-2009, 03:08 AM It depends on how the scrubbing center is setup for the DDoS mitigation. Typically, most providers use a technique called diversion where the destination IPs (because they are easier to inject than the source or attackers) are advertised back into BGP (or the IGP but generally it's BGP) as going through the scrubbing devices. There, logic, filters and rules are applied to hopefully drop all the noise and let the legitimate traffic through to your servers. As a result, the impact should be transparent to you and/or your users. I would think that you wouldn't be using DNS edits to push your traffic to their pseudo-proxy solution as that's not only very unreliable but takes time, even with a low TTL it'll take some time to propagate out. One misnomer and this is the hardware vendors fault - firewalls don't do squat for DDoS regardless of who sells you the product. Firewalls are interested in state. As a result, they have capacity limits as follows: 1) bandwidth 2) new connections per second (CPS) 3) total concurrent connections You will almost always run into the ceiling of #2 and #3 before you hit #1 unless you undersized your firewall. Most firewalls, even the mighty Cisco ASA 5580, Juniper SRX and Crossbeam X-series will fall flat on their face if the CPS traffic gets to sufficient levels (we're talking $100k - $2M solutions here too). They'll all put the DDoS mitigation bullet on their data sheets because of their SYN Cookie and/or TCP intercept features but any DDoS that can spit out 1M new CPS or scale to 3-5M concurrent connections, which is easy and trivial to do with a 100k node botnet, will easily tip over a firewall, thus created non-targeted servers as unintentional DDoS targets (since the firewall is now in the crapper). That doesn't mean don't use a firewall but don't think your firewall is going to save you from an intentional/directed DDoS attack. They might help with some forms of low-bandwidth attacks (based on TCP or session timeouts) but that's just not the role of the firewall and any vendor that promises that to you is not telling you the whole story.
Posted by EricTham, 04-16-2009, 09:39 PM So, which anti-DDOS provider do you recommend for me to subscribe? The provider must have big bandwidth to support if we are attack right? Also to have skillful analyses to identify the pattern and block required ips right? I do understand the first line of firewall hardware is just to get the alert and for small attacks, the big attach i will still need to set the DNS to a anti-DDOS provider right? What hardware is recommended if my bandwidth from my provider is less than 10mbits dedicated? (Should be able to bustable to 100mbits MAX, but if i really do that, my colo host would have blocked me)
Posted by bataviahost, 04-17-2009, 12:13 AM I heard that Prolexic is a reliable DDOS protection provider. EasyDNS is also behind Prolexic network. Jahja.
Posted by CiscoMike, 04-17-2009, 12:23 AM Again, you don't want to rely on DNS if you don't have to. Yes it's cheap but you are relying on low TTLs and limited caching for it to be both effective and maintain your availability. Let's say your zone TTL (not per-record but that also applies) is set to 15 minutes and nobody upstream is using ixfr for quicker updates...that means at a minimum, your secondaries won't grab the updated zone for as long as 15 minutes depending on where in the cycle you are. Next you have to worry about other provider caches (say like Comcast, Time Warner, Qwest, etc) both respecting your TTL value (ha!!!) and actually flushing on time. If that doesn't happen, then legitimate users are still resolving to your old IP and thus cannot access your servers through the proxy solution. Your *best* bet is to go with a provider that offers an out-of-band scrubbing solution. It's obviously not your only choice but it's going to have the fastest response and lowest impact. It's also probably not going to be cheap. The other thing is why do you think you're a target of a DDoS? I've been hosting for 10+ years, some fairly controversial/socially unacceptable stuff at times and never once been hit with anything else than zombie or scanner traffic. Don't give people a reason to attack. Harden your host to make sure you're not an easier target. Yes, you could go with a proxy-DNS solution. It will work but understand that there could be significant lag between the time you identify the attack and the time that all of your traffic is going through the proxied solution which will include (more than likely) an outage for your users until their own individual cache servers catch up. WHT uses a system like that called ProxyShield as offered through Giganet (http://www.gigenet.com/hosting-solut...rotection.html) but it sounds like the "best" option in that case is to have Giganet host your DNS for you which could have faster response times and easier off-loading of the malicious traffic. Still not better than in-band flow analysis but it's not bad either.
Posted by EricTham, 04-17-2009, 10:58 PM Hmm , so it should be 1:Anti-DDOSS Company 2:Firewall 3erver 4:BackupBank ? If i do not use DNS services, how am i able to do a roundrobin if NOC1 is down and switch to NOC2 ?
Posted by CiscoMike, 04-18-2009, 01:54 AM I think you're confusing DNS for name resolution versus DNS for DDoS mitigation. You'll still use DNS to resolve records, just not as your first choice for DDoS. And for GSLB (NOC-to-NOC failover) you really should be looking at a something bigger than simple A records in BIND. But like I said, you can use DNS record mangling to deal with DDoS, just don't expect to be fast with regards to your users and possibly even the attackers, even with a very low TTL. It will work, I'm sure someone from WHT can attest to the ProxyShield solution but again, flow based analysis is going to be your best bet and the most effective solution. And in 99% of those cases, the analysis system will be owned by your provider, not a 3rd party company.
Posted by EricTham, 04-18-2009, 10:27 PM Hmmm , sorry to be a nerd, can i setup in such way: 1: DNS Party (Zoneedit or easyDNS) 2: Firewall 3: Server 4: BackupBank I understand that the provider is the main source if the ddos attack can be handled. So what i do is that to have NOC1, NOC2, NOC3, 3 of them will be using IP mirror (duplicating all files the same in each network), by default i set my DNS to go to NOC1, if there is an attack, i can use zoneedit/easydns to switch to NOC2 and NOC3 ... I understand that it will be rather slow if i keep using the DNS provider to change the A records, (actually how slow will users TTL usually be refreshed?) while i am on a attack, but is there another way? Why i split to 3 data centers is because for normal small attacks, the firewall and NOC1/2 can handle it (Local bandwidth), however if i am seriously attack, i can than switch to NOC3 which noc3 has a bigger bandwidth 100mbits or 1gig port and with a good firewall (hardware) to do the analyze and filtering. Am i correct?
Posted by nibb, 04-19-2009, 06:56 PM There is no point to change providers with a DNS solution since the attack would also change. They are probably doing DNS resolution to the domain for the attack so they the attack flow would also start resolving to the new provider. Using several providers is not better then just one big one. For DDOS attacks its all about how has the biggest pipe line. If the attack is really big they you are really screwed. Solutions like proxy shield are out of the scope of budget for most people, you could buy several servers, load balancers with failover and a big fat pipe line for the same money. I think the best solution is something that can be not be made by a computer. Captcha? So the DDOS attack which uses bot computers could not resolve the codes as well. Dont ask me how its done since i dont know it myself but there should be a way a trick automatic bots at a network level.
Posted by server4sale, 04-20-2009, 12:09 PM Here is a very simple approach. Which we do for many of our client. Have 2 DC. Data being mirrores / synced on them. Get a DDOS Protection from few Major player... I recommend GIGE or Staminus ,or just google it... Get a server there a proxy , Point it to your actual server. In case if there is an outage cause of route etc you will switch to dc2 from your proxy.
Posted by CiscoMike, 04-20-2009, 01:25 PM The last 2 posts make me cringe at a technical level (not to mention the attack on the English language...) A DDoS is not always a bandwidth starvation attack. In fact, in most cases, a DDoS attack is about a resource starvation other than bandwidth, i.e. a web or database daemon. Let's also make one other thing clear - a DDoS is always a DoS but a DoS isn't always a DDoS. The first D in DDoS means it's distributed, meaning there are multiple sources in the attack, generally in the tens to hundreds to thousands. The DoS part means that a resource or resources are being blocked due to the sheer volume of traffic or requests wedging/crashing a service, interface, server or whatever is being offered up. DDoS attacks do not care about whether or not the server actually is able to serve up something. The mention of a captcha system is kinda silly because the point of the DDoS is make the web server too busy answering initial SYNs to even both with any other content. If anything, captcha would make the situation worse since you have a backend script generating a code on the fly and awaiting input while another 499 new connections per second are coming inbound and cause the same script to kick off thus not only tying up the IP stack with a bunch of SYN-WAIT connections but also a CPU getting hammered making a user input code for validation that will never be answered. The risk of a proxy-based solution in the form of DNS redirection is the timing of the detection of the attack and the lag involved of changing A records to point to a new host. It could take 15 minutes or it could take 3 days. If the provider (and by provider I mean the DDoS mitigation solution) is hosting the DNS from day one, keeps low TTLs that will be respected by upstream caching DNS servers, then the solution could work on a "quick" time scale. We're assuming some sort of rapid detection that would case a diversion to a scrubbing center as well. If the detection process is manual, then who knows how long it would take... Enter a flow based solution which is really the only way this should be done. A provider offers a solution that monitors NetFlow, IPFIX, JFlow, etc data and ideally combined with other telemetry sources like syslog output, server load, etc to detect when a threshold has been met. The benefit of flow based analysis is that one can spot low bandwidth and slow resource leak based attacks as well as the more obvious flood based attacks. A flow based solution will allow for automatic redirection of traffic to the destination via a BGP (or IGP like OSPF, IS-IS, EIGRP but in 95% of the cases it will be BGP since it's happening in the provider cloud) route injection, usually a /32 route but can be zone based as well as host based, to a scrubbing center. The scrubbing center only allows connections with a completed handshake to go through and even then will still proxy those connections to make sure they aren't sequence number jumping and/or normalizing the TCP sequence to make sure something funky isn't going on within the TCP segments itself. When the attack dies down, the /32 route is withdrawn and traffic flows to the server like normal again. Beleive it or not, a flow based solution IS NOT expensive but it's also not something you, a server owner, can take part of. Simple flow analysis is "free" since all Cisco/Juniper routers come with flow exports in the base software. There are plenty of open-source NetFlow collectors out there and with IPFIX becoming the standard, it too will have those collection engines. But you also have software providers like Narus, Mazu, NetQOS, Q1 Radar, etc out there that offer a lower cost (compared to an Arbor which is top of the line) analysis/alarming solution. Again, I will state that there are a number of host providers on WHT that specilaize in DDoS protection. The bigger question is why do you feel you are a DDoS target or why have you made yourself one? A vast majority of DDoS attacks are not random, they are focused and deliberate. While that doesn't mean don't protect yourself, that also means you are highly unlikely to garner enough attention to be the subject of a DDoS attack unless you manage to piss someone off. I've been hosting some pretty subversive anti-theist content for 10 years and not once have I been hit with a DDoS attack. And again, don't confuse a single target attack with a DDoS, they are two different things. Protecting against DoS vectors involving weakness in your server and/or software is a different thing and completely unrelated to bandwidth starvation as one potential vector.
Posted by EricTham, 04-20-2009, 10:51 PM Hi, First of all thank you everyone for enlightening me so much, in fact i called up Prolexic, they quoted me 12k for setup and 10k monthly to "protect" us. Mentioning 100gigs of attack also not a issue to them. They mentioned that upon an attack, we just need to route the traffic to them to clean it up and they will pass back the clean traffic to our servers (at a cap of 2mb outgoing to my server). Did i hear wrongly, how can we route the traffic to them from our firewall when my own bandwidth provider hole is only 10mbits dedicated? Do they actually mean to use Zoneedit/EasyDNS and change the domain A records to them? When they say they cleaned the traffic and pump back the traffic to our server (2mb), what if they pump 4mb ? (Every extra 1mb is USD$350). What i mean is, we will not know what they pump back to us is really clean traffic? They can just let 20mb of traffic back to me and charge me... this is dangerous. hi mike, yes my customer is a ddos targeted customer, their business will make them have lots of ninjas trying to kill the king. So i am expecting them to attack.
Posted by CiscoMike, 04-21-2009, 01:02 AM http://www.prolexic.com/admin/source...c_Overview.pdf They're doing a "clean pipes" solution. From their own PDF, they can do it via DNS proxy (what we've discussed), via BGP injection (what I recommended) or via GRE tunneling (interesting concept but still requires a routing protocols + now hardware to support the process) or via a connection to/through them directly. If you did the DNS solution, yes, you would point your A records at them once ***you*** determined there was an attack. They would then spoof proxy the connection back to you after it's gone through their scrubbing center. And as I've said, the effectiveness of this solution is 100% dependent upon you detecting the attack, your DNS server/service allowing low per-record TTL times and upstream DNS caching servers respecting the low TTL. Failure at any one of these links minimizes the effectiveness of a DNS based solution. If/when you sign a contract, have a SLA and agreement in place to hopefully prevent this. 12k for setup and 10k for monthly charge is a lot of cash. You would be better off hosting with a provider that has dedicated equipment for DDoS scrubbing and specalizes in DDoS mitigation. Or co-locate equipment yourself and utilize S/RTBH filtering (a poor-man's DDoS mitigation) in conjunction with an automated system from Narus/Mazu to help manage it for you.
Posted by RajuGuide, 04-21-2009, 03:20 PM quote=EricTham;6137316]Hmm , so it should be 1:Anti-DDOSS Company 2:Firewall 3erver 4:BackupBank ? quote] Hi Eric, Totally agree with CiscoMike when he says 12k for setup and 10k for monthly charge is a lot of cash. You would be better off hosting with a provider that has dedicated equipment for DDoS scrubbing and specalizes in DDoS mitigation. Instead of a generic firewall (such as Cisco ASA, which is not designed for DDoS mitigation), have a look at a hardware firewall that is designed specifically for DDoS mitigation. It should be able to keep you from going to Anti-DDoS companies for a longer period - just in case you have to go there. A hardware logic based firewall such as one recommended by Juniper is going to be the best bet. You can see the features here. Many leading hosts and their end-customers use it globally to do their own collocated DDoS mitigation. Hope that helps.
Posted by CiscoMike, 04-21-2009, 04:01 PM No no no...not a good idea. It's still a firewall that pretends to be able to handle DDoS mitigation and is no different than a Juniper, Cisco, Checkpoint or otherwise badged firewall. It suffers from the exact same limitation that all firewalls do - it's stateful and it's always in-line. It has a max of 1Gbps of real bandwidth (2Gbps is marketing math) and it maxes out at 1M total connections with a ceiling of 100k CPS. A botnet of 10k hosts can tip that thing over in about 10-15 minutes on a slow day. We're going back to flow based analysis engines. The IntruGuard has some nice features (like honeypots, you can do that on pretty much anything out there though) and behavior analysis but things like SYN Proxy (aka TCP Intercept/SYN Cookies) is a self-DoS just begging to happen and rate-limiting is useful only if it can be enforced upstream from the device. Otherwise the rate-limiting is about as useful as mod_deflate or mod_evasive which is not useful at all since the bandwidth is still consumed and a connection entry is still taken up. This is just another case of marketing getting in the way of an actual functioning solution. We need to step away from marketing and the drivel spit out by the hardware vendors and look at solutions that actually do the job correctly. If the total pipe into the customer site is under 1Gbps, then fine, I conceed that the IntruGuard is fine but given those conditions, so is an ASA 5550/5580, Juinper SRX, Crossbeam or Checkpoint appliance. As soon as your pipe is larger than your total firewall/IPS capacity, then the firewall/IPS/in-line security "tool" becomes an inadvertant DDoS target. Even when the bandwidth issue is out of the equation, you still need to worry about the CPS / total connections factor. Specific to the IntruGuard, since it's ASIC based you can't add new features without a re-spin where Cisco and Checkpoint just need to rev their code. Still, firewalls, of which IntruGuard really is, are not ideal for DDoS mitigation. Stick with flow based analysis and scrubbing centers. That's the best way to do it and the only predictable and reliable way to do so. Find providers that offer that sort of protection and be done with it. Everything else is going to be a hodge-podge / band-aid solution that will work under certain conditions but far from all scenarios out there. edit: If the Intruguard could support host route injection and be clustered to scale, then it would be a good scrubbing center solution. The beef here is that the IntruGuard is sitting in-line all the time and has no other intelligence into the rest of the network. • Dynamic Filtering • Active Verification • Anomaly Recognition • Protocol Analysis • Rate Limiting • White-list, Black-list, Non-tracked subnets • State Anomaly Recognition • Stealth Attack filtering • Dark address scan prevention • Source Tracking • Legitimate IP address Matching (for anti-spoofing) are all pretty things but they don't do anything if the box is easily overloaded and are prone to false-positives if they are in-line all of the time. • SYN Proxy • Connection Limiting • Aggressive Aging • Legitimate IP Address Matching • Source Rate Limiting • Granular Rate-limiting are all items that your off-the-shelf firewalls do today only at significantly higher rates (12Gbps on the 5580, up to 120Gbps on the SRX, up to 80Gbps on a CrossBeam X chassis). I'm not trying to put your point down but I think it's important to levelset on what is actually being recommended and again, the IntruShield is really nothing more than a firewall with a pretty dress on it and called a DDoS mitigation tool where it's really not well suited for that role. Last edited by CiscoMike; 04-21-2009 at 04:08 PM.
Posted by RajuGuide, 04-21-2009, 04:38 PM I guess the point here is that how many people today use 12Gbps, 120Gbps or 80Gbps bandwidth over the Internet. I was only mentioning the need for a product for DDoS mitigation when there is a need that firewalls can't meet. Most firewalls will simply allow port 80 traffic or deny it. There is nothing in between there - which DDoS mitigation solutions provide. If the need is for a solution that requires sub-1-Gbps bandwidth and painless nights, these DDoS mitigation solutions are a better fit than directly going to a DDOS-scrubbing center (which are too expensive) and a better solution than generic firewalls which don't handle such attacks too well. Solutions such as IntruGuard and Top Layer are based on Field Progammability and are therefore upgraded regularly based on new attacks - at the same time giving the advantage of being hardware logic - rather than being software. Cisco itself very clearly says firewalls (including their own ASA firewalls) cannot protect from DDoS attacks. I am quoting them here: End Quote. Eric, hope you can above the marketing and get into a technical analysis here of cost/benefits.
Posted by CiscoMike, 04-21-2009, 05:22 PM All stated previously. Will an in-line device work in some cases? Yes. In most cases where the attack is deliberate and focused, probably not. Especially with a 1Gbps cap and a 100k CPS ceiling. The OP already said his customer was going to be a target and he's looking for a high-end solution. That pretty much points to a scrubbing solution and from my first post on, I've said look at a provider solution. The bigger point is that state in-line = high risk of unintended DoS due to rolling over the state engine on the appliance. Believe me, 100k CPS is trivial to exploit.