On Fri, Jan 26, 2001 at 01:07:00PM +0000, ETI - KOUROUMA Aboubacar wrote: > Hi all, > We are installing our new internet access via VSAT. For this I need > your ideas about the new configuration of my ISP site to ensure the > scalability and the reliability of services. > Actually, we have a 64 kbps link with a cisco 2511 connected by > leased line to the Telco network (Sotelgui). From the telco we have a > 126.96.36.199/26 Ip addresses. In a few days (6 days) we will have a > satellite link with 512 k / 256 k bandwidth. From this satellite link we > will get a new /24 IP addresses. We will keep active the link between us > and the telco as back up link and all traffic to 188.8.131.52/24 will be > routed on this link . The default route be the VSAT link. See the > attached .doc file. > How can I configure my site to ensure scalable internet services? I > mean can I use one cisco router or I have to configure more (I have in > mind the OSPF protocol with two routers) ? > Hope read you soon. > Cheers, > Aboubacar. Excellent question; I am copying this reply to a couple of lists as there may well be people in similar situations who have come up with different approaches which may be useful to you. There are a few comments and suggestions I can offer. To save some typing, I will call your 184.108.40.206/26 network "A" and your other network "B", and I'll give it IP range 192.0.2.0/24 for now. Whatever you do with this setup, you need to realise you won't get "full" resilience: that is, if someone is using an IP address of 220.127.116.11, and the link to your Telco fails, they will be unable to continue existing TCP sessions by redirecting their packets over the satellite link. This is because _inbound_ packets to 18.104.22.168/26 will always come in on the Telco link, and inbound packets to 192.0.2.0/24 will only come in over the satellite link. You will not get this level of resilience until you have your own CIDR address range (probably a /19 or /20) and are announcing it to your upstream providers via BGP, who in turn announce it to the rest of the Internet. Having said that, you are still in a good position and you can get a very good level of resilience - you will just need to make use of some of the application-layer failover features which are already built into protocols such as SMTP and DNS. Firstly, I'd say you want to make sure that always at least _one_ of your links is able to carry traffic. That means not putting both links into the same router, and not using the same hub or switch for your whole network. This is easily arranged: just keep the two connections entirely separate by using a second router for your satellite connection. Then if one router or hub fails, the other link is not affected. Telco Satellite | | | | RA RB | | ----+------------ ---+-------- 22.214.171.124/26 192.0.2.0/24 (Network A) (Network B) [There is an extra advantage here: it gives you symmetrical routing. That is, packets both to AND from 126.96.36.199 all go via the Telco link, and packets to AND from 192.0.2.0/24 go via the Satellite. If you had a single router, as in your diagram, the problem is: which provider do you point defaultroute at? To achieve symmetrical traffic flows you would need to use "policy routing" on your single router, which means using a different next-hop depending on the _source_ IP address of each packet. If you have two routers, then anyone on network A just points defaultroute at RA, and anyone on network B points defaultroute at RB] If you are using UPSs or other power filtering, you would be well advised to put Network A's equipment on one UPS and Network B's equipment on a different one, so you are protected against a UPS failure. Now, this will work just fine. The only problem is, if a machine on Network A wants to transfer data to a machine on Network B, it will go all the way via the Internet and back... which will be very slow and waste bandwidth on your expensive links. To fix this, you need a short-cut path between the two networks. If this was between two different providers this would be called "peering" and you would use BGP, but since it's all your own network you can just run OSPF. The most resilient way to do this is with a third router, with two ethernet ports. This could even be a FreeBSD or Linux PC running gated. Telco Satellite | | | | RA RB | | ^ ----+------------ RC -----------+-------- OSPF 188.8.131.52/26 192.0.2.0/24 v (Network A) (Network B) Enable OSPF on each of the interfaces, and remember "redistribute connected subnets" so they tell the other routers about the networks they are connected to. On RA and RB you can announce defaultroute, like this: router ospf 1 passive-interface serial0 ! Must NOT talk OSPF to Telco/Satellite! redistribute connected subnets redistribute static subnets default-information originate metric 100 ip route 0.0.0.0 0.0.0.0 serial0 ! Defaultroute via Telco/Satellite If the uplink fails at layer 1 or layer 2, the static default route will be withdrawn, and the router will in turn stop announcing it into OSPF. [You can get slightly better detection of a failed link by getting your upstream provider to announce a defaultroute to you using BGP, but this probably isn't worth the effort. Use Cisco HDLC or PPP keepalives to monitor the link] Now, if a machine on network A pings a machine on network B, it will first go to its defaultroute, i.e. router RA. Because of OSPF, this will have learned the route to B via RC, so the packet goes there. Then RC will deliver it to the machine on B. Because this is one hop more than the 'optimal' route, RA will generate an ICMP redirect to try and tell the machine on A the better route. This can cause more problems than it's worth, so I strongly recommend you turn off ICMP redirects on all interfaces on all routers - and also proxy ARP. interface ethernet0 no ip redirects no ip proxy-arp Now, you say that you want all traffic to 184.108.40.206/24 to go via the Telco link, presumably because these machines are all within your country and reached via the Telco. This is now easy to arrange: on router RA only, add a static route. ip route 220.127.116.11 255.255.255.0 serial0 This will be learned by the other routers via OSPF. So even if a machine on Network B pings 18.104.22.168, the packet will be routed out via RC and RA. Unfortunately, we can't control the path the packet takes on the way back. When machine 22.214.171.124 sends a packet to 192.0.2.1, it will hit the Telco network which will send it out via the Internet, i.e. the "long way". If you want to fix this, you would have to get Telco to add a static route to network B on _their_ router - or set up proper peering using BGP. -------------------------------------------------------------------------- Right, so now you have two networks, linked together to optimise the traffic flow. But still, if one link fails, half your systems go down. How can you use this to improve your service? Well, you need to work with the built-in resilience mechanisms that exist in application-level protocols. (1) DNS primary and secondary If you are running your own authoritative DNS, put the primary on Network A and the secondary on Network B. Then, if one of your links fail, the outside world will still be able to reach one of your nameservers. Note that renumbering a nameserver is not trivial - if you have lots of zones, which have 'glue' records in the zones above them, you will have to do some extra work. (2) Mail delivery using MX records You can arrange that incoming mail will continue to arrive, even if one of your links fails, by using different-priority MX records and a backup mail relay. example.com. MX 10 mail1.example.com. MX 20 mail2.example.com. mail1 A 192.0.2.1 mail2 A 126.96.36.199 In this simple example, mail1 is your 'main' mailserver, e.g. which holds POP3 accounts. Any mail to email@example.com will be delivered to this machine if the satellite link is up, because it has the lower MX record priority. But if the satellite link is down, the machine on the Internet which is sending to you will try it and fail. It will then fall back to using the second MX record, and deliver to mail2. If mail2 is configured properly, it will accept the mail and relay it to mail1 (which it talk to via router RC). So mail will flow normally. [You could choose to make mail1 and mail2 the same machine, with two network interfaces. But this is awkward to make work because it needs to learn which defaultroute to use, depending on which outside links are up and down. So you would end up having to run gated on your mail server... and you are not protected against the machine itself failing. So for these reasons, I recommend you have two separate mailservers instead] (3) Outgoing mail delivery If your customers send their outgoing mail via mail1, it will normally deliver directly out to the Internet. But what if the satellite link goes down? You can configure mail1 to try to deliver normally, and if it fails, to relay via mail2. Then mail2 can try again to deliver the message, using the link to Telco. Configuring this sort of fall-back scenario is I think possible in Exim. But there is a neater solution: (4) All outgoing UDP/TCP connections The problem with outgoing mail, in (3), is that the mail server has an IP address on the 192.0.2.0 network. If the satellite link fails, then actually outgoing IP packets _will_ be routed via RC and RA, because RA is announcing defaultroute. But the return packets will never come back, because they can only come down the (failed) satellite link. There is a solution here: configure Network Address Translation (NAT) on router RC, for packets which pass through it. You would need a ruleset which says: "If I am forwarding a packet with a destination address of 192.0.2.0/24 or 188.8.131.52/26, do not perform any NAT. Otherwise, if the source address is 192.0.2.0/24, NAT using my Network A interface address. If the source address is 184.108.40.206/26, NAT using my Network B interface address." I don't know how to configure this on a Cisco, as I have never used Cisco NAT. It's very straightforward under Linux though, something like the following although I've not checked the exact syntax: ipchains -A forward -d 192.0.2.0/24 -j ACCEPT # fwd but don't NAT ipchains -A forward -d 220.127.116.11/26 -j ACCEPT # fwd but don't NAT ipchains -A forward -s 192.0.2.0/24 -j MASQ # fwd and NAT ipchains -A forward -s 18.104.22.168/26 -j MASQ # fwd and NAT Then any machine (or modem) with a network B IP address still _will_ be able to access the Internet via the Telco link; the source address of each packet will be changed using NAT to a network A address, and so the responses will come back via Telco to network A. So you don't even need solution (3); your mailserver will be able to sending outgoing SMTP anyway. What it _can't_ fix is incoming connections. So if you have www.example.com. A 192.0.2.4 in the DNS, and your satellite link fails, then users on the Internet will be unable to reach www.example.com. If that's important to you, then host your website outside the country, or at least a mirror of it. But this does show it is possible to host the most important IP services in a very resilient fashion using your two links, namely: - E-mail (inbound and outbound) - Web surfing (outgoing HTTP connections) and other TCP/UDP services - DNS If you wish, you can have a degree of 'load balancing' between your two links, by choosing which machines/modems are plugged into network A, and which into network B. Hope this is useful, Brian.