Free (for all) Peering

Given that people keep bringing up the old "Peering is free Internet Access" chestnut, I thought I'd run through a quick pricing exercise.

Let's imagine that you live in London Docklands. You want some of this free Internet access you've been hearing about recently (e.g. this recent podcast: TWiG 12).

First off, you need to get into one of the many Colocation centres around London where there is a public internet exchange point. Lets choose Telehouse North.

A 100Mb/s ethernet link to Telehouse from somewhere in Docklands is likely to cost you around £2500 to install, and then £500 a month to keep running.

Now to peer at an IXP you need to talk BGP. This means you need your own IP address range, and an autonomous system number. The easiest way to get these things is to join the local internet registry for your region. For London this would be RIPE. Checking the 2010 fee schedule the sign-up fee is €2000 with an annual fee of €1300 for the extra-small registry class.

Once you have this sorted out you can sign up to the exchange.

Current LINX Membership fees for 100Mb/s ports are: £1800 annual membership fee. £160 per month for a 100Mb/s port.

The install fee for a 100Mb/s port in Telehouse is: £550

Lets add that up:

Link to Telehouse:	£2500
RIPE Membership:	£1800 (exch rate 0.9)
LINX Port:	£550
Total Install:	£4850
Link to Telehouse:	£500
Ripe Membership:	£100
LINX Membership:	£150
LINX Port:	£160
Total Monthly:	£910

So assuming a usage pattern of 60Mb/s you are looking at an initial outlay of nearly £5000 and getting your internet access at around £15 a month per Megabit/s. And you won't get all the sites you want either. You will only get access to other LINX members that want to exchange traffic with you.

Does peering still look free to you? I'd rather take a limited service cheap DSL service at home myself.

I have made a lot of assumptions in this email and the situation I have described is not realistic but I hope it has indicated that even though peering may not involve any cash changing hands between peers it does involve cost to all parties involved.

Peering (a.k.a SFI) certainly makes very good sense when you are doing a lot of internet traffic. It is not free though. The costs do come down quite a lot when there are very large volumes of traffic but they never go to zero.

Ping and OSI Annoyances

OK. Its been a while since I had a rant here but the wording in a couple of stories grabbed from my google reader stream this morning have sparked me off again...

First off, Lifehacker posted a recommendation for a tool called Pingtest. The guys who wrote this are the same guys behind speedtest.net I am sure they know the limits of ping and have made sure this will be reliable; however I can't applaud perpetuating the myth that ping is a reliable performance monitoring tool rather than a simple connectivity checker.

Secondly. Juniper and O'Reilly have just released a new book on green networking named "The Sustainable Network". This in itself isn't bad, and I intend to check it out using my Safari account. What annoyed me about the press release is the sentence: "(The Global network)'s ... future depends on ... industry innovation at all layers of the OSI model".

The OSI 7 layer model is a conceptual model rather than a practical one. It is still a useful learning tool but network designers and protocol architects will generally move past using it for actual deployments pretty rapidly.

The core Internet protocols have been around longer than the OSI model. TCP/IP layers do not correspond with the OSI layers. Even the ISO protocol designers had a hard time implementing the OSI model in real life and often utilised "pass through" funtions to skip pass layers they couldn't shoehorn in.

Network innovation doesn't depend on the OSI model. It depends on finding the right recursive layering model and evolving the control plane so that the current exponential growth curve of the default free zone can be flattened out.

For more on these subjects see:

Day, John - Patterns in Network Architecture (Prentice Hall) (website at http://pnabook.com/)

Various - Proposals to the Routing Research Group on addressign routing table scaling

Layering in the Modern Internet

OK, so I'm still reading Day's "Patterns in Network Architecture" and it's gradually bringing areas of what I do into better focus.

Open any networking book and you'll immediately be shown the classic OSI 7 layer model. As models go it does a pretty good job. It doesn't match the architecture of the Internet protocols though. Most of the books will point out that the OSI model came later. Few actually point out that the OSI model isn't flexible enough to model a scalable global Internet.

In simple networks you have services, hosts, interfaces and links. If you have two hosts communicating you have service communication encapsulated in TCP/UDP/etc. IP doesn't have a host layer (see HIP/SHIM6 for a couple of proposals) but DNS allows a mapping of a host name to an interface identifier at the host communication layer (I.e. IP address). Interface level communication is trending towards Ethernet. This leads back to the simplified view: TCP is transport/session layer; IP is network layer; MAC is data-link layer.

The Internet has a massive variety of configurations. A better (but still vastly simplified) model could be: Hosts talk using IP and could be relayed at a number of aggregation levels such as: Network (e.g. IGP area), inter-area (e.g. BGP confederation member or RR cluster), inter-entity (e.g. External ASN). Going back 10 years or so these boundaries would primarily just be defined by routing at the IP layer but now there are real moves to encapsulating protocols at these boundaries too. Within a service provider network the IGP is simply a bootstrap protocol for bringing up the MPLS infrastructure for intra-entity communication, BGP for intra-ASN MPLS (this is usually only seen in larger cores, this can also be used for inter-ASN MPLS but such uses are not generally for Internet applications). Inter-ASN communication today uses the same addressing scope as inter-host communication but there is active work to change this (See proposals under discussion in the routing research group such as LISP).

Some of these encapsulations allow for flexible mappings for services. An example would be that the ethernet layer in the first example could actually be running over the MPLS layer of the second example. For this reason I think that any inter-ASN method needs to support carrying more than just IPv4/IPv6 (the thought of configuring IPv4-over-Ethernet-over-MPLS-over-GRE-over-IPv6-over-LISP in the future leaves a bad taste in my mouth)

Recursive layering is a powerful concept but care must be taken that there is a real architectural reason for the layer. Layers should be created to solve problems, not mask them.

Save us Obi-Wan Torvalds, You're Our Only Hope

OK, random thought time.

I'm currently reading the extremely interesting "Patterns in Network Architecture: A Return to Fundamentals" by John Day. In it he points out that one of the fundamental benefits that caused the internet protocol suite to win out is that it was designed by operating system people, rather than telecomms people. He also puts forward that when addresses started getting low we missed a chance to go back to these principals and design a new scalable protocol using contemporary thinking on IPC, virtual addressing and the like and instead we just threw header bits at the problem. I'm not sure I'm 100% convinced by all of the arguments in the book but it is certainly thought provoking.

Now back to the title.

We all know what happened when Linus Torvalds got the itch to improve on the Minix OS. Many of us also know what happened when he got the itch to improve on the available distributed version control tools available a few years back. I am wondering whether he is itching as much as I am over the lack of take-up of IPv6 and the number of features still missing to facilitate migration from a v4 Internet to a v6 Internet.

If he is, and if Day is right, then an itchy Linus plus some time to code could be the best solution to the closing v4 exhaustion problem.

Does 21CN go far enough?

I read a post on a mailing list that got me to thinking. Recently there have been a number of live events (e.g. Wimbledon, The Ashes) which have been broadcast live online by one means or another (I personally quite like the video scorecard the BBC are streaming alongside the Ashes coverage). These events have caused a noticable rising of sustained traffic over business grade DSL during office hours (i.e. people tuning in while at work).

The perennial argument against IP Multicast is that the massive growth in IP video content is primarily on-demand where multicast doesn't give any gain; however the peaks seen during live events, as well as other research shows that there would still be significant bandwidth savings if live content could leverage multicast proplerly.

This moves on to my thoughts about 21CN. The primary DSL delivery methods in 21CN are WBC and WMBC (there is also IPSC for access to 20CN IPstream services). Largely these services mirror the IPStream service in that they are L2TP delivery of PPP services. Because these services are delivered to the ISP as a point to point L2 service this means that the ISP has to aggregate traffic within their network and streams have to be replicated before being fed in to the BT network. No cost saving to the ISP generally equals no driver to do any work on multicast services.

Of course a lot of this cost saving argument is moot if the ISP are charging their customers correctly. If you are being charged on a usage basis by your supplier, then you have to charge on a usage basis to your customer. If you charge flat rate and then get pwned when the customers actually use what they are paying for then you have no right to complain...

An alternative L2 for this could have been a VPLS or VLAN based ethernet broadcast network. In this type of deployment (similar to many cable deployments but not usually used for DSL) multicast traffic could be sent out using ethernet level multicast (being careful not to flood, otherwise this traffic would be sent to users who don't want it). This method of delivery would have a lot of positives but it is also very different from existing strategies. Much of the resistance to this idea would be from inertia (I certainly cringed when the idea popped into my head); however the more I think about it the more I like the possibilities for future expansion. The biggest scare factors would be direct customer to customer traffic at ethernet layer (could be mitigated by private vlan like split broadcast domain) and broadcast / flooded traffic.

CPE support could also be an issue.

A Fresh Cupcake

A new cupcake update just appeared on my mobile. Appears to be a security update rather than feature update.

Official build strings: Firmware version 1.5, Kernel Version: "2.6.27-00393-g6607056 san@sandroid #1" Build number: CRB43

User agent: "Mozilla/5.0 (Linux; U; Android 1.5; en-gb; T-Mobile G1 Build/CRB43) AppleWebKit/528.5+ (KHTML, like Gecko) Version/3.1.2 Mobile Safari/525.20.1" "-"

Same webkit version, so not a browser vuln... minor change in kernel version...

The Emperor's New Transport

All of the hype in networking at the moment seems to be surrounding the move to packet based transport with particular emphasis on Ethernet.

I have to admit to being confused.

Don't get me wrong, I'm a packet guy through and through and can totally understand the benefits of moving to a packet core. What I don't understand is the obsession on using Ethernet to do it.

One quick pedantic point I want to get off my chest: Ethernet PDUs are "Frames" not "Packets". That's just a terminology point though and just annoys me rather than being an argument for or against the applicability of Ethernet in the Packet Transport network.

Ethernet is a great LAN protocol. I have been working with it for years and have a pretty good understanding of it (there's a reason one of my unofficial job titles is "Switch Bitch"). But using the vanilla 802.1d and 803.3 standards it doesn't scale well.

Ethernet is a CSMA/CD technology (Carrier Sense Multi-Access with Collision Detection). These things are all important when considering the most basic setup with a number of devices communicating over a dumb hub/repeater.

I real modern day networks the CD part has largely been rendered irrelevant due to the prevalence of micro-segmentation and full-duplex links; when operating in full duplex you don't get collisions.

Lets imagine our next generation all-singing all-dancing packet transport network. Each node in the network is a packet switch. If you try and build this sort of network with Ethernet switches it won't work; spanning tree simply does not scale this way. Future enhancements such as TRILL (see RFC5556) will allow further scaling at the Ethernet layer but the protocol designers only see this scaling up to high 100s of nodes in the network. Plenty for a metro, but when talking of a National or International network that's not so much.

Another solution that seems well liked is MPLS-TP. I have nothing against this. I've been working with MPLS in the IP world for years and it generally does what it says on the tin. I tend to prefer an intelligently signalled MPLS network over an NMS configured MPLS-TP network but reducing complexity in the core allows for faster and cheaper boxes so it is forgivable.

If the node is an MPLS LSR then forwarding decisions are made based on MPLS labels, not on MAC addresses and the links between nodes are point to point (meaning the MA in CSMA/CD becomes redundant, leaving us with CS - what L2 protocols don't have carrier sense?). I suppose technically you could put a L2 switch between the nodes but why would you want to?

I still believe that in a point to point network a point to point protocol makes more sense. PPP over HDLC as done traditionally for POS certainly has its problems, the main one being unknown packing overhead due to having the escape the SOF/EOF flags. If you use PPP over GFP though the downsides pretty much go away. I would much rather see an international network built using some flavour of PPP over WDM than Ethernet over WDM.

Ethernet just doesn't seem to fit the requirements and the standards groups all seem to be rapidly bolting new features on (e.g. MAC-in-MAC, Ethernet OAM, 1588 Synchronisation) to make it more and more like the networks it is trying to replace...

I'm going to follow up on this post but I'll leave it for now with a question:

Is my standpoint Bill Gates saying "No-one will need more than 640K" or am I the little boy saying "The Emperor has no Clothes!"?