Using Gigabit Ethernet to Extend the Internet Exchange to the Metropolitan Area
Keith Mitchell
keith@linx.org
Executive Chairman
London Internet Exchange
The Gigabit Ethernet Conference
London, 1st July 1998
Disclaimer
•	This talk is not quite what was planned
•	We have been let down by:
•	Telecoms carriers
•	Site providers
•	Delays beyond our control
•	Lessons are relevant, if not all technical !
What is the LINX ?
•	UK National Internet Exchange Point (IXP)
•	Layer-2 LAN interconnection between layer-3 WAN Internet Providers (ISPs)
•	Not-for-profit co-operative of ISPs with international connectivity
•	Keeps UK domestic Internet traffic in UK
LINX Status
•	Established Oct 94 by 5 member ISPs
•	Now 48 members
•	steady linear growth
•	approx 1 new member every month
•	UK, European, International members
•	Neutral location at London Telehouse
•	Currently total inbound = outbound traffic peaking about 300Mbps
•	Traffic doubling every 4-6 months !
LINX Members
http://www.linx.net/members.html
Exchange Point History
•	Initially established in 1992 by:
•	MFS, Washington DC - “MAE-East”
•	Commercial Internet Exchange, Silicon Valley - “CIX-West”
•	Amsterdam, Stockholm, others soon afterwards
•	Now at least one in every European, G8, OECD etc country
IXP Architectures
•	Initially:
•	10baseT router to switch
•	FDDI between switches
•	commonly DEC Gigaswitches
•	More recently:
•	100baseT between routers and switches
•	Cisco Catalyst 5000 popular
LINX Architecture
•	Originally Cisco Catalyst 1200s:
•	10baseT to member routers
•	FDDI ring between switches
•	Now:
•	Member primary connections by FDDI and 100baseT
•	Backup connections by 10baseT
•	FDDI and 100baseT inter-switch
LINX Topology
LINX Infrastructure
•	5 Cisco Switches:
•	2 x Catalyst 5000, 3 x Catalyst 1200
•	2 Plaintree switches
•	2 x WaveSwitch 4800
•	FDDI backbone
•	Switched FDDI ports
•	10baseT & 100baseT ports
•	Media convertors for fibre ether (>100m)
Telehouse
•	Located in London Docklands
•	on meridian line at 0º longitude !
•	24x7 manned, controlled access
•	Highly resilient infrastructure
•	Diverse SDH fibre from most UK carriers
•	Diverse power from national grid, multiple generators
•	Owned by consortium of Japanese banks, KDD, BT
LINX and Telehouse
•	Telehouse is “co-locate” provider
•	computer and telecoms “hotel”
•	LINX is customer
•	About 100 ISPs are customers, including 40 LINX members
•	other members get space from LINX
•	Facilitates LAN interconnection
LINX Growth Issues
•	Lack of space for new members
•	Exponential traffic growth
•	Potential bottleneck in inter-switch links
•	Need inter-switch capacity higher than member capacity
•	Nx100Mbps trunking does not scale (MAE problems)
IXP Technologies
•	10Mbps Ethernet
•	100Mbps Ethernet
•	FDDI
•	ATM
•	Gigabit Ethernet
IXP Technologies -Ethernet
•	10baseT is only really an option for small members with 1 or 2 E1 circuits and no servers at IXP site
•	all speeds of Ethernet will be present in ISP backbones for servers for some time to come
IXP Technologies -
100baseT
•	Cheap
•	Proven
•	Supports full duplex
•	Meets most non-US ISP switch 	port bandwidth requirements
•	Range limitations can be overcome using 100baseFL

IXP Technologies - 
FDDI
•	Proven
•	Bigger 4k MTU
•	Dual-attached more resilient
•	Longer maximum distance
•	Full-duplex proprietary only
IXP Technologies -
ATM
•	Only used at US federally-sponsored NAPs, PARIX
•	Sprint, Pacbell, Ameritech, FT
•	Initially serious deployment problems
•	“packet-shredding” led to poor bandwidth efficiency
•	Now about 800-900Mbps traffic at NAPs
IXP Technologies -
ATM
•	Some advantages:
•	inter-member bandwidth limits
•	inter-member bandwidth measurement
•	“hard” enforcement of peering policy restrictions
•	But:
•	High per-port cost, especially for >155Mbps
•	Limited track record for IXP applications
IXP Technologies -
Gigabit Ethernet
•	Cost-effective and simple high bandwidth
•	Ideal to scale inter-switch links
•	Not good router vendor support yet
•	Standards very new
•	Highly promising for metropolitan and even longer distance links
LINX Growth Solutions
•	Find second site within 5km Gigabit Ethernet range via open tender
•	Secure diverse dark/dim fibre between sites from carriers
•	Upgrade switches to support Gigabit links between them
•	Do not offer Gigabit member connections yet
LINX Growth Obstacles
•	Poor response to Q4 97 site ITT:
•	only 3 serious bidders
•	none bundled any fibre
•	successful bidder pulled out after messing us around for 6 months :-(
•	Only two carriers were prepared and able to offer dark/dim fibre after months of discussions
LINX 2nd Site Status
•	Have secured good deal with two carriers for fibre
•	but only because LINX is special case
•	New ITT:
•	bid deadline mid-August
•	plan to have site go live early October
LINX Traffic Growth
•	Weekly total traffic (2-hour average)

•	Yearly total traffic (1-day average)
LINX Traffic Issues
•	Bottleneck is inter-switch link between Catalyst 5000s
•	Cisco FDDI can no longer cope
•	100baseT will soon fill
•	Need to upgrade to Gigabit Ethernet within existing site ASAP
Gigabit Switch Options
•	Looking at 5 vendors:
•	Cabletron/Digital, Cisco, Extreme, Foundry, Plaintree
•	Some highly cost-effective options available
•	But need non-blocking, modular, future-proof equipment, not workgroup boxes
Metro Gigabit
•	No real MAN-distance fibre to test kit out on :-(
•	LINX member COLT have kindly lent us a “big drum of fibre”
•	Most kit appears to work up to 5km
•	Some interoperability issues with dim to dark management convertor boxes
IXP Gigabit Futures
•	Vendor claims of 1000baseProprietary 50km+ range are interesting
•	Need abuse prevention tools:
•	port filtering, RMON
•	Need traffic control tools:
•	member/member bandwidth limiting and measurement
•	What inter-switch technology will support Gigabit member connections ?
Conclusions
•	Extending Gigabit beyond your LAN is hard, but not technically
•	Only worth trying if you have your own fibre
•	If carriers will not make dark/dim fibre available, they should offer managed Gigabit MAN services
•	should be cost-effective
Contact Information
•	http://www.linx.net/
•	info@linx.org
•	Tel +44 1733 705000
•	Fax +44 1733 353929