NGI-NZ Technical Specification
DRAFT
This document is a DRAFT that sets out the overall specification for the New Zealand NGI network.
Introduction
The task of linking the partners of the NGI Society with a high speed network at reasonable cost is one which requires a simple design using commodity components. The over-riding design model can be characterised as a Nearlynet approach as described in "Permanet, Nearlynet, and Wireless Data",
http://shirky.com/writings/permanet.html
The NGI network will based on a `Research and Education' approach to network provisioning, for both its service model and its pricing.
Although some applications (such as `Access Grid' conferencing) require tens of Mb/s connectivity for periods of hours, others (such as Grid computing) will be much more bursty, requiring hundreds of Mb/s intermittently for periods of minutes. To support the expected traffic requirements, although the network links will run at speeds which are multiples of 1 Gb/s, they will be operated at utilisations of the order of 5-10% so as to provide headroom for the bursty
traffic.
As the load increases we expect that additional link capacity will be added so as to maintain the low utilisation level. NGI-NZ members further expect that the network pricing model will be based on cost-sharing for the actual costs of installing and running it. In particular, network pricing (at least for the network backbone) will not be based on traffic usage or on backbone link speed.
Our design principle is to keep the network simple and open, so as to allow NGI-NZ to take advantage of emerging new technologies, or changes in the pricing of existing technologies.
Since the design is based on commodity network equipment, and NGI-NZ needs to contain network operations costs, the network reliability level is not specified; instead it is left for negotiation between NGI-NZ and potential suppliers. Nonetheless, experience with networks of this kind suggests that acceptable reliability is achievable.
Architecture
The network needs to support a variety of protocols and services (including IPv4, IPv6, multicast and other as yet unspecified facilities), with equal ease of connectivity for each member. To reflect this, the network design is based on a simple layer 2 backbone.
The initial network will be built using an architecture based on `GigaPoP' interconnection points at neutral physical locations, initially in each of the six `campus' cities, i.e. Auckland, Hamilton, Palmerston North, Wellington, Christchurch and Dunedin. We expect that additional GigaPops will be installed in other centres quite quickly as the demand develops.
The GigaPoPs will allow connections from users other than the initial Society members and, since the GigaPoPs will be sited in neutral locations, they will also allow network capacity to be provided by alternative suppliers. The initial network roll-out should not impose barriers which would preclude such additions. As well as allowing the ability to move data at high speed and low cost, the GigaPoP design will allow the construction of a nationwide environment for Border Gateway Protocol (BGP) peering between the entities connected to it.
In keeping with the view that the design should use commodity components to minimise capital spend and network management overhead, the network will use links which are presented as copper gigabit Ethernet interfaces to the various network devices operated by the NGI consortium and its members. In summary, the design philosophy is not "IP everywhere" but "Ethernet everywhere".
Network Backbone
The Network will consist of four logical parts, as described below.
GigaPoPs
There will be six GigaPoPs, located in Auckland, Hamilton, Palmerston
North, Wellington, Christchurch and Dunedin. The GigaPoPs will be open interconnection points, housed at neutral sites, i.e. each GigaPoP site will allow access to all connectivity providers on equal terms.
Each GigaPoP will be based on a gigabit Ethernet switch. These switches will provide Layer 2 connections between GigaPoPs, and will terminate access connections for NGI-NZ member sites.
Inter-GigaPoP links
Inter-GigaPoP links will interconnect the GigaPoP switches. The links
need not themselves be Ethernet-based, but they must behave as a gigabit Ethernet segment so as to plug directly into gigabit Ethernet interfaces on the switches.
In order to support the expected growth in NGI traffic, we require the ability to add further gigabit segments so as to increase link bandwidth in 1 Gb/s steps. When pricing becomes attractive, we expect to move to using 10 Gb/s links in the network backbone.
One way to achieve this would be to use Coarse Wavelength Division Multiplexing (CWDM) on dedicated fibre. Note, however, that we are not necessarily looking for direct access to dark fibre. What we require is circuits that provide multiples of gigabit Ethernet connectivity.
When bonding of multiple gigabit Ethernet links is used in the future
to provide higher speeds on the core, we expect the links provided to
be carried via the same path through the supplier's network.
Where necessary, we will also be seeking clear indication of physical
diversity in the provider premises and between the GigaPoP locations.
This will be needed to support the use of redundant GigaPoP
facilities in each of the "campus" cities.
NGI-NZ Member Access
NGI-NZ members will work with the NGI Network Operator so as to provide the most effective connection to their nearest GigaPoP. Member access is discussed in detail in section (3) below.
International Gateway
The NGI-NZ International Gateway will provide peering with Research & Education (R&E) networks in other countries, e.g. AARNET, Internet2, APAN, GEANT.
Peering with such networks will be negotiated by the NGI-NZ Board, and transit capacity negotiated by the NGI-NZ board (with technical support from NGI-NZ Network Operator).
Acceptable Usage Policies (AUPs) for the R&E networks will be implemented by the NGI-NZ network's BGP policy, so that each member site will receive routes only for those R&E networks they may connect to.
Technical Requirements for network elements
The initial network will provide a Layer 2 Ethernet network
distributed across six GigaPoPs so as to form a logical backbone
linking access routers at each NGI Member site.
No spanning tree information must be passed from any supplier'snetwork to the GigaPoP switches. Spanning tree Bridge Protocol Data Units (BPDU's) from the GigaPoP switches must be passed transparently without modification through the network.
Ethernet multicast and IGMP must be transparently passed on all links.
802.1p prioritisation must be transparently passed on all links
Gigabit Ethernet (1 Gb/s) or Fast Ethernet (100 Mb/s) links should be
presented on copper (RJ45) rather than fibre wherever possible.
Network links must:
(a) Support Jumbo Frames, i.e. the maximum transmission unit (MTU)
for the network will be 9000 bytes.
(b) 802.1q VLANs with 802.1p priority signaling.
This implies that if a circuit is commissioned over a supplier's
switched network which uses 802.1q, then that network needs to
support the transmission of 802.1p and q within 802.1q.
(c) Support IPv4 Internet (layer 3) traffic.
(d) Support IPv6 Internet traffic natively within New Zealand,
and either native or via 6over4 tunnels elsewhere.
Transit time for intercity links shall be no more than four times the speed-of-light time between the cities, e.g. Auckland-Wellington 9 ms, Auckland-Dunedin 18 ms.
Within any 30-minute period the average IP Delay Variation (IPDV) on any link shall be no more than 1 ms. The maximum IPDV within any 10-second period shall be no more than 3 ms.
NGI-NZ expects to measure network performance on an ongoing basis.
Measurements needed to support daily operations will be made by the NGI-NZ Network Operator, provision should be made for other measurements (and measurement research) by research groups from NGN-NZ member sites.
Details of the network design will be finalised by discussion between NGI-NZ (represented by it's Technical Working Group) and NGI-NZ's Network Operator.
NGI-NZ Member Access
Typically, each institution will be connected to its nearest GigaPoP by a connection that behaves like a direct gigabit Ethernet link, though in some situations 100Mb Fast Ethernet may be used to provide an initial connection.
Where possible existing infra-structure will be used to establish access for each member site. If there is no existing infra-structure in place, then NGI-NZ (with technical support from its Network Operator) will discuss establishing a link jointly with the member in question, any other affected member, and suppliers.
Where several member sites are located near to each other it may be sensible to establish a single link to one site, and install a switch there to share the access link among the member sites. Such a switch would form an extension to the GigaPoP switch, and would be maintained and managed by the NGI-NZ Network Operator.
Overall, NGI-NZ will help to ensure that the most effective solution for each member or group of members can be found.
As mentioned in section (1) above, the NGI-NZ network is based on 'Research and Education' requirements, i.e. it provides gigabit-speed communication between member sites, high-speed access to international R&E networks, and a cost-sharing pricing model.
In the longer term NGI-NZ may wish to provide commodity Internet access for its members, Initially, however, our goal is to establish the network and demonstrate it's performance and stability. During this startup phase, members will be expected to configure their boundary routers so as to multihome, using BGP peering, with both NGI-NZ and with whichever commodity ISP they choose. In most cases that will be their current ISP.
It should be noted that while the circuits which provide access from the NGI members to the GigaPoP and the circuits which provide the core capacity linking the GigaPoP may well be provided by the same provider using the same technology platform, the design and delivery of those circuits should be independent and make no assumptions about how the edge or core will operate.
Other Services
The sections above have described the initial NGI-NZ network, and its initial requirement to offer effective `R&E-style,' i.e. high-speed bursty, connectivity. Once the network is well-established, NGI-NZ may consider providing further services over it. The following sections describe some possible services.
Commodity Internet Connectivity
For NGI-NZ to provide commodity Internet connectivity, we would peer with New Zealand ISPs via several (at least two) of the GigaPoPs, and buy service from those ISPs in bulk on behalf of NGI-NZ members. Providing this service would effectively make NGI-NZ an ISP (as viewed from its member sites). It would also introduce more stringent service level requirements than are needed for an R&E service, such requirements could well differ from site to site.
Plans to provide such a service will clearly need to be fully discussed by the NGI-NZ members before any technical aspects of it were considered.
Converged Service Access for Member Sites
Longer term, sites might be able reduce their access costs to other network services by using their NGI-NZ access link to reach their nearest GigaPoP, and using a switch located there to connect to services other than NGI-NZ.
In such cases the GigaPoP would be telehousing the site's switch; such arrangements will need to be arranged with the NGI-NZ Network Operator.
Considerations for the Future
While the network will be established using Layer 2 switching, as the complexity of the network increases NGI-NZ needs to retain the option of replacing any of its network elements - switches, routers, inter-GigaPoP links or Access links - with newer-technology alternatives.
We believe that our design principle (keep it simple and open) offers the best way to achieve this without major restructuring of the circuit components.
ENDS
|
