VINI User's Guide
This guide is meant to help new users of VINI to create virtual topologies and understand how to forward traffic over them.
Creating Virtual Topologies
All PlanetLab users can now access the VINI testbed, and create virtual networks within their slices, via the SFA federation interface. This interface is being developed in the context of the NSF GENI project.
On VINI, the network inside a slice is virtualized. By default there are no network interfaces visible inside a slice. To communicate with other slivers inside your slice, you need to set up a virtual topology connecting the slivers with virtual links. The endpoints of these virtual links will appear as interfaces inside your slice once they are created. A restriction imposed by VINI is that virtual links can only be added between nodes that are adjacent in the physical topology (meaning there are no intervening VINI nodes). The maps on the front page show the physical topologies of NLR and Internet2, along with the PoP locations where VINI nodes are installed within these networks. So for example, if you add slivers to nodes node1.chic.internet2.vini-veritas.net and node2.kans.internet2.vini-veritas.net, a virtual link connecting these slivers will appear inside them since Chicago and Kansas City are considered adjacent sites.
If you already have a PlanetLab slice, and you wish to add some VINI nodes, the basic steps are as follows:
-
Download and install the SFI command-line tool. We recommend using version 1.0-7 or later of the SFI tool.
-
Create the sfi_config file. Set the value of SFI_SM to:
SFI_SM='http://www.vini-veritas.net:12346/'
-
Get a description of the available resources, called an RSpec (resource specification), by running sfi.py resources. This will tell you the physical nodes and links on which it is possible to create slivers and virtual links.
-
Edit the RSpec to describe the resources to allocate to your slice. Add sliver tags to the nodes on which you want slivers, and vlink tags to the links on which you want to create virtual links between slivers. You can set bandwidth limits on the virtual links as well.
-
Submit your resource request by running sfi.py create slice-HRN rspec-file. If this operation does not return an error message, then the resources have been added to your slice.
- You can see what resources are currently allocated by running sfi.py resources slice-HRN. This operation will return a list of all the resources in the testbed, with sliver and vlink tags to indicate which resources have been allocated to your slice.
What You'll See
When you login to a slice, you can see the virtual topology using standard Linux tools like ifconfig. For example, here is the view on one of the slivers within a slice that was created on vini1.*.internet2.vini-veritas.net:
bash-3.2# hostname
vini1.losa.internet2.vini-veritas.net
bash-3.2# /sbin/ifconfig
a101x30 Link encap:Ethernet HWaddr D6:93:3C:D3:3C:5C
inet addr:192.168.21.36 Bcast:192.168.21.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1458 Metric:1
RX packets:12 errors:0 dropped:0 overruns:0 frame:0
TX packets:2 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:768 (768.0 b) TX bytes:168 (168.0 b)
a101x32 Link encap:Ethernet HWaddr 16:DB:82:AF:F3:49
inet addr:192.168.23.36 Bcast:192.168.23.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1458 Metric:1
RX packets:6 errors:0 dropped:0 overruns:0 frame:0
TX packets:3 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:384 (384.0 b) TX bytes:258 (258.0 b)
a101x34 Link encap:Ethernet HWaddr BE:B4:BB:F6:47:77
inet addr:192.168.22.36 Bcast:192.168.22.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1458 Metric:1
RX packets:4 errors:0 dropped:0 overruns:0 frame:0
TX packets:2 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:252 (252.0 b) TX bytes:168 (168.0 b)
nat101 Link encap:Ethernet HWaddr E6:4D:7B:59:FB:89
inet addr:10.0.101.2 Bcast:10.0.101.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:6 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:384 (384.0 b) TX bytes:0 (0.0 b)
bash-3.2# /sbin/route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
192.168.23.0 * 255.255.255.0 U 0 0 0 a101x32
192.168.22.0 * 255.255.255.0 U 0 0 0 a101x34
192.168.21.0 * 255.255.255.0 U 0 0 0 a101x30
default 10.0.101.1 0.0.0.0 UG 0 0 0 nat101
The interfaces starting with "a" are the endpoints of the virtual links. This sliver has virtual links to nodes in Seattle, Salt Lake City, and Houston. The virtual link name consists of the letter "a" + the slice's GRE key + "x" + the ID of the remote node (all nodes have unique IDs in the VINI Central database). So you can figure out the remote endpoint of the virtual link by mapping this node ID to the nodename. Currently, all virtual links are point-to-point between two slivers.
The interface starting with "nat" leads via NAT to the underlying physical network. You can contact remote servers using this interface, for example to pull content into your virtual network.
Automatic Address Assignment
Each virtual interface in the default topology is automatically assigned an IP address. Each virtual link is assigned a /24 network of the form 192.168.linkID.0, where linkID uniquely identifies the physical link between VINI sites in the underlying topology. Each sliver is then assigned an address of 192.168.linkID.nodeID, where nodeID uniquely identifies the VINI node in the VINI Central database. This algorithm ensures that every interface in the virtual topology has a unique address, and both endpoints of a virtual link belong to the same network segment.
The NAT interface is assigned an IP address of 10.0.egreKey.2, where egreKey is a unique EGRE key assigned to the slice. Traffic sent on this interface with a next-hop of 10.0.egreKey.1 will exit through the NAT.
A generated /etc/hosts inside each sliver maps IP addresses to hostnames for all hosts in the virtual topology.
Setting Up Routing
The virtual topology described above does not have packet forwarding configured, so only adjacent nodes in the virtual topology will be reachable. Some additional steps are needed to actually forward packets across the topology. To make it easy for users to get started building routing overlays on VINI, some basic configuration files for Quagga are written inside your slice. You can enable OSPF routing across your virtual topology in a few simple steps:
- Install the Quagga software inside your slice.
- Point Quagga to the generated config files in /etc/quagga:
[pl_trellis@vini1 ~]$ su bash-3.2# cd /etc/quagga bash-3.2# ln -s zebra.conf.generated zebra.conf bash-3.2# ln -s ospfd.conf.generated ospfd.conf
- Start up Quagga:
bash-3.2# /sbin/service zebra start; /sbin/service ospfd start Starting zebra: [ OK ] Starting ospfd: [ OK ]
Once the routes have propagated, you should be able to contact non-adjacent nodes in the virtual topology:
[pl_trellis@vini1 ~]$ hostname vini1.kans.internet2.vini-veritas.net [pl_trellis@vini1 ~]$ traceroute vini1.princeton traceroute to vini1.princeton (192.168.1.1), 30 hops max, 40 byte packets 1 vini1.chic.internet2 (192.168.15.18) 10.755 ms 10.734 ms 10.744 ms 2 vini1.wash.internet2 (192.168.12.22) 27.572 ms 27.570 ms 27.581 ms 3 vini1.newy.internet2 (192.168.16.20) 32.956 ms 33.001 ms 32.977 ms 4 vini1.princeton (192.168.1.1) 36.546 ms 36.521 ms 36.540 ms
Understanding How Routing is Configured
VINI generates some basic Quagga configuration files, customized for the current virtual topology, into /etc/quagga/. VINI configures Quagga to advertise three types of prefixes via the OSPF protocol:
- Virtual links (192.168.x.y/30 subnets)
- OpenVPN clients (10.x/16 subnets)
- Any prefix added as a kernel route (e.g., with /sbin/route add -net ...)
To set up a particular node as an "egress" node for a particular destination, simply add a route for the destination prefix on that node. For example, adding a forwarding table entry for www.vini-veritas.net to the NAT interface on host vini1.princeton.vini-veritas.net causes all other nodes in the virtual network to route packets to www.vini-veritas.net to this node:
[pl_expire@vini1 /]$ hostname vini1.princeton.vini-veritas.net [pl_expire@vini1 /]$ /sbin/ip route list 128.112.139.61 128.112.139.61 via 10.0.106.1 dev nat106
On another node:
[pl_expire@vini2 /]$ hostname vini2.chic.internet2.vini-veritas.net [pl_expire@vini2 /]$ /sbin/ip route list 128.112.139.61 128.112.139.61 via 192.168.38.94 dev a106x23 proto zebra metric 20 [pl_expire@vini2 /]$ traceroute www.vini-veritas.net traceroute to www.vini-veritas.net (128.112.139.61), 30 hops max, 40 byte packets 1 vini2.wash.internet2 (192.168.38.94) 16.809 ms 16.848 ms 16.831 ms 2 vini2.newy.internet2 (192.168.42.93) 22.153 ms 22.207 ms 22.193 ms 3 vini1.princeton (192.168.2.85) 25.793 ms 25.777 ms 25.761 ms 4 * * * 5 vini.CS.Princeton.EDU (128.112.139.61) 25.912 ms 26.063 ms 26.049 ms
Note that the physical and NAT interfaces on vini1.princeton.vini-veritas.net do not show up in the traceroute for some reason.
Connecting External Users
External users can connect to services running inside a virtual topology. For example, you can run an OpenVPN server inside a sliver, and users can run OpenVPN clients on their desktops to forward their traffic into the virtual topology. Currently a VINI administrator must configure attributes on your slice in order to enable this functionality; please contact support@vini-veritas.net for help with this.
Slice Attributes
Here we describe the attributes that VINI admins can configure to enable various forms of user opt-in.
Packets coming from outside the virtual topology, and destined to a server running inside, must be port-forwarded through the NAT interface. The port_fwd attribute configures port forwarding through the NAT. The attribute takes three attributes: a protocol (tcp or udp), a from port (i.e., outside the NAT) and a to port (inside the NAT). So to forward public UDP port 12345 to port 1194 inside the NAT, the port_fwd attribute would be set to udp 12345 1194
Some services require a TUN/TAP device to be configured inside their slice. The create_iface attribute can be set to create the TUN/TAP interface and move it inside the sliver. It typically has a value of tun0 or tap0.
Since OpenVPN requires configuring both of the above attributes, for ease of administration we have created a single openvpn_tun_port attribute that combines them. Its value is the external UDP port to which users should connect. It creates the TUN iface inside the slice and sets up port forwarding (udp XX 1194).
Enabling OpenVPN
If the openvpn_tun_port attribute is set, VINI will also generate some sample OpenVPN configuration files in /etc/openvpn/ inside each sliver. To run OpenVPN, you can follow these steps:
- Install OpenVPN inside each sliver (e.g., 'sudo yum -y install openvpn' on all nodes)
- Link /etc/openvpn/server.conf -> /etc/openvpn/server.conf.generated
- Generate certificates and keys by following the directions in the OpenVPN documentation. Or for testing, simply grab this starter set and copy them to /etc/openvpn/keys inside the sliver. (Note that these keys are insecure of course.)
- Start up openvpn (e.g., 'sudo /sbin/service openvpn start')
The client must also be configured with a client configuration file and keys. A simple client configuration file can be found here; save it on the client machine as client.conf. If you are using the starter keys mentioned above, the client must download client.crt, client.key, and ca.crt and place them in the same directory as the client.conf file. Once properly configured, the client should be able to connect to the OpenVPN server running inside a particular sliver. Suppose the sliver is running on node vini1.foobar.vini-veritas.net and the slice's openvpn_tun_port attribute is set to 1234. Then the client can connect by running (as root):
# openvpn --config client.conf --remote vini1.foobar.vini-veritas.net 1234
This will set up a tun0 interface on the client, with an address in the 10/8 range, that is the endpoint of the VPN tunnel. Packets to 192.168/16 (the address space of the virtual topology) are forwarded into this tunnel by default. It should now be possible to contact addresses within the virtual network from the client:
[acb@cadenza pl_expire]$ ip route list 192.168.0.0/16 192.168.0.0/16 via 10.1.0.5 dev tun0 [acb@cadenza pl_expire]$ traceroute vini2.losa.internet2 traceroute to vini2.losa.internet2 (192.168.66.150), 30 hops max, 60 byte packets 1 10.1.0.1 (10.1.0.1) 1.265 ms 1.355 ms 1.352 ms 2 vini2.newy.internet2 (192.168.2.86) 4.088 ms 4.205 ms 4.202 ms 3 vini2.wash.internet2 (192.168.42.94) 9.505 ms 9.614 ms 9.612 ms 4 vini2.chic.internet2 (192.168.38.93) 26.442 ms 26.553 ms 26.548 ms 5 vini2.kans.internet2 (192.168.38.118) 37.007 ms 37.124 ms 37.121 ms 6 vini2.hous.internet2 (192.168.58.126) 51.201 ms 50.812 ms 50.821 ms 7 vini2.losa.internet2 (192.168.66.150) 82.969 ms 83.169 ms 83.027 ms
Note that IIAS assigns each OpenVPN server its own unique 10.x/16 subnet from which to assign addresses to clients. Quagga automatically advertises these subnets via OSPF within the virtual network. This means that packets destined to a particular OpenVPN client can be forwarded through the virtual network to the specific OpenVPN server to which that client connects, and then across the VPN tunnel to the client.
How To Create A "Virtual ISP"
Let's tie together virtual topologies, routing, and connecting external users. Suppose that an experimenter wants to use a virtual network to serve content from a set of external prefixes to clients -- that is, to act as a miniature ISP. Here are the additional manual steps needed to set this up using VINI.
First, the experimenter will somehow choose which node(s) in the topology will act as egress points for specific prefixes. On each node, the experimenter adds a forwarding table entry for that prefix, directing it out through the NAT. As mentioned above, Quagga will automatically advertise routes to the prefix through the virtual network, so that traffic destined to that prefix will be routed through the virtual network to the chosen egress node(s).
Second, the experimenter must push routes to the prefixes back to the clients. One way to do this is to add a line to the OpenVPN server configuration file for each prefix: push "route <prefix> <netmask>". The client's routing table will then be configured to route traffic to the specified prefix into the VPN connection, and from there it will traverse the virtual network. Since the client's prefix is also being distributed by Quagga, return packets to the client will be routed through the virtual network and OpenVPN tunnel back to it.
For example, on node vini2.losa.internet2.vini-veritas.net:
[pl_expire@vini2 /]$ sudo /sbin/route add -host www.isi.edu gw 10.0.106.1
Add the route to the client so that traffic to www.isi.edu is forwarded into the VPN:
[acb@cadenza pl_expire]$ sudo /sbin/route add -host www.isi.edu gw 10.1.0.5
Now the client can access www.isi.edu over our "virtual ISP":
[acb@cadenza pl_expire]$ traceroute www.isi.edu traceroute to www.isi.edu (128.9.176.20), 30 hops max, 60 byte packets 1 10.1.0.1 (10.1.0.1) 1.331 ms 1.459 ms 1.455 ms 2 vini2.newy.internet2 (192.168.2.86) 4.149 ms 4.251 ms 4.249 ms 3 vini2.wash.internet2 (192.168.42.94) 9.585 ms 9.692 ms 9.691 ms 4 vini2.chic.internet2 (192.168.38.93) 26.420 ms 26.539 ms 26.539 ms 5 vini2.kans.internet2 (192.168.38.118) 37.111 ms 37.224 ms 37.224 ms 6 vini2.hous.internet2 (192.168.58.126) 51.178 ms 50.819 ms 50.781 ms 7 vini2.losa.internet2 (192.168.62.150) 83.070 ms 82.954 ms 82.903 ms 8 * * * 9 64.57.18.65 (64.57.18.65) 83.218 ms 83.203 ms 83.160 ms 10 hpr-lax-hpr--i2-newnet.cenic.net (137.164.26.132) 84.174 ms 83.440 ms * 11 137.164.27.254 (137.164.27.254) 84.103 ms 83.720 ms 83.726 ms 12 198.32.16.29 (198.32.16.29) 83.821 ms 84.033 ms 83.781 ms 13 www.isi.edu (128.9.176.20) 84.040 ms 83.807 ms 84.122 ms