Wednesday 19 June 2013

Metadata via the DHCP namespace

This is a follow-on from the previous post that described how metadata requests go from instances to the Nova metadata service via a namespace proxy in a Quantum router namespace. This post shows how requests can go via a namespace proxy in a DHCP namespace instead.

This method can only be used on isolated subnets. First enable it in /etc/quantum/dhcp_agent.ini and restart the service.


# The DHCP server can assist with providing metadata support on isolated
# networks. Setting this value to True will cause the DHCP server to append
# specific host routes to the DHCP request.  The metadata service will only
# be activated when the subnet gateway_ip is None.  The guest instance must
# be configured to request host routes via DHCP (Option 121).
enable_isolated_metadata = True


This file will also need root_helper = sudo /usr/bin/quantum-rootwrap /etc/quantum/rootwrap.conf if you don't have the fix (due in 2013.1.2) for this.

The subnet must not have a gateway, so it should be created like this:


vagrant@controller:~$ quantum subnet-create net1 172.17.17.0/24 --no-gateway --name=sub1
Created a new subnet:
+------------------+--------------------------------------------------+
| Field            | Value                                            |
+------------------+--------------------------------------------------+
| allocation_pools | {"start": "172.17.17.1", "end": "172.17.17.254"} |
| cidr             | 172.17.17.0/24                                   |
| dns_nameservers  |                                                  |
| enable_dhcp      | True                                             |
| gateway_ip       |                                                  |
| host_routes      |                                                  |
| id               | a507c135-12b9-419c-ab43-ecaa7ca57f83             |
| ip_version       | 4                                                |
| name             | sub1                                             |
| network_id       | 01cce415-8e2b-4096-92b5-7acc67163d79             |
| tenant_id        | 7c52af88d14b45ae87cfbe154d8c8638                 |
+------------------+--------------------------------------------------+


Now the DHCP agent has added address 169.254.169.254 and started a metadata namespace proxy listening on TCP port 80 in the DHCP namespace for this Quantum network on the network node.


root@netnode:/# ip netns | grep qdhcp
qdhcp-01cce415-8e2b-4096-92b5-7acc67163d79

root@netnode:/# ip netns exec qdhcp-01cce415-8e2b-4096-92b5-7acc67163d79 ip -4 a
14: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 
    inet 127.0.0.1/8 scope host lo
23: tapc2ac4ed4-f3: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    inet 172.17.17.1/24 brd 172.17.17.255 scope global tapc2ac4ed4-f3
    inet 169.254.169.254/16 brd 169.254.255.255 scope global tapc2ac4ed4-f3

root@netnode:/# ip netns exec qdhcp-01cce415-8e2b-4096-92b5-7acc67163d79 route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
169.254.0.0     0.0.0.0         255.255.0.0     U     0      0        0 tapc2ac4ed4-f3
172.17.17.0     0.0.0.0         255.255.255.0   U     0      0        0 tapc2ac4ed4-f3
    
root@netnode:/# ip netns exec qdhcp-01cce415-8e2b-4096-92b5-7acc67163d79 netstat -4 -anpt
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      27409/python    
tcp        0      0 172.17.17.1:53          0.0.0.0:*               LISTEN      27348/dnsmasq   
tcp        0      0 169.254.169.254:53      0.0.0.0:*               LISTEN      27348/dnsmasq   

root@netnode:/# ps -f --pid 27409 | fold -s -w 82
UID        PID  PPID  C STIME TTY          TIME CMD
root     27409     1  0 17:47 ?        00:00:00 python 
/usr/bin/quantum-ns-metadata-proxy 
--pid_file=/var/lib/quantum/external/pids/01cce415-8e2b-4096-92b5-7acc67163d79.pid
--network_id=01cce415-8e2b-4096-92b5-7acc67163d79 --state_path=/var/lib/quantum 
--metadata_port=80 --debug --verbose 
--log-file=quantum-ns-metadata-proxy01cce415-8e2b-4096-92b5-7acc67163d79.log 
--log-dir=/var/log/quantum


And the dnsmasq options file includes a classless static route for the metadata IP:


root@netnode:/# ps -f --pid 27348 | fold -s -w 82
UID        PID  PPID  C STIME TTY          TIME CMD
nobody   27348     1  0 17:47 ?        00:00:00 dnsmasq --no-hosts --no-resolv 
--strict-order --bind-interfaces --interface=tapc2ac4ed4-f3 --except-interface=lo 
--pid-file=/var/lib/quantum/dhcp/01cce415-8e2b-4096-92b5-7acc67163d79/pid 
--dhcp-hostsfile=/var/lib/quantum/dhcp/01cce415-8e2b-4096-92b5-7acc67163d79/host 
--dhcp-optsfile=/var/lib/quantum/dhcp/01cce415-8e2b-4096-92b5-7acc67163d79/opts 
--dhcp-script=/usr/bin/quantum-dhcp-agent-dnsmasq-lease-update --leasefile-ro 
--dhcp-range=set:tag0,172.17.17.0,static,120s 
--conf-file=/etc/quantum/dnsmasq.conf --domain=openstacklocal


root@netnode:/# cat /var/lib/quantum/dhcp/01cce415-8e2b-4096-92b5-7acc67163d79/opts 
tag:tag0,option:classless-static-route,169.254.169.254/32,172.17.17.1


Now when DHCP clients request DHCP option 121, they will receive this static route. If the instance is Linux, the DHCP client might use iproute to apply it like this:


ip route add 169.254.169.254/32 via 172.17.17.1


Note: the Cirros test image does not request option 121 - see here.

With this static route in place on the instance, requests to 169.254.169.254:80 will go to the metadata nameserver proxy in the DHCP namespace on the network node. The namespace proxy adds HTTP headers X-Quantum-Network-ID with the network ID and X-Forwarded-For with the instance's IP to the request, and proxies it to the metdata agent via a Unix domain socket. After that it works in almost the same as it did with the Quantum router - see here for more detail. The metadata agent queries the Quantum server for the instance ID that corresponds to the source IP and network ID. Then it adds X-Instance-ID and X-Instance-ID-Signature to the request and proxies it to the Nova metadata service.