On the way to OpenStack

When I tried configuring openstack on Fedora 22 VM using devstack. Everything went well, I could ssh into the VM, but when tried to open Horizon from physical machine could not open it, while the same was accessible on VM browser on the same URL.

Initially I doubted SELinux as the cause of the problem. So changed SELinux to be permissive and then went on to disable it completely, but still horizon was not accessible from outside the virtual machine. So SELinux was not the culprit.

Meanwhile Openstack Summit is happening in Tokyo and all the talks and session videos are available on youtube. I saw a hands-on session video called Neutron Network Know-How – A Hands-On Workshop for Solving Neutron Nightmares, where the speaker had deep knowledge about neutron and how to troubleshoot it, he emphasized on the usage of tcpdump, importance of the basic concepts like OVS, Linux Bridge, IPtables, etc to name few. So I started with tcpdump.

First checked VM interfaces

[fedora@localhost ~]$ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:54:00:33:d2:8e brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.174/24 brd 192.168.122.255 scope global dynamic eth0
       valid_lft 2674sec preferred_lft 2674sec
    inet6 fe80::5054:ff:fe33:d28e/64 scope link 
       valid_lft forever preferred_lft forever

Inspired by the talk tried tcpdump on the port eth0 through which the guest VM is connected to physical machine.

[fedora@localhost ~]$ sudo tcpdump -n -i eth0

Then tried accessing Horizon from physical machine

# curl http://192.168.122.174/dashboard
curl: (7) Failed to connect to 192.168.122.174 port 80: No route to host

With no success of getting access to Horizon. On VM where tcpdump was running, saw this particular packet entry

[fedora@localhost ~]$ sudo tcpdump -n -i eth0
22:53:52.014835 IP 192.168.122.174 > 192.168.122.1: ICMP host 192.168.122.174 unreachable - admin prohibited, length 68

This only helped in knowing that this packet is being filtered. Then thought of looking into iptables as I was totally unaware of it before. Watched these videos about iptables and found out it is similar to access control lists on the Cisco Routers. So tried looking at iptable on the VM.

So took a look at iptables on VM

# iptables -L -v -n 

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     udp  --  virbr0 *       0.0.0.0/0            0.0.0.0/0            udp dpt:53
    0     0 ACCEPT     tcp  --  virbr0 *       0.0.0.0/0            0.0.0.0/0            tcp dpt:53
    0     0 ACCEPT     udp  --  virbr0 *       0.0.0.0/0            0.0.0.0/0            udp dpt:67
    0     0 ACCEPT     tcp  --  virbr0 *       0.0.0.0/0            0.0.0.0/0            tcp dpt:67
41524 2527K ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED
    1    84 ACCEPT     icmp --  *      *       0.0.0.0/0            0.0.0.0/0           
    4   260 ACCEPT     all  --  lo     *       0.0.0.0/0            0.0.0.0/0           
    4   240 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            state NEW tcp dpt:22
   82  6695 REJECT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            reject-with icmp-host-prohibited

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     all  --  *      virbr0  0.0.0.0/0            192.168.124.0/24     ctstate RELATED,ESTABLISHED
    0     0 ACCEPT     all  --  virbr0 *       192.168.124.0/24     0.0.0.0/0           
    0     0 ACCEPT     all  --  virbr0 virbr0  0.0.0.0/0            0.0.0.0/0           
    0     0 REJECT     all  --  *      virbr0  0.0.0.0/0            0.0.0.0/0            reject-with icmp-port-unreachable
    0     0 REJECT     all  --  virbr0 *       0.0.0.0/0            0.0.0.0/0            reject-with icmp-port-unreachable
    0     0 REJECT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            reject-with icmp-host-prohibited

Chain OUTPUT (policy ACCEPT 63 packets, 10497 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     udp  --  *      virbr0  0.0.0.0/0            0.0.0.0/0            udp dpt:68

Looking at this in the first attempt doesn’t help. So tried keeping track of the packet increment on this particular rule which is the last rule in the INPUT chain.

   82  6695 REJECT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            reject-with icmp-host-prohibited

The packet count increased everytime I unsuccessfully tried accessing the Horizon from physical machine. This shows that packet is being dropped at this rule. So I needed some rule which is permissible and needed it in table before this dropping rule, so added a rule which allowed everything, as following.

# iptables -I INPUT --in-interface eth0 -j ACCEPT

Above rule means in iptables on INPUT chain from interface eth0 ACCEPT everything.

# iptables -L -v -n
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
   21  1468 ACCEPT     all  --  eth0   *       0.0.0.0/0            0.0.0.0/0           
    0     0 ACCEPT     udp  --  virbr0 *       0.0.0.0/0            0.0.0.0/0            udp dpt:53
    0     0 ACCEPT     tcp  --  virbr0 *       0.0.0.0/0            0.0.0.0/0            tcp dpt:53
    0     0 ACCEPT     udp  --  virbr0 *       0.0.0.0/0            0.0.0.0/0            udp dpt:67
    0     0 ACCEPT     tcp  --  virbr0 *       0.0.0.0/0            0.0.0.0/0            tcp dpt:67
41647 2537K ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED
    1    84 ACCEPT     icmp --  *      *       0.0.0.0/0            0.0.0.0/0           
    4   260 ACCEPT     all  --  lo     *       0.0.0.0/0            0.0.0.0/0           
    4   240 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            state NEW tcp dpt:22
   82  6695 REJECT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            reject-with icmp-host-prohibited

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     all  --  *      virbr0  0.0.0.0/0            192.168.124.0/24     ctstate RELATED,ESTABLISHED
    0     0 ACCEPT     all  --  virbr0 *       192.168.124.0/24     0.0.0.0/0           
    0     0 ACCEPT     all  --  virbr0 virbr0  0.0.0.0/0            0.0.0.0/0           
    0     0 REJECT     all  --  *      virbr0  0.0.0.0/0            0.0.0.0/0            reject-with icmp-port-unreachable
    0     0 REJECT     all  --  virbr0 *       0.0.0.0/0            0.0.0.0/0            reject-with icmp-port-unreachable
    0     0 REJECT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            reject-with icmp-host-prohibited

Chain OUTPUT (policy ACCEPT 11 packets, 1256 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     udp  --  *      virbr0  0.0.0.0/0            0.0.0.0/0            udp dpt:68

You can see new rule at the top in INPUT chain that is

   21  1468 ACCEPT     all  --  eth0   *       0.0.0.0/0            0.0.0.0/0

Now when tried accessing Horizon from the physical machine it worked, also the packet count in this rule started increasing, since this rule allowed all the traffic from the network connected to interface eth0.

References

iptables

Neutron Network Know-How

Advertisements

3 thoughts on “On the way to OpenStack

  1. Hi Suraj !

    Thanks for this blog ! Just FYI … I started with Openstack deployment on Debian based operating system like Ubuntu using Devstack as its more specific for the same . For RPM based operating systems Openstack installation should be done with Packstack .

    Liked by 1 person

    1. Yes, if you are want to do development then Devstack is recommended to use. And if you would like to do POC in matter of minutes then try Packstack or RDO.

      Liked by 1 person

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s