Can't access Apache outside local subnet

[dingleberry78]

Hi everyone,

I just semi-successfully upgraded from RedHat 8 to FC2, and I've run into a strange problem. I have a number of servers running on my box including Apache, CVS, Webmin, and SSH. All of these worked before the upgrade, and they mostly continue to work after the upgrade. The problem is that I can't access Apache from outside the local subnet -- I'm not sure whether to blame Apache or some other component at this point though, so the Networking forum seemed most appropriate.

The details:
I have this FC2 box and a Windows box on the local subnet, with a Linksys Router at the perimeter. The router forwards ports 80 and 22 through to the FC2 box -- I should note that both HTTP and SSH connections worked perfectly through port forwarding before the upgrade.

Now that the upgrade is complete however, I can access all server/services normally from the Windows box and from the console, but Apache doesn't respond when I try to access it externally (no errors to indicate the port is closed -- it just doesn't respond).

Just for kicks I forwarded port 10000 through and Webmin works fine from the outside if I do; also SSH continues to work fine from the inside and outside.

I have ensured that iptables isn't running (/etc/init.d/iptables stop) and I've looked through httpd.conf many times (it's pretty much the default config file with the hostname added in to prevent the warnings about that). When I access Apache from the local net the requests are logged as you would expect, but when I try to access it from outside, the Apache logs remain unchanged (even at "debug" verbosity).

I've also used tcpdump to ensure that the router isn't somehow preventing things from working and I do see packets on port 80 on the FC2 box when I initiate an HTTP request from outside so that would seem to rule that out.

I'm out of ideas on this, so if anyone can tell me why Apache is ignoring external requests, I'd be most grateful.

[Picomp314]

it *could* be a problem with SELinux
i would have suggested to look at httpd.conf as well

post the output of
netstat -na | grep tcp
netstat -na | grep udp

[dingleberry78]

I'm afraid I don't know much about SELinux -- Anaconda didn't mention anything about it during the upgrade and the docs I've read indicate that it's disabled by default...?

Here are the dumps you requested:

netstat -na|grep tcp:
tcp 0 0 0.0.0.0:32768 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:2401 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:32842 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:3306 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:139 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:10000 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:631 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:445 0.0.0.0:* LISTEN
tcp 0 0 192.168.1.200:445 192.168.1.100:3047 ESTABLISHED
tcp 0 48 192.168.1.200:22 192.168.1.100:3564 ESTABLISHED

netstat -na|grep udp:
udp 0 0 0.0.0.0:32768 0.0.0.0:*
udp 0 0 127.0.0.1:32774 0.0.0.0:*
udp 0 0 192.168.1.200:137 0.0.0.0:*
udp 0 0 0.0.0.0:137 0.0.0.0:*
udp 0 0 192.168.1.200:138 0.0.0.0:*
udp 0 0 0.0.0.0:138 0.0.0.0:*
udp 0 0 0.0.0.0:10000 0.0.0.0:*
udp 0 0 0.0.0.0:111 0.0.0.0:*
udp 0 0 0.0.0.0:884 0.0.0.0:*
udp 0 0 0.0.0.0:631 0.0.0.0:*

Do these shed any light...?

[Picomp314]

everything looks fine...
you might try configuring httpd.conf so that it accepts addresses over your lan IP instead of 0.0.0.0 (all addresses)
AFAIK SELinux is ENABLED by default

try this
edit /boot/grub/grub.conf
append selinux=0 to your current kernel and save

i.e.

title Fedora Core (2.6.7 Super Fast Ownage)
root (hd0,1)
kernel /vmlinuz-2.6.7 ro root=/dev/hda3 selinux=0 vga=791 splash=silent
initrd /initrd-2.6.7.img

[merith]

can you ping the box?

Also have you tried to telnet to the box?

in linux telnet address port
i.e telnet 111.111.111.111 3333

if you get a connection established(it says its connected ok) it means that the server has been found and is listening for requests. To close a sucessful conncetion hit ctrl ]

[dingleberry78]

you might try configuring httpd.conf so that it accepts addresses over your lan IP instead of 0.0.0.0 (all addresses)

AFAIK SELinux is ENABLED by default
append selinux=0 to your current kernel and save


I modified the Listen directive in httpd.conf to:
Listen 192.168.1.200:80

I also added the selinux=0 switch to the kernel in Grub.

Unfortunately, neither of these measures has helped. :confused:

[dingleberry78]

can you ping the box?

Also have you tried to telnet to the box?


I can ping the box fine from within the local net, and cannot from outside, but that is to be expected as the Linksys router blocks ICMP packets.

I have also tried to telnet to port 80, which works correctly from the local net, and fails to connect at all from outside (no error message, just hangs forever).

:(

[Picomp314]

it has to be a problem with httpd.conf
i believe there is some line which speaks of access control, and trusted IPs, i believe right now it is not set properly, and is thus denying access to anyone outside the local subnet

[dingleberry78]

it has to be a problem with httpd.conf
i believe there is some line which speaks of access control, and trusted IPs, i believe right now it is not set properly, and is thus denying access to anyone outside the local subnet

I believe you're referring to the following lines in httpd.conf:

<Directory "/var/www/html">
...
Order allow,deny
Allow from all
...
</Directory>

However, the above settings shouldn't place any restrictions on access (AFAIK).

I have posted the contents of my /etc/httpd/conf and conf.d directories (minus the SSL stuff) in case anyone can see anything I'm missing:

http://www.cs.uleth.ca/~schelld4/httpd

:confused:

[Picomp314]

man this is driving me nuts; i looked through your httpd.conf, and all *seems* to be in order
perhaps i could give you my working copy in hopes that there is some stupid typo on one of the 1061 lines in the thing

[dingleberry78]

As it turns out, I've already tried that too -- earlier this afternoon I copied httpd.conf and the contents of conf.d from a working FC2 box at work to the problem FC2 box and it didn't make a bit of difference.

Given that none of this ever shows up in the Apache logs, and there seems to be no evidence that I'm even able to establish a TCP connection on port 80 from outside the local net, I'm not convinced that this is an Apache issue. It seems suspiciously like there's something else getting in the way (though I certainly can't imagine what).

I'm starting to wonder if the standard "Windows solution" is in order -- that is, re-install the OS from scratch. I don't cherish the time and effort involved in going this route, but I don't know what else to try. There is a catch with this plan though -- I have a bunch of large files stored on that box that I have no way of backing up or storing elsewhere right at this moment, so is it possible to do a "clean" install of FC2 *without* wiping out all my existing partitions? (Ideas or solutions to the original problem are still welcome of course :))

[superbnerd]

I read you posted httpd.conf and I think I have spoted the problem, but you should know I am not a web server expert :)
you have #
# Listen: Allows you to bind Apache to specific IP addresses and/or
# ports, in addition to the default. See also the <VirtualHost>
# directive.
#
# Change this to Listen on specific IP addresses as shown below to
# prevent Apache from glomming onto all bound IP addresses (0.0.0.0)
#
#Listen 12.34.56.78:80
Listen 192.168.1.200:80
and #
# If your host doesn't have a registered DNS name, enter its IP address here.
# You will have to access it by its address anyway, and this will make
# redirections work in a sensible way.
#
#ServerName new.host.name:80
ServerName localhost You'll notice that each time you're binding the server to addresses that are only accessible from within you subnet:
1) your ip address is a private address it should be the address of your router since the router will forward the port to apache. or just put something like *:80
2) your server name is localhost. this is also only accesible from within your lan. Do you have a domain name if not, buy one or get one from dyndns (http://www.dyndns.org/) and use that.

[dingleberry78]

1) your ip address is a private address it should be the address of your router since the router will forward the port to apache. or just put something like *:80

I changed this to "Listen 80" which should clear up any questions on whether it is listening on the proper interface. My sense though is that the IP address specified here just indicates which IP address on the local machine that Apache will bind to -- thus putting the router's IP here would be incorrect.


2) your server name is localhost. this is also only accesible from within your lan. Do you have a domain name if not, buy one or get one from dyndns (http://www.dyndns.org/) and use that.
I do have a dyndns DNS name, so I changed the Servername directive to "ServerName my.hostname.org:80". Unfortunately, these changes didn't help.

Incidentally, when I tried to restart Apache after these changes I got an error message about "No space left on device". Using df confirmed that I do indeed have free space on that partition (although only 8%). I used the suggestion in the following FAQ to resolve the problem, but just thought I'd mention it in case it's somehow relevant to the whole situation (http://faq.otrs.org/otrs/faq.pl?Action=&ID=4).

[superbnerd]

your right. as I said web servers are not my spaecialty, but about #1, you changed it to "Listen 80". I am not sure that is correct. I apache seems to want an ip and port, so if you only have one ip for that server than try Listen *:80

[dingleberry78]

your right. as I said web servers are not my spaecialty, but about #1, you changed it to "Listen 80". I am not sure that is correct. I apache seems to want an ip and port, so if you only have one ip for that server than try

I tried "Listen *:80" but no luck there either. :confused:

[Picomp314]

give the output of iptables -L

[dingleberry78]

The results of iptables -L:

Chain INPUT (policy ACCEPT)
target prot opt source destination

Chain FORWARD (policy ACCEPT)
target prot opt source destination

Chain OUTPUT (policy ACCEPT)
target prot opt source destination

[dingleberry78]

I just got thinking that maybe this has something to do with routing (I have no idea really, but it's a thought).

For reference, the FC2 box is at 192.168.1.200, the Linksys router is at 192.168.1.1, and the Windows box on my local net is at 192.168.1.100.
isp_dns_server is the ip of my ISP's dns server (shocking I know ;))
remote_host is a Solaris box that I'm SSH'd to and am using Lynx to attempt an HTTP connection back to Apache.

Here are the contents of `route`:

Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
192.168.1.0 * 255.255.255.0 U 0 0 0 eth0
169.254.0.0 * 255.255.0.0 U 0 0 0 eth0
default 192.168.1.1 0.0.0.0 UG 0 0 0 eth0

and `route -C`:
192.168.1.200 isp_dns_server 192.168.1.1 0 0 12 eth0
192.168.1.200 192.168.1.100 192.168.1.100 0 1 0 eth0
192.168.1.100 192.168.1.200 192.168.1.200 il 0 0 178 lo
192.168.1.1 192.168.1.255 192.168.1.255 ibl 0 0 15 lo
192.168.1.200 isp_dns_server 192.168.1.1 0 0 12 eth0
192.168.1.200 192.168.1.100 192.168.1.100 0 1 2 eth0
192.168.1.200 192.168.1.255 192.168.1.255 bl 0 0 1 eth0
192.168.1.100 192.168.1.255 192.168.1.255 ibl 0 0 4 lo
remote_host 192.168.1.200 192.168.1.200 l 0 0 5 lo
192.168.1.200 remote_host 192.168.1.1 0 0 10 eth0
isp_dns_server 192.168.1.200 192.168.1.200 l 0 0 17 lo

Anyone see anything out of the ordinary here?

[kronck]

I am having the same issue i believe, but one thing you might be able to help me out with.....

One thing i am not clear on with your setup is how the system you are trying to access the apache server from is on a different subnet....it appears that the Windows machine is on the same subnet, not a seperate subnet. if the machine is on a different subnet make sure the default route (gateway) is compatiable. for example

Server IP = 192.168.111.1
Server Gateway = 192.168.111.250

Workstation IP = 192.168.114.1
Workstation Gateway = 192.168.114.250

If the workstation Gateway is not set at all or set to anything but .250 it will not see the Server. This could be why you cannot ping, telnet, ssh, ect to the server.

If any of that was confusing, just make sure the gateway on the system that cannot access the server is 192.168.(something).1

With my setup i can telnet, ssh, ping ect, but apache will NOT work across a routed subnet. i have verified that its not a routing issue by setting up another apache server on the same server subnet, and running apache2 it works just fine. The older server running Apache1.3 is not running fine however. Im certian that Apache1.3 is able to work just fine, but im not seeing what the issue is. httpd.conf looks fine...at least concerning access rules.

/bump.

[kronck]

After a bit of troubleshooting, it dawned on me that the system might not be using iptables at all. Ipchains however was running and cuasing the issue. after stoping that service everything works just fine.

Might check ipchains, not just iptables.