Issues with static DNS hosts and nginx reverse proxy

[napstersquest]

I have multiple apps like nextcloud, pihole, syncthing, photoprism, etc set up on TrueNAS Scale machine, along with Nginx Proxy Manager.
This is how each proxy host in NPM looks like:

SOURCE	DESTINATION	SSL	ACCESS	STATUS
	nextcloud.mydomain.com Created: 21st September 2022	http://<truenasMachineIP>:<nextcloudPort>	Let's Encrypt	Public	Online

All is well and good, https works, I can access the apps through URL like nextcloud.mydomain.com

Now I realized that all the traffic from client to server, even after being in same network, was travelling the internet and back as I use Cloudflare as my DNS. So around 120ms ping for each query, and some file uploads would eat up the internet bandwidth.

So I set up some static DNS hosts in the client's hosts file (Windows 11)
Like this:

# Copyright (c) 1993-2009 Microsoft Corp.
#
# This is a sample HOSTS file used by Microsoft TCP/IP for Windows.
#
# This file contains the mappings of IP addresses to host names. Each
# entry should be kept on an individual line. The IP address should
# be placed in the first column followed by the corresponding host name.
# The IP address and the host name should be separated by at least one
# space.
#
192.168.X.X nextcloud.mydomain.com

So using tracert, I did find that the packet was not travelling outside LAN, which was the ideal result.

But now the apps are giving me "Connection Refused" errors on their URL. DNS resolution is fine, because if I try to access using domain:IP, it works (for example nextcloud.mydomain.com:<port>)

 

[lockhrt999]

If you are using pi hole then you might want to use its Local DNS. Check 'Local DNS' inside pi hole admin panel. Hardcode your domain to the IP.

But now the apps are giving me "Connection Refused" errors on their URL. DNS resolution is fine, because if I try to access using domain:IP, it works (for example nextcloud.mydomain.com:<port>)

Your HTTPS certificates are probably linked to your public IP and not the local IP.

 

[napstersquest]

If you are using pi hole then you might want to use its Local DNS. Check 'Local DNS' inside pi hole admin panel. Hardcode your domain to the IP.

That's what I tried first. Same thing.

• If I do not use my pihole DNS on client, and only use the hosts file method for static DNS hosts
• Or if I use local dns of pihole and use pihole as dns for the client

In both cases the issue is same.

Your HTTPS certificates are probably linked to your public IP and not the local IP.

I am not sure about this, are the certificates linked to the IP? I mean in both cases I am accessing through the same client and end PC is also same, so keys used for the SSL connection will still be the same.

 

[jinx]

That's what I tried first. Same thing.

DNS is working as intended don't think it's an issue.

I am not sure about this, are the certificates linked to the IP?

Nope, letsencrypt doesn't issue certs to IPs (so does many other issuers)


It's either something with your firewall rules(listen port/interface) or nginx config

Use telnet to see if it's allowing connections (command: telnet <host> <port>) and if it doesn't make sure you're listening on local IP/interface/port.

If that isn't the issue, post your nginx.conf here.

 

[napstersquest]

Use telnet to see if it's allowing connections (command: telnet <host> <port>) and if it doesn't make sure you're listening on local IP/interface/port.

telnet <url> <80 or 443> fails

telnet <lan IP> <nextcloud port> connects, replies with message Your browser sent a request that this server could not understand which is understandable I guess (for nextcloud)

If that isn't the issue, post your nginx.conf here.

I am using nginx proxy manager through GUI (deployed as a docker container).
Here is the docker-compose I used:

version: '3'
services:
  app:
    image: 'jc21/nginx-proxy-manager:latest'
    restart: unless-stopped
    ports:
      - 'XX80:80'
      - 'XX81:81'
      - 'XX43:443'
    volumes:
      - ./data:/data
      - ./letsencrypt:/etc/letsencrypt

For each of the apps, have added a proxy host in NPM's dashboard with following settings,
Access list => Publicly available
Block common exploits => On/Off - has no effect
Websockets support => On/Off - has no effect
Force SSL => Enabled
HTTP/2 support => Enabled
HSTS => disabled

if I try to access using domain:IP, it works (for example nextcloud.mydomain.com:<port>)

Also this ^

Few more things to add:

Ping from inside the NPM container to 192.168.X.X which is the machine IP works fine.
So I am assuming NPM can route the traffic no problem, and also that without static DNS records everything was working fine, but it was as if I am not local so there would be latency issues + would eat up the internet bandwidth.

 

[jinx]

telnet <url> <80 or 443> fails

Does the same happen with local IP (instead of the URL)?

Also, can you share how your network looks like? Does it have multiple interfaces? Does it directly get the public IP or is it port forwarded? (Output of "ip a" or ifconfig on the trunas machine would be helpful)

 

[napstersquest]

Does the same happen with local IP (instead of the URL)?

Yes. LAN IP and 80/443 with telnet fail as well.

Also, can you share how your network looks like? Does it have multiple interfaces? Does it directly get the public IP or is it port forwarded?

Static IP is on my TP Link router. 80 and 443 ports are forwarded for the respective ports of LAN IP of server on the router.

But I guess that would not be an issue as I can access everything over the Internet.

 

[b.life]

Yes. LAN IP and 80/443 with telnet fail as well.

If my understanding is correct local pc --> nginx --> server is a problem(no matter through domain or IP) but not local pc --> internet --> router --> nginx --> server, so the packets from router is accepted but not from lan. So I would suggest check the firewall rules and also the network interface(listining) of nginx dosn't seems to a problem as it is already on lan IP(If not then that needs to be checked too)

 

[jinx]

ports: - 'XX80:80' - 'XX81:81' - 'XX43:443'

Is xx43 same as 443? Cuz it's host: container port so if not try that particular port with the IP/host.

 

[napstersquest]

Is xx43 same as 443? Cuz it's host: container port so if not try that particular port with the IP/host.

This must be it.

local pc --> nginx --> server is a problem(no matter through domain or IP) but not local pc --> internet --> router --> nginx --> server,

Yes exactly.

If I try https://app.mydomain.com , and app.mydomain.com translates to LAN IP of the machine, it fails.
In the same scenario, if app.mydomain.com translates to my WAN IP (Static IP purchased from Airtel), it is successful.

With current port forwarding, as @jinx mentioned, 80 and 443 ports of the NPM docker container are not exposed as 80 and 443 of the NAS.
On TrueNAS Scale, by default apps can only publish ports 9000 and up, hence I was using ports eg 9080:80 and 9043:443, and then forwarding 9080,9043 of NAS to 80, 443 of WAN IP via the port forwarding on router.

In order for the reverse proxy to work, the IP we are trying to access via app.mydomain.com should have its 80/443 ports pointing to 80/443 ports of NPM container. This is fundamental of the Nginx Proxy Manager, and I did not catch it.

Fortunately, I can create a custom docker image with TrueNAS Scale and deploy the NPM that way, exposing 80, 443 ports and not the 9080, 9043 ports on the NAS. Will confirm if it works.

 

[b.life]

Good Morning,
Did you solve the issue? If you did what was the issue? I am curious to know(never used nginx proxy manager and whats to try it now) so can you please update?

Regards

 

[napstersquest]

Works perfectly!

Did you solve the issue? If you did what was the issue? I am curious to know(never used nginx proxy manager and whats to try it now) so can you please update?

Yes I did end up solving the issue.

How NPM Works:

If you hit a URL myapp.mydomain.com which points to an IP let's say 123.145.67.89 (could be your static IP you purchased) -> And that IP's 80/443 ports point to NPM container's 80/443 ports,

NPM will route all traffic on the port (APP Port - let's say 9880) we specify that is accessible to NPM container on 192.168.X.Y (or whatever LAN IP which is accessible by NPM Container) to the myapp.mydomain.com (Https or Http), so we can access the app as myapp.mydomain.com instead of <IP>:<Port> with added benefit of SSL certificate if we want.


What was the problem:

The NPM container's internal 80/443 ports were exposed as 9080/9043 on the local Server. My router then was forwarding them as 80/443 to the internet.
If I tried accessing myapp.mydomain.com via internet -> Client could see NPM listening on 80/443 (NPM container's 80/443 -> Server's 9080/9043 -> Router's 80/443).

But, when I created a static DNS entry for 192.168.X.Y on my client or the Pihole DNS, it would look for 192.168.X.Y's 80/443 ports and see that there is no listener (Remember 9080 and 9043 were exposed at that machine instead of 80 and 443) so would give the error "connection refused".
I now exposed NPM container's 80/443 as 80/443 of Server machine, and router now forwards 80/443 of server to the internet.

Edit:

Thanks @lockhrt999 @jinx and @b.life for your help! I love this forum!

 

[b.life]

Hi,
Glad it got solved and thanks for the explanation regarding Nginx proxy manager, I really appreciate that

Regards