How to diagnose and solve Nodegrid devices prevented from connecting to ZPE Cloud due SSL/TLS deep inspection configuration at a site's Firewall

What is the SSL/TLS deep Inspection Configuration Running at the Site's Firewall

An on-site SSL/TLS deep inspection firewall configuration is a security setup where the firewall decrypts encrypted traffic to inspect it for threats, then re-encrypts it before forwarding it to its destination. This is done by having the firewall act as a "man-in-the-middle," creating two separate SSL connections: one with the client and one with the destination server. This configuration is usually set on firewall servers linked to routers in the corporate networks.
For this to work seamlessly, all user devices on the network must trust the firewall's certificate, which is typically achieved by pre-installing the certificate on company-owned devices. This is many times done by an application installed from IT running on the corporate user devices, but not so seamlessly for IT devices, since they must be managed by IT team.

The Symptom of the Problem

On ZPE Cloud, the status of the device remains as "Never Connected" or "Offline".
On NodeGrid Operating System, when navigating to Tracking :: ZPE Cloud::Device Information page, it displays the unit status as "Disconnected - Requesting Pairing".

How to Identify the Problem

Below you will find 3 scenarios that can be used to identify if the problem is occurring.

Scenario 1.

As an administrator check the device status to verify if the Status is "Offline" or "Never Connected".

Scenario 2

As an administrator navigate to Tracking::ZPE Cloud::Device Information and check if the Status is "Disconnected - Requesting Pairing".

Scenario 3.

Check to see if the system returns a self-signed certificate that was added to the chain of certificates after using openssl s_client -showcerts below.
For ZPE Cloud US:

openssl s_client -showcerts -servername api.astarte.zpecloud.com -connect api.astarte.zpecloud.com:443 < /dev/null;

For ZPE Cloud EU:

openssl s_client -showcerts -servername api.astarte.zpecloud.eu -connect api.astarte.zpecloud.eu:443 < /dev/null;

Examples of possible outputs:
Example 1:
The output of the certificate issuer is not included in the CA path (/etc/ssl/certs/< ca>.pem) and CA file (/etc/ssl/certs/ca-certificates.crt):

CONNECTED(00000003)
depth=1 C = US, O = ZScaler, CN = zscaler.net
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 C = US, ST = California, L = Fremont, O = "ZPE Systems, Inc.", CN = astarte.zpecloud.com
verify return:1

Example 2:
The output is a self-signed certificate in the certificate chain:

CONNECTED(00000003)
depth=1 C = US, O = ZScaler, CN = zscaler.net
verify error:num=19:self-signed certificate in certificate chain
verify return:1
depth=0 C = US, ST = California, L = Fremont, O = "ZPE Systems, Inc.", CN = astarte.zpecloud.com
verify return:1

Example 3:
The output of a working certificate:

depth=3 C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert Global Root G2
verify return:1
depth=2 C = US, O = DigiCert Inc, CN = DigiCert Global G2 TLS RSA SHA256 2020 CA1
verify return:1
depth=1 C = US, O = ZScaler, CN = zscaler.net
verify return:1
depth=0 C = US, ST = California, L = Fremont, O = "ZPE Systems, Inc.", CN = *.astarte.zpecloud.com
verify return:1

Fix for the Problem

Option 1

Exclude ZPE Cloud domain/IPs from SSL traffic/deep inspection:

1. Login into the Firewall service platform
2. Go to SSL deep inspection/Traffic inspection
3. Disable the SSL inspection for ZPE Cloud domain or NG OS IP/mac
4. Login as a customer admin to ZPE Cloud
5. Go to Devices::Enrolled
6. Check to see if the unit is online after 1-3 minutes (Depends on the network velocity, GSM is slower)

Option 2

Add SSL traffic/deep inspection certificates signed by a known CA:

1. Login into the Firewall service platform
2. Go to SSL deep inspection/Traffic inspection
3. Change the self-signed certificates to Let’s encrypt signed certificates
4. Login as a customer admin to ZPE Cloud
5. Go to Devices::Enrolled
6. Check to see if the unit is online after 1-3 minutes (Depends on the network velocity, GSM is slower)

Option 3

Move your device to another network without SSL traffic/deep inspection:

1. Unplug the ethernet cable and connect another ethernet cable to a new network or make your GSM a primary connection to the NodeGrid unit
2. Login as a customer admin to ZPE Cloud
3. Go to Devices::Enrolled
4. Check to see if the unit is online after 1-3 minutes (Depends on the network velocity, GSM is slower)

Option 4

Add the SSL deep inspection self-signed certificate to the System Administrator trust store:

1. Download the self-signed certificate and install if on the unit
   1. Via SCP:
      1. scp mycertificate.crt admin@192.168.10.48:/home/admin/
   2. Via File Manager:
      1. Login as admin in NG GUI
      2. Go to Access::Table
      3. Click to open File Manager
         File Manager is displayed
      4. Click on admin_home directory on File Manager
      5. Click to upload
      6. Select the certificate file from the local directory
      7. Click to confirm the upload
      8. Check if the certificate is listed on File Manager
2. Login as root in NG Shell
3. Move the certificate to the System Administrator trust store, under ZPE Cloud folder:
   1. mv /home/admin/mycertificate.crt /usr/local/share/ca-certificates/zpe-cloud-ca-certificates/;
4. Update the trust store. Execute:
   1. update-ca-certificates -f
5. Confirm that the certificate is now trusted using the prompt below:
   1. for ZPE Cloud US: openssl s_client -showcerts -servername api.astarte.zpecloud.com -connect api.astarte.zpecloud.com:443 < /dev/null;
      for ZPE Cloud EU: openssl s_client -showcerts -servername api.astarte.zpecloud.eu -connect api.astarte.zpecloud.eu:443 < /dev/null;
   2. Check if the output displays no error to trust certificate chain;
6. Login as admin in NG GUI
7. Be sure Enable ZPE Cloud is checked under Security::Services
   1. You may Disable, Save, Enable, Save the ‘Enable ZPE Cloud’ configuration to accelerate the time for the unit to show Online;
8. See after 1-3 minutes the unit shows Online (Depends on the network velocity - GSM is slower -);

Acronyms