How to Troubleshoot ZPE Cloud Environment

About this Document

This guide is designed to help you identify, understand, and resolve common issues that may arise while using ZPE Cloud. Whether you are experiencing connectivity problems, encountering errors during software updates, or facing challenges with feature performance, this guide provides step-by-step instructions and recommendations to quickly restore optimal functionality.

Troubleshooting ZPE Cloud Environment

This section provides information on essential commands used in troubleshooting environment for ZPE Cloud services on Nodegrid Devices. The most common troubleshooting involves similar checks, as they typically relate to disk, network, date or time settings, and other environmental factors. However, each section may have specific checks relevant to that area. It includes steps to check the following:

• Enrollment status

• Device status

  • Online

  • Offline

  • Never connected

  • Failover (Pending)

• Remote access

• ZTP (Zero Touch Provisioning)

  • Profiles

  • Backup

Basic Environment Checks

These are the essential system checks that must be functioning properly before troubleshooting the applications. The listed commands are run on diagnostic data, allowing you to identify and address minor environmental issues. If these commands fail, it indicates that the environment needs to be fixed before proceeding.

Execute the following with api.zpecloud.com and google.com.

1. Ensure the service is configured:

   1. sudo su - admin -c cli <<< "show /settings/zpe_cloud"

   2. sudo su - admin -c cli <<< "show /settings/zpe_cloud enable_zpe_cloud"

   3. sudo su - admin -c cli <<< "show /settings/zpe_cloud enable_remote_access"

2. Ensure date and time are accurate:

   1. chronyc tracking - provides detailed information about the system time synchronization.

   2. chronyc ntpdata - displays information about the network time protocol (NTP).

   3. date - shows the current system date and time.

3. Ensure network access is up and running.

   NOTE

   DNS and TCP are required, ping as network metric are optional.

   1. nmcli c

   2. nslookup api.zpecloud.com;

   3. ping api.zpecloud.com;

      1. ping -4 -I wwan0 api.zpecloud.com;

      2. ping -4 -I eth0 api.zpecloud.com;

      3. ping -6 -I wwan0 api.zpecloud.com;

      4. ping -6 -I eth0 api.zpecloud.com;

   4. telnet api.zpecloud.com 443;

   5. traceroute api.zpecloud.com;

4. Ensure TLS/SSL works.
   

   NOTE

   Use TLS version >1.1 as these versions are supported in ZPE Cloud.

   1. openssl s_client -showcerts -servername api.zpecloud.com -connect api.zpecloud.com:443;

   2. ls -lah /etc/ssl/certs | wc -l;

   3. ls -lah /etc/ssl/certs/ca-certificates.crt;

NOTE

The IPv6 connections require a proxy to IPv4.

Operating System and Hardware Checks

1. Check Disk and File System:

   1. df -h /var- Verify disk usage

   2. cat /var/log/fsck.log - Review file system check log

   3. ls -lah /var/zpe-cloud - List contents and permissions in /var/zpe-cloud

   4. du -h -d 1 /var/ - Check disk usage for directories under /var

2. Ensure Daemons are running:

   1. Network processes

      1. ps -ef | grep Network;

   2. TPM processes

      1. TPM1: ps -ef | grep tcsd;

      2. TPM2: ps -ef | grep tpm2;

   3. ZPE Cloud

      1. ps -ef | grep zpe_cloud_forwarder;

      2. ps -ef | grep zpe_cloud_connect;

      3. ps -ef | grep zpe_cloud_agent;

3. Verify service states:

   1. Network access

      1. nmcli c;

   2. TPM

      1. TPM1: tpm_version;

      2. TPM2: tpm2_getcap properties-variable;

Device Enrollment Verification

1. Ensure network access is up and running.

   NOTE

   DNS and TCP are required, ping as network metric are optional.

   1. ping api.zpecloud.com;

   2. telnet api.zpecloud.com 443;

   3. traceroute api.zpecloud.com;

2. Ensure TLS/SSL works.

   NOTE

   Use TLS version >1.1 as these versions are supported in ZPE Cloud.

   1. openssl s_client -showcerts -servername api.zpecloud.com -connect api.zpecloud.com:443;

3. Verify service states:

   1. Enrollment is successful:

      1. zpe_cloud_enroll --enroll-status -u https://zpecloud.com; (-u defaults to https://zpecloud.com);

      2. zpe_cloud_enroll --enroll-list (new versions only);

   2. Enroll secret remains valid:

      1. zpe_cloud_enroll --pairing-status -u https://zpecloud.com; (new versions only); (-u defaults to https://zpecloud.com);

Device Status Online/Offline Status Checks

1. Pair:

   1. Ensure the service is configured:

      1. sudo su - admin -c cli <<< "show /settings/zpe_cloud"

      2. sudo su - admin -c cli <<< "show /settings/zpe_cloud enable_zpe_cloud"

   2. Ensure network access is up and running.

      NOTE

      DNS and TCP are required, ping as network metric are optional.

      1. ping api.astarte.zpecloud.com;

      2. telnet api.astarte.zpecloud.com 443;

      3. traceroute api.astarte.zpecloud.com;

   3. Ensure TLS/SSL works.

      NOTE

      Use TLS version >1.1 as these versions are supported in ZPE Cloud.

      1. openssl s_client -showcerts -servername api.astarte.zpecloud.com -connect api.astarte.zpecloud.com:443;

   4. Verify service states:

      1. Enroll secret remains valid and can pair:

         1. zpe_cloud_enroll --pairing-status -u https://zpecloud.com; (new versions only); (-u defaults to https://zpecloud.com);

         2. zpe_cloud_enroll --pairing-request -u https://zpecloud.com; (new versions only); (-u defaults to https://zpecloud.com);

2. Broker:

   1. Ensure network access is up and running.

      NOTE

      DNS and TCP are required, ping as network metric are optional.

      1. ping broker.astarte.zpecloud.com;

      2. telnet broker.astarte.zpecloud.com 443;

      3. traceroute broker.astarte.zpecloud.com;

   2. Ensure TLS/SSL works.

      NOTE

      Use TLS version >1.1 as these versions are supported in ZPE Cloud.

      1. openssl s_client -showcerts -servername broker.astarte.zpecloud.com -connect broker.astarte.zpecloud.com:443;

   3. Ensure service states are functional.

      1. CRT remains valid and was paired:

         1. zpe_cloud_enroll --pairing-status-u https://zpecloud.com; (new versions only); (-u defaults to https://zpecloud.com);

         2. zpe_cloud_enroll --pairing-request -u https://zpecloud.com; (new versions only); (-u defaults to https://zpecloud.com);

      2. Ensure can connect to broker;

         1. TPM 1.2

            1. openssl s_client -showcerts -servername broker.astarte.zpecloud.com -connect broker.astarte.zpecloud.com:443 -engine tpm -keyform engine -key /var/zpe-cloud/astarte/persistencycrypto/astartekey.pem -cert /var/zpe-cloud/astarte/persistency/endpoint/api.astarte.zpecloud.com/mqtt_broker.crt;

         2. TPM2.0

            1. openssl s_client -showcerts -servername broker.astarte.zpecloud.com -connect broker.astarte.zpecloud.com:443 -engine tpm2tss -keyform engine -key /var/zpe-cloud/astarte/persistency/crypto/astartekey.pem -cert /var/zpe-cloud/astarte/persistency/endpoint/api.astarte.zpecloud.com/mqtt_broker.crt;

            2. openssl s_client -showcerts -servername broker.astarte.zpecloud.com -connect broker.astarte.zpecloud.com:443 -engine libtpm2tss -keyform engine -key /var/zpe-cloud/astarte/persistency/crypto/astartekey.pem -cert /var/zpe-cloud/astarte/persistency/endpoint/api.astarte.zpecloud.com/mqtt_broker.crt;

Remote Access Check

This section is applicable for the Nodegrid devices running versions 5.10.0 and higher version.

1. Ensure the service is configured:

   1. sudo su - admin -c cli <<< "show /settings/zpe_cloud"

   2. sudo su - admin -c cli <<< "show /settings/zpe_cloud enable_zpe_cloud"

   3. sudo su - admin -c cli <<< "show /settings/zpe_cloud enable_remote_access"

2. Ensure network access is up and running.

   NOTE

   DNS and TCP are required, ping as network metric are optional.

   1. ping access.zpecloud.com;

   2. telnet access.zpecloud.com 443;

   3. traceroute access.zpecloud.com;

3. Ensure TLS/SSL works.

   NOTE

   Use TLS version >1.1 as these versions are supported in ZPE Cloud.

   1. openssl s_client -showcerts -servername access.zpecloud.com -connect access.zpecloud.com:443;

4. Verify service states:

   1. Device is Connected:

      1. sudo su - admin -c cli <<< “show /system/zpe_cloud";

      2. sudo su - admin -c cli <<< “show /system/zpe_cloud status";

      3. sudo su - admin -c cli <<< “show /system/zpe_cloud url";

   2. Has certificates signed:

      1. ls -lah /var/zpe-cloud/.crypto/zpecloud.com/;

      2. ls -lah /var/zpe-cloud/.crypto/zpecloud.com/access.crt;

   3. Certificate is valid

      1. openssl x509 -noout -in /var/zpe-cloud/.crypto/zpecloud.com/access.crt -text -dates -issuer;

   4. Ensure can connect to access

      1. openssl s_client -showcerts -servername access.zpecloud.com -connect access.zpecloud.com:443 -key /var/zpe-cloud/.crypto/zpecloud.com/access.pem -cert /var/zpe-cloud/.crypto/zpecloud.com/access.crt < /dev/null;

         NOTE

         You can also replace with with curl command.

      2. curl --key /var/zpe-cloud/.crypto/zpecloud.com/access.pem --cert /var/zpe-cloud/.crypto/zpecloud.com/access.crt --tlsv1.3 https://access.zpecloud.com;

ZTP Check

1. Profile download

   1. Ensure the service is configured:

      1. sudo su - admin -c cli <<< "show /settings/zpe_cloud"

      2. sudo su - admin -c cli <<< "show /settings/zpe_cloud enable_zpe_cloud"

      3. sudo su - admin -c cli <<< "show /settings/zpe_cloud enable_file_protection"

      4. sudo su - admin -c cli <<< "show /settings/zpe_cloud enable_file_encryption"

   2. Ensure date and time are accurate.

      NOTE

      ZTP schedule for profiles could be between 10 minutes to few hours for software upgrade.

      During ZTP, upload of files may fail if the date and time are incorrect and not in sync with the actual date and time.

      1. chronyc tracking;

      2. chronyc ntpdata;

      3. date;

   3. Ensure network access is up and running.

      NOTE

      DNS and TCP are required, ping as network metric are optional.

      1. ping device-api.zpecloud.com;

      2. telnet device-api.zpecloud.com 443;

      3. traceroute device-api.zpecloud.com;

   4. Ensure TLS/SSL works

      NOTE

      Use TLS version >1.1 as these versions are supported in ZPE Cloud.

      1. openssl s_client -showcerts -servername device-api.zpecloud.com -connect device-api.zpecloud.com:443;

   5. Verify service states.

      1. Device is Connected:

         1. sudo su - admin -c cli <<< “show /system/zpe_cloud";

         2. sudo su - admin -c cli <<< “show /system/zpe_cloud status";

         3. sudo su - admin -c cli <<< “show /system/zpe_cloud url";

      2. Ensure can connect to access

         1. wget https://device-api.zpecloud.com;

2. Backup and Profile output upload

   1. Ensure date and time.

      NOTE

      ZTP schedule for profiles could be between 10 minutes to few hours for software upgrade.

      During ZTP, upload of files may fail if the date and time are incorrect and not in sync with the actual date and time.

      1. chronyc tracking;

      2. chronyc ntpdata;

      3. date;

   2. Ensure network access is up and running.

      NOTE

      DNS and TCP are required, ping as network metric are optional.

      1. ping device-apiv2.zpecloud.com;

      2. telnet device-apiv2.zpecloud.com 443;

      3. traceroute device-apiv2.zpecloud.com;

   3. Ensure TLS/SSL works.

      NOTE

      Use TLS version >1.1 as these versions are supported in ZPE Cloud.

      1. openssl s_client -showcerts -servername device-apiv2.zpecloud.com -connect device-apiv2.zpecloud.com:443;

   4. Verify service states.

      1. Device is Connected:

         1. sudo su - admin -c cli <<< “show /system/zpe_cloud";

         2. sudo su - admin -c cli <<< “show /system/zpe_cloud status";

         3. sudo su - admin -c cli <<< “show /system/zpe_cloud url";

      2. Ensure can connect to access

         1. wget https://device-apiv2.zpecloud.com;

         2. openssl s_client -showcerts -servername device-apiv2.zpecloud.com -connect device-apiv2.zpecloud.com:443 -key /var/zpe-cloud/.crypto/zpecloud.com/access.pem -cert /var/zpe-cloud/.crypto/zpecloud.com/access.crt < /dev/null;

            NOTE:

            You can replace it with curl as well.

         3. curl --key /var/zpe-cloud/.crypto/zpecloud.com/access.pem --cert /var/zpe-cloud/.crypto/zpecloud.com/access.crt --tlsv1.3 https://device-apiv2.zpecloud.com;

Recommended Network Properties for ZPE Cloud

To ensure ZPE Cloud performs as expected, your Nodegrid device’s network must meet the following configuration requirements to avoid issues such as device remote access delays or failures, delayed software upgrade times, connection instability, and failed data transfers.

To ensure a seamless experience with ZPE Cloud, it is important to maintain a well-configured and reliable network. A stable connection ensures quick and successful remote access, smooth software updates, and consistent connection stability without frequent reconnections. The network link connecting to a Nodegrid device plays a vital role, as it serves as the bridge between your managed devices and ZPE Cloud. For the best results, ensure this link is robust and dependable. Additionally, when accessing ZPE Cloud through a browser, a strong network connection for Nodegrid will further enhance performance and usability.