How to Troubleshoot ZPE Cloud Environment

Updated on 06 Nov 2024
5 Minutes to read

Print
Dark
Light
PDF

Article summary

Did you find this summary helpful?

Thank you for your feedback

Overview

This document provides information on essential commands used in troubleshooting environment for ZPE Cloud services on Nodegrid Devices. The most common troubleshooting involves similar checks, as they typically relate to disk, network, date or time settings, and other environmental factors. However, each section may have specific checks relevant to that area.

The document includes steps to check the following:

Enrollment status
Device status
- Online
- Offline
- Never connected
- Failover (Pending)
Remote access
ZTP (Zero Touch Provisioning)
- Profiles
- Backup

Basic Environment Checks

These are the essential system checks that must be functioning properly before troubleshooting the applications. The listed commands are run on diagnostic data, allowing you to identify and address minor environmental issues. If these commands fail, it indicates that the environment needs to be fixed before proceeding.

Execute the following with api.zpecloud.com and google.com.

Ensure the service is configured:

sudo su - admin -c cli <<< "show /settings/zpe_cloud"

sudo su - admin -c cli <<< "show /settings/zpe_cloud enable_zpe_cloud"

sudo su - admin -c cli <<< "show /settings/zpe_cloud enable_remote_access"

Ensure date and time are accurate:
1. chronyc tracking - provides detailed information about the system time synchronization.
2. chronyc ntpdata - displays information about the network time protocol (NTP).
3. date - shows the current system date and time.
Ensure network access is up and running.
NOTE
DNS and TCP are required, ping as network metric are optional.
1. nmcli c
2. nslookup api.zpecloud.com;
3. ping api.zpecloud.com;
  1. ping -4 -I wwan0 api.zpecloud.com;
  2. ping -4 -I eth0 api.zpecloud.com;
  3. ping -6 -I wwan0 api.zpecloud.com;
  4. ping -6 -I eth0 api.zpecloud.com;
4. telnet api.zpecloud.com 443;
5. traceroute api.zpecloud.com;
Ensure TLS/SSL works.
NOTE
Use TLS version >1.1 as these versions are supported in ZPE Cloud.
1. openssl s_client -showcerts -servername api.zpecloud.com -connect api.zpecloud.com:443;
2. ls -lah /etc/ssl/certs | wc -l;
3. ls -lah /etc/ssl/certs/ca-certificates.crt;

NOTE
The IPv6 connections require a proxy to IPv4.

Operating System and Hardware Checks

Check Disk and File System:
1. df -h /var- Verify disk usage
2. cat /var/log/fsck.log - Review file system check log
3. ls -lah /var/zpe-cloud - List contents and permissions in /var/zpe-cloud
4. du -h -d 1 /var/ - Check disk usage for directories under /var
Ensure Daemons are running:
1. Network processes
  1. ps -ef | grep Network;
2. TPM processes
  1. TPM1: ps -ef | grep tcsd;
  2. TPM2: ps -ef | grep tpm2;
3. ZPE Cloud
  1. ps -ef | grep zpe_cloud_forwarder;
  2. ps -ef | grep zpe_cloud_connect;
  3. ps -ef | grep zpe_cloud_agent;
Verify service states:
1. Network access
  1. nmcli c;
2. TPM
  1. TPM1: tpm_version;
  2. TPM2: tpm2_getcap properties-variable;

Device Enrollment Verification

Ensure network access is up and running.
NOTE
DNS and TCP are required, ping as network metric are optional.
1. ping api.zpecloud.com;
2. telnet api.zpecloud.com 443;
3. traceroute api.zpecloud.com;
Ensure TLS/SSL works.
NOTE
Use TLS version >1.1 as these versions are supported in ZPE Cloud.
1. openssl s_client -showcerts -servername api.zpecloud.com -connect api.zpecloud.com:443;
Verify service states:
1. Enrollment is successful:
  1. zpe_cloud_enroll --enroll-status -u https://zpecloud.com; (-u defaults to https://zpecloud.com);
  2. zpe_cloud_enroll --enroll-list (new versions only);
2. Enroll secret remains valid:
  1. zpe_cloud_enroll --pairing-status -u https://zpecloud.com; (new versions only); (-u defaults to https://zpecloud.com);

Device Status Online/Offline Status Checks

Pair:
1. Ensure the service is configured:
  1. ```
  sudo su - admin -c cli <<< "show /settings/zpe_cloud"
```
2. ```
sudo su - admin -c cli <<< "show /settings/zpe_cloud enable_zpe_cloud"
```
2. Ensure network access is up and running.
  NOTE
  DNS and TCP are required, ping as network metric are optional.
  1. ping api.astarte.zpecloud.com;
  2. telnet api.astarte.zpecloud.com 443;
  3. traceroute api.astarte.zpecloud.com;
3. Ensure TLS/SSL works.
  NOTE
  Use TLS version >1.1 as these versions are supported in ZPE Cloud.
  1. openssl s_client -showcerts -servername api.astarte.zpecloud.com -connect api.astarte.zpecloud.com:443;
4. Verify service states:
  1. Enroll secret remains valid and can pair:
    1. zpe_cloud_enroll --pairing-status -u https://zpecloud.com; (new versions only); (-u defaults to https://zpecloud.com);
    2. zpe_cloud_enroll --pairing-request -u https://zpecloud.com; (new versions only); (-u defaults to https://zpecloud.com);
Broker:
1. Ensure network access is up and running.
  NOTE
  DNS and TCP are required, ping as network metric are optional.
  1. ping broker.astarte.zpecloud.com;
  2. telnet broker.astarte.zpecloud.com 443;
  3. traceroute broker.astarte.zpecloud.com;
2. Ensure TLS/SSL works.
  NOTE
  Use TLS version >1.1 as these versions are supported in ZPE Cloud.
  1. openssl s_client -showcerts -servername broker.astarte.zpecloud.com -connect broker.astarte.zpecloud.com:443;
3. Ensure service states are functional.
  1. CRT remains valid and was paired:
    1. zpe_cloud_enroll --pairing-status-u https://zpecloud.com; (new versions only); (-u defaults to https://zpecloud.com);
    2. zpe_cloud_enroll --pairing-request -u https://zpecloud.com; (new versions only); (-u defaults to https://zpecloud.com);
  2. Ensure can connect to broker;
    1. TPM 1.2
      1. openssl s_client -showcerts -servername broker.astarte.zpecloud.com -connect broker.astarte.zpecloud.com:443 -engine tpm -keyform engine -key /var/zpe-cloud/astarte/persistencycrypto/astartekey.pem -cert /var/zpe-cloud/astarte/persistency/endpoint/api.astarte.zpecloud.com/mqtt_broker.crt;
    2. TPM2.0
      1. openssl s_client -showcerts -servername broker.astarte.zpecloud.com -connect broker.astarte.zpecloud.com:443 -engine tpm2tss -keyform engine -key /var/zpe-cloud/astarte/persistency/crypto/astartekey.pem -cert /var/zpe-cloud/astarte/persistency/endpoint/api.astarte.zpecloud.com/mqtt_broker.crt;
      2. openssl s_client -showcerts -servername broker.astarte.zpecloud.com -connect broker.astarte.zpecloud.com:443 -engine libtpm2tss -keyform engine -key /var/zpe-cloud/astarte/persistency/crypto/astartekey.pem -cert /var/zpe-cloud/astarte/persistency/endpoint/api.astarte.zpecloud.com/mqtt_broker.crt;

Remote Access Check

This section is applicable for the Nodegrid devices running versions 5.10.0 and higher version.

Ensure the service is configured:

sudo su - admin -c cli <<< "show /settings/zpe_cloud"

sudo su - admin -c cli <<< "show /settings/zpe_cloud enable_zpe_cloud"

sudo su - admin -c cli <<< "show /settings/zpe_cloud enable_remote_access"

Ensure network access is up and running.
NOTE
DNS and TCP are required, ping as network metric are optional.
1. ping access.zpecloud.com;
2. telnet access.zpecloud.com 443;
3. traceroute access.zpecloud.com;
Ensure TLS/SSL works.
NOTE
Use TLS version >1.1 as these versions are supported in ZPE Cloud.
1. openssl s_client -showcerts -servername access.zpecloud.com -connect access.zpecloud.com:443;
Verify service states:
1. Device is Connected:
  1. sudo su - admin -c cli <<< “show /system/zpe_cloud";
  2. sudo su - admin -c cli <<< “show /system/zpe_cloud status";
  3. sudo su - admin -c cli <<< “show /system/zpe_cloud url";
2. Has certificates signed:
  1. ls -lah /var/zpe-cloud/.crypto/zpecloud.com/;
  2. ls -lah /var/zpe-cloud/.crypto/zpecloud.com/access.crt;
3. Certificate is valid
  1. openssl x509 -noout -in /var/zpe-cloud/.crypto/zpecloud.com/access.crt -text -dates -issuer;
4. Ensure can connect to access
  1. openssl s_client -showcerts -servername access.zpecloud.com -connect access.zpecloud.com:443 -key /var/zpe-cloud/.crypto/zpecloud.com/access.pem -cert /var/zpe-cloud/.crypto/zpecloud.com/access.crt < /dev/null;
    NOTE
    You can also replace with with curl command.
  2. curl --key /var/zpe-cloud/.crypto/zpecloud.com/access.pem --cert /var/zpe-cloud/.crypto/zpecloud.com/access.crt --tlsv1.3 https://access.zpecloud.com;

ZTP Check

Profile download
1. Ensure the service is configured:
  1. ```
  sudo su - admin -c cli <<< "show /settings/zpe_cloud"
```
2. ```
sudo su - admin -c cli <<< "show /settings/zpe_cloud enable_zpe_cloud"
```
  3. ```
  sudo su - admin -c cli <<< "show /settings/zpe_cloud enable_file_protection"
```
4. ```
sudo su - admin -c cli <<< "show /settings/zpe_cloud enable_file_encryption"
```
2. Ensure date and time are accurate.
  NOTE
  ZTP schedule for profiles could be between 10 minutes to few hours for software upgrade.
  During ZTP, upload of files may fail if the date and time are incorrect and not in sync with the actual date and time.
  1. chronyc tracking;
  2. chronyc ntpdata;
  3. date;
3. Ensure network access is up and running.
  NOTE
  DNS and TCP are required, ping as network metric are optional.
  1. ping device-api.zpecloud.com;
  2. telnet device-api.zpecloud.com 443;
  3. traceroute device-api.zpecloud.com;
4. Ensure TLS/SSL works
  NOTE
  Use TLS version >1.1 as these versions are supported in ZPE Cloud.
  1. openssl s_client -showcerts -servername device-api.zpecloud.com -connect device-api.zpecloud.com:443;
5. Verify service states.
  1. Device is Connected:
    1. sudo su - admin -c cli <<< “show /system/zpe_cloud";
    2. sudo su - admin -c cli <<< “show /system/zpe_cloud status";
    3. sudo su - admin -c cli <<< “show /system/zpe_cloud url";
  2. Ensure can connect to access
    1. wget https://device-api.zpecloud.com;
Backup and Profile output upload
1. Ensure date and time.
  NOTE
  ZTP schedule for profiles could be between 10 minutes to few hours for software upgrade.
  During ZTP, upload of files may fail if the date and time are incorrect and not in sync with the actual date and time.
  1. chronyc tracking;
  2. chronyc ntpdata;
  3. date;
2. Ensure network access is up and running.
  NOTE
  DNS and TCP are required, ping as network metric are optional.
  1. ping device-apiv2.zpecloud.com;
  2. telnet device-apiv2.zpecloud.com 443;
  3. traceroute device-apiv2.zpecloud.com;
3. Ensure TLS/SSL works.
  NOTE
  Use TLS version >1.1 as these versions are supported in ZPE Cloud.
  1. openssl s_client -showcerts -servername device-apiv2.zpecloud.com -connect device-apiv2.zpecloud.com:443;
4. Verify service states.
  1. Device is Connected:
    1. sudo su - admin -c cli <<< “show /system/zpe_cloud";
    2. sudo su - admin -c cli <<< “show /system/zpe_cloud status";
    3. sudo su - admin -c cli <<< “show /system/zpe_cloud url";
  2. Ensure can connect to access
    1. wget https://device-apiv2.zpecloud.com;
    2. openssl s_client -showcerts -servername device-apiv2.zpecloud.com -connect device-apiv2.zpecloud.com:443 -key /var/zpe-cloud/.crypto/zpecloud.com/access.pem -cert /var/zpe-cloud/.crypto/zpecloud.com/access.crt < /dev/null;
      NOTE:
      You can replace it with curl as well.
    3. curl --key /var/zpe-cloud/.crypto/zpecloud.com/access.pem --cert /var/zpe-cloud/.crypto/zpecloud.com/access.crt --tlsv1.3 https://device-apiv2.zpecloud.com;

Was this article helpful?

What's Next

Securing ZPE Systems Network

Table of contents

Overview
Basic Environment Checks
Operating System and Hardware Checks
Device Enrollment Verification
Device Status Online/Offline Status Checks
Remote Access Check
ZTP Check