Troubleshoot CLI's - ESX, NSX, NSX-Intelligence, Edge, FW, IDS/IPS

 On NSX mgr
==========
nsx-mgr-1> get managers
nsx-mgr-1> get management-cluster status
nsx-mgr-1> get cluster status
nsx-mgr-1> get nodes
nsx-mgr-1> get interfaces
nsx-mgr-1> get transport-nodes status 
nsx-mgr-1> get configuration
nsx-mgr-1> get cluster config
nsx-mgr-1> get firewall summary
nsx-mgr-1> get firewall published-entity
nsx-mgr-1> get firewall status
nsx-mgr-1> get logical-routers
nsx-mgr-1> get logical-switches
nsx-mgr-1> get service manager
nsx-mgr-1> get log-file syslog follow 
nsx-mgr-1> get logging-servers
root@nsx-mgr-1:~# tail -f ./var/log/proton/nsxapi.log
root@nsx-mgr-1:~# tail -f /var/log/policy/policy.log ## policy mgr logs
nsx-mgr-1> set service manager logging-level debug
nsx-mgr-1> get logical-switch <id> arp-table
 

nsxcli commands on ESX
===================
> get intelligence flow config
Displays current PACE flow configuration
> get intelligence flow stats
Displays flow export statistics
> get intelligence flow stats ack
Displays flow export acknowledgement stats

On ESX:
======
List the VIBs loaded on ESXi
[root@prom-05056b17085:~] esxcli software vib list | grep -e nsx -e vsip
Host to mgr communication
[root@prom-05056b17085:~] esxcli network ip connection list | grep 1235
[root@prom-05056b17085:~] esxcli network nic list
[root@prom-05056b17085:~] esxcli network firewall ruleset set -r syslog -e true
[root@prom-05056b17085:~] esxcfg-nics -l
[root@prom-05056b17085:~] tail -f /var/log/dfwpktlogs.log

Firewall logs on ESX:
===============
[root@prom-05056b17085:~] tail -f /var/log/dfwpktlogs.log
[root@prom-05056b17085:~]  summarize-dvfilter   ## get the VM interface  ID
[root@prom-05056b17085:~] vsipioctl getrules -f nic-6389857-eth2-vmware-sfw.2
[root@prom-05056b17085:~] cat /var/log/dfwpktlogs.log
[root@prom-05056b17085:~] nsxcli
prom-05056b17085.system.test> get logical-switches
prom-05056b17085.system.test> get firewall vifs
prom-05056b17085.system.test> get firewall rule-stats
prom-05056b17085.system.test> get firewall status
prom-05056b17085.system.test> get firewall <vifuuid> ruleset rules
prom-05056b17085.system.test> get firewall <vifuuid> addrsets
prom-05056b17085.system.test> get logical-router 51515dfa-b15a-45d6-9d60-90e4c40023e1 interfaces
prom-05056b17085.system.test> get logical-router 51515dfa-b15a-45d6-9d60-90e4c40023e1 interface 85d532cc-2627-4ae0-8ee6-e7ba38f67572
prom-05056b17085.system.test> get host-switch PROD-Overlay-NVDS tunnels

ESX:
===
[root@blrkv-hs1-b0606:~] esxcli network ip interface ipv4 get
[root@blrkv-hs1-b0606:~] esxcli vm process list
[root@blrkv-hs1-b0606:~] esxcli storage vmfs extent list

NSX Edge:
=========
edge> get service ssh
edge> get configuration 
edge> get interfaces
edge> get managers
edge> get host-switches
edge> get tunnel-ports
edge> get vteps
edge> get logical-routers
edge> get logical-switches
edge> get firewall interfaces
edge> get firewall <interfaceid> ruleset
edge> get firewall <interfaceid> ruleset rules
edge> get dhcp ip-pools
edge> get dhcp leases

On NSX-Intelligence:
===============
Intel> get service
Intel> get version
Intel> get services
get log-file syslog | find pace-monitor
Intel>restart service service-name
root@Intel:~# cat /var/log/pace/token-registration.log 
root@Intel:~# cat /var/log/pace/pace-server.log

root@Intel:~# tail -f /var/log/pace/nsx-config.log

Periodic health check service called pace-monitor that posts the overall appliance health to syslog.
root@Intel:~# journalctl -u pace-monitor -n 60 --no-pager
Intel> get service druid

API:
====

root@systest-runner:~[1154]# curl -k -u username:password  https://<nsx-ip>/api/v1/logical-ports
root@systest-runner:~[1156]# curl -k -u username:password https://<nsx-ip>/api/v1/firewall/status
root@systest-runner:~[1158]# curl -k -u username:password https://<nsx-ip>/api/v1/firewall/sections/<id>/rules
root@systest-runner:~[1168]# curl -I -k -u admin:VMware1VMware! https://<nsx-ip>/policy/api/v1/infra/realized-state/realized-entities

On worknode for scan:
=================
root@ubuntu-1804:~# nmap -T4 -Pn -A -v <server-ip>
Starting Nmap 7.60 ( https://nmap.org ) at 2021-12-07 11:01 UTC
NSE: Loaded 146 scripts for scanning.
-T<0-5>: Set timing template (higher is faster)
-A: Enable OS detection, version detection, script scanning, and traceroute
-v: Increase verbosity level (use -vv or more for greater effect)
 -V: Print version number
-Pn: Treat all hosts as online -- skip host discovery

IPS/IDS On ESX
===========
[root@prom-05056b17085:~] nsxcli -c get ids engine stats
[root@prom-05056b17085:~] nsxcli -c get ids events stats

Capture on the edge:
================

edge> get logical-routers
edge(tier1_sr[1])> get interfaces
edge(tier1_sr[1])> get interfaces stats
edge> start capture interface <id>
edge> start capture interface <id>

From client initiate traffic:
root@ubuntu-1804:~# nmap --script http-vuln-cve2014-3704 --script-args http-vuln-cve2014-3704.uri=“/drupal”,http-vuln-cve2014-3704.cleanup=false <server-ip> --disable-arp-ping

Logs:
[root@blrkv-hs1-b0606:~] cd /var/log/nsx-idps
 [root@blrkv-hs1-b0606:/var/log/nsx-idps] summarize-dvfilter
 
To confirm that IDS is enabled on this host, run the command get ids status
On ESX -> nsxcli

prom-05056b17085.system.test> get ids profile

To review IDS profile (engine) statistics including the number of packets processed and alerts generated, 

run the command get ids engine profilestats <tab_to_select_profile_ID>

[root@prom-05056b96efd:~] summarize-dvfilter | grep -A 3 vmm

Get all the DFW rules using "vsipioctl getrules -f <VM_interface_ID>"

Get the flow statistics for rule using command "vsipioctl getrules -f <VM_interface_ID> -s"

Get the addrset/groups used in the VM's Firewall rules

[root@prom-05056b17085:/var/log/nsx-idps] vsipioctl getaddrset -f <nic>

Get the active Firewall flow per VM. Trigger some traffic and verify
[root@prom-05056b1c8f2:/var/log/nsx-idps] vsipioctl getflows -f <nic>

Get the active Full Firewall config per VM
[root@prom-05056b17085:/var/log/nsx-idps] vsipioctl getfwconfig -f <nic>