Troubleshooting NSX-T Backups

Overview:

Backups are an important part of maintaining an NSX-T environment. Troubleshooting backups through the UI can be challenging since the errors are often generic. In this post we’ll begin with a review of NSX-T backup configuration parameters, look at some troubleshooting options performed from the NSX-T Manager CLI, and troubleshoot an SFTP backup failure.

Setting up for SFTP backup on NSX-T:

Let’s begin with a review of the settings, accessed from System, Backup & Restore:

Edit Backup Configuration settings :

Port:

Port 22 is default for sftp, and should work for most SFTP Server installations.

Protocol:

Only sftp is supported.

Password:

Only required to be entered on the first backup, and then leave it blank to reuse the same password on subsequent backups.

Destination Directory:

A relative path is not supported, an absolute path is required
An absolute path points to the same location in a file system, regardless of the current working directory. To do that, it must include the root directory

Encryption Passphrase restrictions:

The passphrase specified must be at least 8 characters in length and must contain at least one lowercase, one uppercase, one numeric character and one special character.
Record the passphrase since you need to use this same passphrase to restore from the backup

SSH fingerprint:

The syntax starts with SHA256:
For example:
- SHA256:gxjoFPjc62jLhS2RPUXgYjvcsxHrCQT2+Hp3HWncNTI
It can be added automatically if supported by the SFTP server, or it can be collected manually from the server and then entered.

FIPS compliance:

NSX-T is FIPS compliant as of NSX-T 2.4.0, and requires a FIPS compliant SFTP Server.

Reviewing Supported Cyphers and Macs, making sure your SFTP Server has matching algorithms:

These can be determined from the NSX-T Manager CLI, using the following commands:

root@nsxtmgr02:/etc/ssh# ssh -Q kex
root@nsxtmgr02:/etc/ssh# ssh -Q key
root@nsxtmgr02:/etc/ssh# ssh -Q cipher
root@nsxtmgr02:/etc/ssh# ssh -Q cipher-auth
root@nsxtmgr02:/etc/ssh# ssh -Q mac

When a SSH client connects to a server, each side offers lists of connection parameters to the other. If the client and server are unable to agree on a mutual set of parameters then the connection will fail.

From this NSX-T Manager supports the following options:

Key Exchange Methods: curve25519-sha256@libssh.org, ecdh-sha2-nistp256, ecdh-sha2-nistp384, ecdh-sha2-nistp521, diffie-hellman-group-exchange-sha256, diffie-hellman-group-exchange-sha1, diffie-hellman-group14-sha1

Key Algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com, ecdsa-sha2-nistp384-cert-v01@openssh.com, ecdsa-sha2-nistp521-cert-v01@openssh.com, ssh-ed25519-cert-v01@openssh.com, ssh-rsa-cert-v01@openssh.com, ecdsa-sha2-nistp256, ecdsa-sha2-nistp384, ecdsa-sha2-nistp521, ssh-ed25519, rsa-sha2-512, rsa-sha2-256,ssh-rsa

Ciphers: aes128-ctr, aes192-ctr, aes256-ctr, aes128-gcm@openssh.com, aes256-gcm@openssh.com, chacha20-poly1305@openssh.com

MACs: hmac-sha2-256-etm@openssh.com, hmac-sha2-512-etm@openssh.com, hmac-sha2-256, hmac-sha2-512

Compressions: none, zlib@openssh.com, zlib

Setting up an SFTP Server on a VMware Photon OS 3.0 Guest VM:

In my lab, all testing was done with a VMware Photon OS 3.0 Guest VM – Deployed from the following OVA: photon-hw11-3.0-26156e2.ova

Setup Guest VM with static IP , referencing: https://vmguru.com/2018/10/getting-started-with-photon-os/
Photon OS default user: root, password: changeme
SFTP Server is running by default
There are no additional packages to install:

root@photon-machine [ / ]# sftp 
usage: sftp [-46aCfpqrv] [-B buffer_size] [-b batchfile] [-c cipher]               [-D sftp_server_path] [-F ssh_config] [-i identity_file] [-l limit]               [-o ssh_option] [-P port] [-R num_requests] [-S program][-s subsystem | sftp_server] destination

NSX-T Backup Troubleshooting Steps:

Enter Engineering mode on the NSX-T manager:

nsxtmgr02> st en
Password:
***************************************************************************
NOTICE TO USERS

WARNING! Changes made to NSX Data Center while logged in as the root user
can cause system failure and potentially impact your network. Please be
advised that changes made to the system as the root user must only be made
under the guidance of VMware.
***************************************************************************

root@nsxtmgr02:~

Monitor NSX-T Manager logs during the backup attempt:

root@nsxtmgr02:~# tail -f ./var/log/nvpapi/api_server.log
root@nsxtmgr02:~# tail -f ./var/log/proton/nsxapi.log

Use ncat (nc) to verify the SFTP server IP and port are accessible:

root@nsxtmgr02:~# nc -zv 192.168.70.50 22
 Connection to 192.168.70.50 22 port [tcp/ssh] succeeded!

To collect the public key from your SFTP Server, to be used as the SSH fingerprint, using a Photon OS 3.0 Server as an example :

root@photon-machine [ ~ ]# cat /etc/ssh/sshd_config | grep HostKey
 HostKey /etc/ssh/ssh_host_rsa_key
 HostKey /etc/ssh/ssh_host_ecdsa_key
 HostKey /etc/ssh/ssh_host_ed25519_key

 root@photon-machine [ ~ ]# cd /etc/ssh
 root@photon-machine [ /etc/ssh ]# ls -larth *.pub
 -rw-r--r-- 1 root root 401 Jun 21 10:38 ssh_host_rsa_key.pub
 -rw-r--r-- 1 root root 609 Jun 21 10:38 ssh_host_dsa_key.pub
 -rw-r--r-- 1 root root 181 Jun 21 10:38 ssh_host_ecdsa_key.pub
 -rw-r--r-- 1 root root 101 Jun 21 10:38 ssh_host_ed25519_key.pub

root@photon-machine [ /etc/ssh ]# ssh-keygen -l -f ssh_host_ecdsa_key.pub
 256 SHA256:gxjoFPjc62jLhS2RPUXgYjvcsxHrCQT2+Hp3HWncNTI root@photon-machine (ECDSA) <------ Verify that this is the SSH fingerprint being used.

Test SSH Access from the NSX-T Manager to the SFTP Server:

root@nsxtmgr02:~# ssh root@192.168.70.50
 Password:
 Last login: Fri Jun 28 19:47:12 2019 from 192.168.110.10
  21:36:27 up 2 days, 47 min,  3 users,  load average: 0.00, 0.00, 0.0

If SSH fails, then reattempt with maximum verbosity, successful connection shown:

root@nsxtmgr02:~# ssh -vvv root@192.168.70.50
 OpenSSH_7.2p2 Ubuntu-4ubuntu2.6, OpenSSL 1.0.2p-fips  14 Aug 2018
 debug1: Reading configuration data /etc/ssh/ssh_config
 debug1: /etc/ssh/ssh_config line 19: Applying options for *
 debug2: resolving "192.168.70.50" port 22
 debug2: ssh_connect_direct: needpriv 0
 debug1: Connecting to 192.168.70.50 [192.168.70.50] port 22.
 debug1: Connection established.
 debug1: permanently_set_uid: 0/0
 debug1: key_load_public: No such file or directory
 debug1: identity file /root/.ssh/id_rsa type -1
 debug1: key_load_public: No such file or directory
 debug1: identity file /root/.ssh/id_rsa-cert type -1
 debug1: key_load_public: No such file or directory
 debug1: identity file /root/.ssh/id_dsa type -1
 debug1: key_load_public: No such file or directory
 debug1: identity file /root/.ssh/id_dsa-cert type -1
 debug1: key_load_public: No such file or directory
 debug1: identity file /root/.ssh/id_ecdsa type -1
 debug1: key_load_public: No such file or directory
 debug1: identity file /root/.ssh/id_ecdsa-cert type -1
 debug1: key_load_public: No such file or directory
 debug1: identity file /root/.ssh/id_ed25519 type -1
 debug1: key_load_public: No such file or directory
 debug1: identity file /root/.ssh/id_ed25519-cert type -1
 debug1: Enabling compatibility mode for protocol 2.0
 debug1: Local version string SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.6
 debug1: Remote protocol version 2.0, remote software version OpenSSH_7.8
 debug1: match: OpenSSH_7.8 pat OpenSSH* compat 0x04000000
 debug2: fd 3 setting O_NONBLOCK
 debug1: Authenticating to 192.168.70.50:22 as 'root'
 debug3: hostkeys_foreach: reading file "/root/.ssh/known_hosts"
 debug3: record_hostkey: found key type ECDSA in file /root/.ssh/known_hosts:1
 debug3: load_hostkeys: loaded 1 keys from 192.168.70.50
 debug3: order_hostkeyalgs: prefer hostkeyalgs: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
 debug3: send packet: type 20
 debug1: SSH2_MSG_KEXINIT sent
 debug3: receive packet: type 20
 debug1: SSH2_MSG_KEXINIT received
 debug2: local client KEXINIT proposal
 debug2: KEX algorithms: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,ext-info-c
 debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa
 debug2: ciphers ctos: aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com
 debug2: ciphers stoc: aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com
 debug2: MACs ctos: hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha2-256,hmac-sha2-512
 debug2: MACs stoc: hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha2-256,hmac-sha2-512
 debug2: compression ctos: none,zlib@openssh.com,zlib
 debug2: compression stoc: none,zlib@openssh.com,zlib
 debug2: languages ctos:
 debug2: languages stoc:
 debug2: first_kex_follows 0
 debug2: reserved 0
 debug2: peer server KEXINIT proposal
 debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1
 debug2: host key algorithms: rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256,ssh-ed25519
 debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
 debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
 debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
 debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
 debug2: compression ctos: none
 debug2: compression stoc: none
 debug2: languages ctos:
 debug2: languages stoc:
 debug2: first_kex_follows 0
 debug2: reserved 0
 debug1: kex: algorithm: curve25519-sha256@libssh.org
 debug1: kex: host key algorithm: ecdsa-sha2-nistp256
 debug1: kex: server->client cipher: aes128-ctr MAC: hmac-sha2-256-etm@openssh.com compression: none
 debug1: kex: client->server cipher: aes128-ctr MAC: hmac-sha2-256-etm@openssh.com compression: none
 debug3: send packet: type 30
 debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
 debug3: receive packet: type 31
 debug1: Server host key: ecdsa-sha2-nistp256 SHA256:gxjoFPjc62jLhS2RPUXgYjvcsxHrCQT2+Hp3HWncNTI
 debug3: hostkeys_foreach: reading file "/root/.ssh/known_hosts"
 debug3: record_hostkey: found key type ECDSA in file /root/.ssh/known_hosts:1
 debug3: load_hostkeys: loaded 1 keys from 192.168.70.50
 debug1: Host '192.168.70.50' is known and matches the ECDSA host key.
 debug1: Found key in /root/.ssh/known_hosts:1
 debug3: send packet: type 21
 debug2: set_newkeys: mode 1
 debug1: rekey after 4294967296 blocks
 debug1: SSH2_MSG_NEWKEYS sent
 debug1: expecting SSH2_MSG_NEWKEYS
 debug3: receive packet: type 21
 debug1: SSH2_MSG_NEWKEYS received
 debug2: set_newkeys: mode 0
 debug1: rekey after 4294967296 blocks
 debug2: key: /root/.ssh/id_rsa ((nil))
 debug2: key: /root/.ssh/id_dsa ((nil))
 debug2: key: /root/.ssh/id_ecdsa ((nil))
 debug2: key: /root/.ssh/id_ed25519 ((nil))
 debug3: send packet: type 5
 debug3: receive packet: type 7
 debug1: SSH2_MSG_EXT_INFO received
 debug1: kex_input_ext_info: server-sig-algs=
 debug3: receive packet: type 6
 debug2: service_accept: ssh-userauth
 debug1: SSH2_MSG_SERVICE_ACCEPT received
 debug3: send packet: type 50
 debug3: receive packet: type 51
 debug1: Authentications that can continue: publickey,password,keyboard-interactive
 debug3: start over, passed a different list publickey,password,keyboard-interactive
 debug3: preferred gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive,password
 debug3: authmethod_lookup publickey
 debug3: remaining preferred: keyboard-interactive,password
 debug3: authmethod_is_enabled publickey
 debug1: Next authentication method: publickey
 debug1: Trying private key: /root/.ssh/id_rsa
 debug3: no such identity: /root/.ssh/id_rsa: No such file or directory
 debug1: Trying private key: /root/.ssh/id_dsa
 debug3: no such identity: /root/.ssh/id_dsa: No such file or directory
 debug1: Trying private key: /root/.ssh/id_ecdsa
 debug3: no such identity: /root/.ssh/id_ecdsa: No such file or directory
 debug1: Trying private key: /root/.ssh/id_ed25519
 debug3: no such identity: /root/.ssh/id_ed25519: No such file or directory
 debug2: we did not send a packet, disable method
 debug3: authmethod_lookup keyboard-interactive
 debug3: remaining preferred: password
 debug3: authmethod_is_enabled keyboard-interactive
 debug1: Next authentication method: keyboard-interactive
 debug2: userauth_kbdint
 debug3: send packet: type 50
 debug2: we sent a keyboard-interactive packet, wait for reply
 debug3: receive packet: type 60
 debug2: input_userauth_info_req
 debug2: input_userauth_info_req: num_prompts 1

Test SFTP Server access from the NSX-T Manager to the SFTP Server:

root@nsxtmgr03:/var/log/proton# sftp root@192.168.70.50
 The authenticity of host '192.168.70.50 (192.168.70.50)' can't be established.
 ECDSA key fingerprint is SHA256:gxjoFPjc62jLhS2RPUXgYjvcsxHrCQT2+Hp3HWncNTI.
 Are you sure you want to continue connecting (yes/no)? yes
 Warning: Permanently added '192.168.70.50' (ECDSA) to the list of known hosts.
 Password:
 Connected to 192.168.70.50.
 sftp> exit

If SFTP access fails, then reattempt with maximum verbosity, successful connection shown:

root@nsxtmgr02:~# sftp -vvv root@192.168.70.50
 OpenSSH_7.2p2 Ubuntu-4ubuntu2.6, OpenSSL 1.0.2p-fips  14 Aug 2018
 debug1: Reading configuration data /etc/ssh/ssh_config
 debug1: /etc/ssh/ssh_config line 19: Applying options for *
 debug2: resolving "192.168.70.50" port 22
 debug2: ssh_connect_direct: needpriv 0
 debug1: Connecting to 192.168.70.50 [192.168.70.50] port 22.
 debug1: Connection established.
 debug1: permanently_set_uid: 0/0
 debug1: key_load_public: No such file or directory
 debug1: identity file /root/.ssh/id_rsa type -1
 debug1: key_load_public: No such file or directory
 debug1: identity file /root/.ssh/id_rsa-cert type -1
 debug1: key_load_public: No such file or directory
 debug1: identity file /root/.ssh/id_dsa type -1
 debug1: key_load_public: No such file or directory
 debug1: identity file /root/.ssh/id_dsa-cert type -1
 debug1: key_load_public: No such file or directory
 debug1: identity file /root/.ssh/id_ecdsa type -1
 debug1: key_load_public: No such file or directory
 debug1: identity file /root/.ssh/id_ecdsa-cert type -1
 debug1: key_load_public: No such file or directory
 debug1: identity file /root/.ssh/id_ed25519 type -1
 debug1: key_load_public: No such file or directory
 debug1: identity file /root/.ssh/id_ed25519-cert type -1
 debug1: Enabling compatibility mode for protocol 2.0
 debug1: Local version string SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.6
 debug1: Remote protocol version 2.0, remote software version OpenSSH_7.8
 debug1: match: OpenSSH_7.8 pat OpenSSH* compat 0x04000000
 debug2: fd 3 setting O_NONBLOCK
 debug1: Authenticating to 192.168.70.50:22 as 'root'
 debug3: hostkeys_foreach: reading file "/root/.ssh/known_hosts"
 debug3: record_hostkey: found key type ECDSA in file /root/.ssh/known_hosts:1
 debug3: load_hostkeys: loaded 1 keys from 192.168.70.50
 debug3: order_hostkeyalgs: prefer hostkeyalgs: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
 debug3: send packet: type 20
 debug1: SSH2_MSG_KEXINIT sent
 debug3: receive packet: type 20
 debug1: SSH2_MSG_KEXINIT received
 debug2: local client KEXINIT proposal
 debug2: KEX algorithms: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,ext-info-c
 debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa
 debug2: ciphers ctos: aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com
 debug2: ciphers stoc: aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com
 debug2: MACs ctos: hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha2-256,hmac-sha2-512
 debug2: MACs stoc: hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha2-256,hmac-sha2-512
 debug2: compression ctos: none,zlib@openssh.com,zlib
 debug2: compression stoc: none,zlib@openssh.com,zlib
 debug2: languages ctos:
 debug2: languages stoc:
 debug2: first_kex_follows 0
 debug2: reserved 0
 debug2: peer server KEXINIT proposal
 debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1
 debug2: host key algorithms: rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256,ssh-ed25519
 debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
 debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
 debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
 debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
 debug2: compression ctos: none
 debug2: compression stoc: none
 debug2: languages ctos:
 debug2: languages stoc:
 debug2: first_kex_follows 0
 debug2: reserved 0
 debug1: kex: algorithm: curve25519-sha256@libssh.org
 debug1: kex: host key algorithm: ecdsa-sha2-nistp256
 debug1: kex: server->client cipher: aes128-ctr MAC: hmac-sha2-256-etm@openssh.com compression: none
 debug1: kex: client->server cipher: aes128-ctr MAC: hmac-sha2-256-etm@openssh.com compression: none
 debug3: send packet: type 30
 debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
 debug3: receive packet: type 31
 debug1: Server host key: ecdsa-sha2-nistp256 SHA256:gxjoFPjc62jLhS2RPUXgYjvcsxHrCQT2+Hp3HWncNTI
 debug3: hostkeys_foreach: reading file "/root/.ssh/known_hosts"
 debug3: record_hostkey: found key type ECDSA in file /root/.ssh/known_hosts:1
 debug3: load_hostkeys: loaded 1 keys from 192.168.70.50
 debug1: Host '192.168.70.50' is known and matches the ECDSA host key.
 debug1: Found key in /root/.ssh/known_hosts:1
 debug3: send packet: type 21
 debug2: set_newkeys: mode 1
 debug1: rekey after 4294967296 blocks
 debug1: SSH2_MSG_NEWKEYS sent
 debug1: expecting SSH2_MSG_NEWKEYS
 debug3: receive packet: type 21
 debug1: SSH2_MSG_NEWKEYS received
 debug2: set_newkeys: mode 0
 debug1: rekey after 4294967296 blocks
 debug2: key: /root/.ssh/id_rsa ((nil))
 debug2: key: /root/.ssh/id_dsa ((nil))
 debug2: key: /root/.ssh/id_ecdsa ((nil))
 debug2: key: /root/.ssh/id_ed25519 ((nil))
 debug3: send packet: type 5
 debug3: receive packet: type 7
 debug1: SSH2_MSG_EXT_INFO received
 debug1: kex_input_ext_info: server-sig-algs=
 debug3: receive packet: type 6
 debug2: service_accept: ssh-userauth
 debug1: SSH2_MSG_SERVICE_ACCEPT received
 debug3: send packet: type 50
 debug3: receive packet: type 51
 debug1: Authentications that can continue: publickey,password,keyboard-interactive
 debug3: start over, passed a different list publickey,password,keyboard-interactive
 debug3: preferred gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive,password
 debug3: authmethod_lookup publickey
 debug3: remaining preferred: keyboard-interactive,password
 debug3: authmethod_is_enabled publickey
 debug1: Next authentication method: publickey
 debug1: Trying private key: /root/.ssh/id_rsa
 debug3: no such identity: /root/.ssh/id_rsa: No such file or directory
 debug1: Trying private key: /root/.ssh/id_dsa
 debug3: no such identity: /root/.ssh/id_dsa: No such file or directory
 debug1: Trying private key: /root/.ssh/id_ecdsa
 debug3: no such identity: /root/.ssh/id_ecdsa: No such file or directory
 debug1: Trying private key: /root/.ssh/id_ed25519
 debug3: no such identity: /root/.ssh/id_ed25519: No such file or directory
 debug2: we did not send a packet, disable method
 debug3: authmethod_lookup keyboard-interactive
 debug3: remaining preferred: password
 debug3: authmethod_is_enabled keyboard-interactive
 debug1: Next authentication method: keyboard-interactive
 debug2: userauth_kbdint
 debug3: send packet: type 50
 debug2: we sent a keyboard-interactive packet, wait for reply
 debug3: receive packet: type 60
 debug2: input_userauth_info_req
 debug2: input_userauth_info_req: num_prompts 1
 Password:

Transfer a file using SFTP

Find a suitable file to transfer, say /var/log/proton/nsxapi.log
 root@nsxtmgr03: cd /var/log/proton
 root@nsxtmgr03:/var/log/proton# ls -larth nsxapi.log
 -rw-r----- 1 uproton uproton 79M Jun 28 20:15 nsxapi.log
  
 Move to the Destination Directory and transfer the sample file:
 sftp> cd /tmp
 sftp> put nsxapi.log
 Uploading nsxapi.log to /tmp/nsxapi.log
 nsxapi.log                                    100%   78MB   4.6MB/s   00:17
  
 Verify the file has been successfully transferred:
 sftp> ls -larth
 drwx------    0 0        0             60B Jun 26 20:49 systemd-private-5f7f479451b5457ea04c5800459695c7-systemd-timesyncd.service-BMdu2q
 drwxrwxrwt    0 0        0             40B Jun 26 20:49 .Test-unix
 drwxrwxrwt    0 0        0             40B Jun 26 20:49 .font-unix
 drwxrwxrwt    0 0        0             40B Jun 26 20:49 .XIM-unix
 drwxrwxrwt    0 0        0             40B Jun 26 20:49 .ICE-unix
 drwxrwxrwt    0 0        0             40B Jun 26 20:49 .X11-unix
 drwx------    0 0        0             60B Jun 26 20:49 systemd-private-5f7f479451b5457ea04c5800459695c7-systemd-resolved.service-xbVg5u
 drwx------    0 0        0             60B Jun 26 20:49 systemd-private-5f7f479451b5457ea04c5800459695c7-systemd-networkd.service-Cobwbl
 drwxr-xr-x    0 0        0            4.0K Jun 26 20:51 ..
 drwx------    0 0        0             40B Jun 26 20:51 vmware-root_293-2084453149
 drwxr-x---    0 0        0             60B Jun 28 17:29 cluster-node-backups
 drwxr-x---    0 0        0             60B Jun 28 17:32 ccp-backups
 drwxrwxrwt    0 0        0            280B Jun 28 20:11 .
 -rw-r-----    0 0        0           78.4M Jun 28 20:16 nsxapi.log <<<<<<<<<

Here is an example of a backup failure reported from the NSX-T Manager UI: “backup operation unknown error”

There isn’t enough detail to identify root cause of the failure. Let’s see what we can learn from the CLI and NSX-T Manager logs.

root@nsxtmgr03:/#tail -f ./var/log/proton/nsxapi.log

2019-06-30T20:37:23.862Z  INFO task-executor-20 BackupGenerationServiceImpl - SYSTEM [nsx@6876 comp="nsx-manager" subcomp="manager"] Backing up using cluster filename [/cluster-node-backups/2.4.1.0.0.13716579-e7ef1642-0874-02fb-c93c-ba978da1f374-192.168.110.19/backup-2019-06-30T20_37_23UTC/cluster_backup-e7ef1642-0874-02fb-c93c-ba978da1f374-192.168.110.19-nsx-manager.tar, /cluster-node-backups/2.4.1.0.0.13716579-e7ef1642-0874-02fb-c93c-ba978da1f374-192.168.110.19/backup-2019-06-30T20_37_23UTC/cluster_backup-e7ef1642-0874-02fb-c93c-ba978da1f374-192.168.110.19-nsx-policy-manager.tar], node filename /cluster-node-backups/2.4.1.0.0.13716579-e7ef1642-0874-02fb-c93c-ba978da1f374-192.168.110.19/backup-2019-06-30T20_37_23UTC/node_backup-e7ef1642-0874-02fb-c93c-ba978da1f374-192.168.110.19.tar to URI sftp://192.168.70.50:22/tmp
 2019-06-30T20:37:23.863Z  INFO task-executor-20 BackupGenerationServiceImpl - SYSTEM [nsx@6876 comp="nsx-manager" subcomp="manager"] Authenticating with: fingerprint "SHA256:gxjoFPjc62jLhS2RPUXgYjvcsxHrCQT2+Hp3HWncNTI"
 2019-06-30T20:37:23.863Z  INFO task-executor-20 GenerationUtilities - SYSTEM [nsx@6876 comp="nsx-manager" subcomp="manager"] Creating remote directory sftp://192.168.70.50:22/tmp/cluster-node-backups
 2019-06-30T20:37:23.867Z  INFO task-executor-20 NapiBackupGenerationServiceImpl - SYSTEM [nsx@6876 comp="nsx-manager" subcomp="manager"] Issuing command http://127.0.0.1:7441/api/v1/node/file-store?action=create_remote_directory
 2019-06-30T20:37:26.477Z  INFO pool-198-thread-1 CorfuDbConnector - - [nsx@6876 comp="nsx-manager" subcomp="manager"] The database seems operational.
 2019-06-30T20:37:26.670Z  INFO http-nio-127.0.0.1-7440-exec-4 PreAuthenticatedAuthenticationProvider - - [nsx@6876 comp="nsx-manager" subcomp="manager"] User YWRtaW4=. Granted authorities: 'c3VwZXJ1c2Vycw== '
 2019-06-30T20:37:26.670Z  INFO http-nio-127.0.0.1-7440-exec-1 PreAuthenticatedAuthenticationProvider - - [nsx@6876 comp="nsx-manager" subcomp="manager"] User admin. Granted authorities: ''
 2019-06-30T20:37:26.674Z  INFO http-nio-127.0.0.1-7440-exec-1 PreAuthenticatedAuthenticationProvider - - [nsx@6876 comp="nsx-manager" subcomp="manager"] User admin. Granted authorities: ''
 2019-06-30T20:37:26.747Z  INFO http-nio-127.0.0.1-7440-exec-1 FabricNodeFacadeImpl - - [nsx@6876 comp="nsx-manager" subcomp="manager"] Listing nodes with request parameters NodeListRequestParametersDto= { ,
  pageSize: 1000,
  sortAscending: null }
 2019-06-30T20:37:26.800Z  WARN http-nio-127.0.0.1-7440-exec-4 FacadeInterceptorHelperImpl - - [nsx@6876 comp="nsx-manager" subcomp="manager"] API result did not pass validationField level validation errors: {value backup of property operation_type is not one of the allowed values [NONE, BACKUP]}
 2019-06-30T20:37:26.811Z  INFO http-nio-127.0.0.1-7440-exec-4 AuditingServiceImpl - SYSTEM [nsx@6876 audit="true" comp="nsx-manager" reqId="349ccc02-1080-4927-8353-508ce18053e4" subcomp="manager"] UserName="admin", ModuleName="BackupConfiguration", Operation="GetBackupStatus", Operation status="success"
 2019-06-30T20:37:27.152Z  INFO http-nio-127.0.0.1-7440-exec-1 FabricNodeFacadeImpl - - [nsx@6876 comp="nsx-manager" subcomp="manager"] List node result returned with size 5
 2019-06-30T20:37:27.161Z  INFO http-nio-127.0.0.1-7440-exec-1 AuditingServiceImpl - FABRIC [nsx@6876 audit="true" comp="nsx-manager" reqId="b8df962f-cb31-490f-9654-6017d57be37a" subcomp="manager"] UserName="admin", ModuleName="FabricNode", Operation="ListNodes", Operation status="success", New value=[{"page_size":1000}]
 2019-06-30T20:37:27.179Z  INFO http-nio-127.0.0.1-7440-exec-5 PreAuthenticatedAuthenticationProvider - - [nsx@6876 comp="nsx-manager" subcomp="manager"] User admin. Granted authorities: ''
 2019-06-30T20:37:27.179Z  INFO http-nio-127.0.0.1-7440-exec-5 PreAuthenticatedAuthenticationProvider - - [nsx@6876 comp="nsx-manager" subcomp="manager"] User admin. Granted authorities: ''
 2019-06-30T20:37:27.260Z  INFO http-nio-127.0.0.1-7440-exec-5 AuditingServiceImpl - SYSTEM [nsx@6876 audit="true" comp="nsx-manager" reqId="c7ebebc5-b933-4f96-8884-c960b49e7153" subcomp="manager"] UserName="admin", ModuleName="ClusterManagement", Operation="ListClusterNodeConfigs", Operation status="success"
 2019-06-30T20:37:28.456Z ERROR task-executor-20 NapiBackupGenerationServiceImpl - SYSTEM [nsx@6876 comp="nsx-manager" errorCode="MP29256" subcomp="manager"]
 org.springframework.web.client.HttpClientErrorException: 400 Bad Request
         at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:94) ~[spring-web-5.0.8.RELEASE.jar:5.0.8.RELEASE]
         at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:79) ~[spring-web-5.0.8.RELEASE.jar:5.0.8.RELEASE]
         at org.springframework.web.client.ResponseErrorHandler.handleError(ResponseErrorHandler.java:63) ~[spring-web-5.0.8.RELEASE.jar:5.0.8.RELEASE]
         at org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:766) ~[spring-web-5.0.8.RELEASE.jar:5.0.8.RELEASE]
         at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:724) ~[spring-web-5.0.8.RELEASE.jar:5.0.8.RELEASE]
         at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:680) ~[spring-web-5.0.8.RELEASE.jar:5.0.8.RELEASE]
         at org.springframework.web.client.RestTemplate.postForEntity(RestTemplate.java:466) ~[spring-web-5.0.8.RELEASE.jar:5.0.8.RELEASE]
         at com.vmware.nsx.management.backup.service.impl.NapiBackupGenerationServiceImpl.callHelper(NapiBackupGenerationServiceImpl.java:272) ~[backup-1.0.jar:?]
         at com.vmware.nsx.management.backup.service.impl.NapiBackupGenerationServiceImpl.callHelper(NapiBackupGenerationServiceImpl.java:242) ~[backup-1.0.jar:?]
         at com.vmware.nsx.management.backup.service.impl.NapiBackupGenerationServiceImpl.createRemoteDirectory(NapiBackupGenerationServiceImpl.java:496) ~[backup-1.0.jar:?]
         at com.vmware.nsx.management.backup.impl.GenerationUtilities.createRemotePathTo(GenerationUtilities.java:167) ~[backup-1.0.jar:?]
         at com.vmware.nsx.management.backup.service.impl.BackupGenerationServiceImpl.createAndUploadFullClusterBackup(BackupGenerationServiceImpl.java:207) ~[backup-1.0.jar:?]
         at com.vmware.nsx.management.backup.facade.BackupConfigurationFacadeImpl.requestOnetimeBackup_aroundBody0(BackupConfigurationFacadeImpl.java:91) ~[backup-1.0.jar:?]
         at com.vmware.nsx.management.backup.facade.BackupConfigurationFacadeImpl$AjcClosure1.run(BackupConfigurationFacadeImpl.java:1) ~[backup-1.0.jar:?]
         at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149) ~[aspectjtools-1.8.13.jar:?]
         at com.vmware.nsx.management.container.application.FacadeInterceptorHelperImpl.proceed_aroundBody0(FacadeInterceptorHelperImpl.java:211) ~[nsx-framework-1.0.jar:?]
         at com.vmware.nsx.management.container.application.FacadeInterceptorHelperImpl.proceed_aroundBody1$advice(FacadeInterceptorHelperImpl.java:87) ~[nsx-framework-1.0.jar:?]
         at com.vmware.nsx.management.container.application.FacadeInterceptorHelperImpl.proceed_aroundBody2(FacadeInterceptorHelperImpl.java:1) ~[nsx-framework-1.0.jar:?]
         at com.vmware.nsx.management.container.application.FacadeInterceptorHelperImpl.proceed_aroundBody3$advice(FacadeInterceptorHelperImpl.java:87) ~[nsx-framework-1.0.jar:?]
         at com.vmware.nsx.management.container.application.FacadeInterceptorHelperImpl.proceed(FacadeInterceptorHelperImpl.java:1) ~[nsx-framework-1.0.jar:?]
         at com.vmware.nsx.management.container.application.FacadeInterceptorHelperImpl.proceed(FacadeInterceptorHelperImpl.java:364) ~[nsx-framework-1.0.jar:?]
         at com.vmware.nsx.management.common.api.FacadeInterceptorAspect.annotatedMethod(FacadeInterceptorAspect.java:73) ~[common-1.0.jar:?]
         at com.vmware.nsx.management.backup.facade.BackupConfigurationFacadeImpl.requestOnetimeBackup(BackupConfigurationFacadeImpl.java:90) ~[backup-1.0.jar:?]
         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_192]
         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_192]
         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_192]
         at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_192]
         at com.vmware.nsx.management.common.api.task.AsyncApiTask.execute(AsyncApiTask.java:49) ~[common-1.0.jar:?]
         at com.vmware.nsx.management.common.api.task.TrackableTask.run(TrackableTask.java:89) ~[common-1.0.jar:?]
         at com.vmware.nsx.management.common.executor.TaskExecutorImpl$TaskWrapper$1.run(TaskExecutorImpl.java:238) ~[common-1.0.jar:?]
         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_192]
         at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_192]
         at com.vmware.nsx.management.common.executor.TaskExecutorImpl$TaskWrapper.run(TaskExecutorImpl.java:271) ~[common-1.0.jar:?]
         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_192]
         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_192]
         at java.lang.Thread.run(Thread.java:748) [?:1.8.0_192]
 2019-06-30T20:37:28.491Z ERROR task-executor-20 NapiBackupGenerationServiceImpl - SYSTEM [nsx@6876 comp="nsx-manager" errorCode="MP99" subcomp="manager"] Unknown response code 36209
 2019-06-30T20:37:28.491Z ERROR task-executor-20 BackupGenerationServiceImpl - SYSTEM [nsx@6876 comp="nsx-manager" errorCode="MP99" subcomp="manager"] Cluster backup had error
 2019-06-30T20:37:28.507Z  INFO task-executor-20 PrincipalOwnerValidator - - [nsx@6876 comp="nsx-manager" subcomp="manager"] XXX Principal 'admin' attempts to delete or modify an unprotected object of type BackupOperationHistory. (createUser=system)
 2019-06-30T20:37:28.688Z ERROR task-executor-20 BackupConfigurationFacadeImpl - - [nsx@6876 comp="nsx-manager" errorCode="MP99" s2comp="backup-restore" subcomp="manager"] Cluster backup failed with Status [status=GENERIC_ERROR, statusDetail=Error negotiating with remote host: Unable to negotiate with 192.168.70.50 port 22: no matching MAC found. Their offer: hmac-md5
 Couldn't read packet: Connection reset by peer, startTime=1561927043865, endTime=1561927048491].
 2019-06-30T20:37:28.688Z ERROR task-executor-20 BackupConfigurationFacadeImpl - - [nsx@6876 comp="nsx-manager" errorCode="MP99" subcomp="manager"] Status [status=GENERIC_ERROR, statusDetail=Error negotiating with remote host: Unable to negotiate with 192.168.70.50 port 22: no matching MAC found. Their offer: hmac-md5

SSH Fails since there is no no matching key exchange method found:

root@nsxtmgr03:~# nc -zv 192.168.70.50 22
 Connection to 192.168.70.50 22 port [tcp/ssh] succeeded!

 root@nsxtmgr03:~# ssh root@192.168.70.50
 Unable to negotiate with 192.168.70.50 port 22: no matching MAC found. Their offer: hmac-md5

 root@nsxtmgr03:~# ssh -vvv root@192.168.70.50
 OpenSSH_7.2p2 Ubuntu-4ubuntu2.6, OpenSSL 1.0.2p-fips  14 Aug 2018
 debug1: Reading configuration data /etc/ssh/ssh_config
 debug1: /etc/ssh/ssh_config line 19: Applying options for *
 debug2: resolving "192.168.70.50" port 22
 debug2: ssh_connect_direct: needpriv 0
 debug1: Connecting to 192.168.70.50 [192.168.70.50] port 22.
 debug1: Connection established.
 debug1: permanently_set_uid: 0/0
 debug1: key_load_public: No such file or directory
 debug1: identity file /root/.ssh/id_rsa type -1
 debug1: key_load_public: No such file or directory
 debug1: identity file /root/.ssh/id_rsa-cert type -1
 debug1: key_load_public: No such file or directory
 debug1: identity file /root/.ssh/id_dsa type -1
 debug1: key_load_public: No such file or directory
 debug1: identity file /root/.ssh/id_dsa-cert type -1
 debug1: key_load_public: No such file or directory
 debug1: identity file /root/.ssh/id_ecdsa type -1
 debug1: key_load_public: No such file or directory
 debug1: identity file /root/.ssh/id_ecdsa-cert type -1
 debug1: key_load_public: No such file or directory
 debug1: identity file /root/.ssh/id_ed25519 type -1
 debug1: key_load_public: No such file or directory
 debug1: identity file /root/.ssh/id_ed25519-cert type -1
 debug1: Enabling compatibility mode for protocol 2.0
 debug1: Local version string SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.6
 debug1: Remote protocol version 2.0, remote software version OpenSSH_7.8
 debug1: match: OpenSSH_7.8 pat OpenSSH* compat 0x04000000
 debug2: fd 3 setting O_NONBLOCK
 debug1: Authenticating to 192.168.70.50:22 as 'root'
 debug3: hostkeys_foreach: reading file "/root/.ssh/known_hosts"
 debug3: record_hostkey: found key type ECDSA in file /root/.ssh/known_hosts:1
 debug3: load_hostkeys: loaded 1 keys from 192.168.70.50
 debug3: order_hostkeyalgs: prefer hostkeyalgs: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
 debug3: send packet: type 20
 debug1: SSH2_MSG_KEXINIT sent
 debug3: receive packet: type 20
 debug1: SSH2_MSG_KEXINIT received
 debug2: local client KEXINIT proposal
 debug2: KEX algorithms: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,ext-info-c
 debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa
 debug2: ciphers ctos: aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com
 debug2: ciphers stoc: aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com
 debug2: MACs ctos: hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha2-256,hmac-sha2-512
 debug2: MACs stoc: hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha2-256,hmac-sha2-512
 debug2: compression ctos: none,zlib@openssh.com,zlib
 debug2: compression stoc: none,zlib@openssh.com,zlib
 debug2: languages ctos:
 debug2: languages stoc:
 debug2: first_kex_follows 0
 debug2: reserved 0
 debug2: peer server KEXINIT proposal
 debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1
 debug2: host key algorithms: rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256,ssh-ed25519
 debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
 debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
 debug2: MACs ctos: hmac-md5
 debug2: MACs stoc: hmac-md5
 debug2: compression ctos: none
 debug2: compression stoc: none
 debug2: languages ctos:
 debug2: languages stoc:
 debug2: first_kex_follows 0
 debug2: reserved 0
 debug1: kex: algorithm: curve25519-sha256@libssh.org
 debug1: kex: host key algorithm: ecdsa-sha2-nistp256
 Unable to negotiate with 192.168.70.50 port 22: no matching MAC found. Their offer: hmac-md5

For a successful connection, there must be at least one mutually-supported choice for each parameter.

If the client and server are unable to agree on a mutual set of parameters then the connection will fail. OpenSSH (7.0 and greater) will produce an error message like this:

Unable to negotiate with legacyhost: no matching key exchange method found.
Their offer: hmac-md5

In this scenario, NSX-T Manager supports the following ciphers/MACs:

root@nsxtmgr03:~# ssh -Q cipher
 3des-cbc
 blowfish-cbc
 cast128-cbc
 arcfour
 arcfour128
 arcfour256
 aes128-cbc
 aes192-cbc
 aes256-cbc
 rijndael-cbc@lysator.liu.se
 aes128-ctr
 aes192-ctr
 aes256-ctr
 aes128-gcm@openssh.com
 aes256-gcm@openssh.com
 chacha20-poly1305@openssh.com

 root@nsxtmgr03:~# ssh -Q mac
 hmac-sha1
 hmac-sha1-96
 hmac-sha2-256
 hmac-sha2-512
 hmac-md5
 hmac-md5-96
 hmac-ripemd160
 hmac-ripemd160@openssh.com
 umac-64@openssh.com
 umac-128@openssh.com
 hmac-sha1-etm@openssh.com
 hmac-sha1-96-etm@openssh.com
 hmac-sha2-256-etm@openssh.com
 hmac-sha2-512-etm@openssh.com
 hmac-md5-etm@openssh.com
 hmac-md5-96-etm@openssh.com
 hmac-ripemd160-etm@openssh.com
 umac-64-etm@openssh.com
 umac-128-etm@openssh.com

The SFTP Server only supports the following ciphers/MACs:

root@photon-machine [ /etc ]# sshd -T | grep "(ciphers|macs)"
 ciphers chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
 macs hmac-md5

This is a typical failure, where either available ciphers or MAC don’t match what was offered. The solution is to enable additional MACs on the SFTP Server, which is currently setup to support only the weaker hmac-md5 MAC method, which is not supported by NSX-T Manager.

In summary, by using the NSX-T MAnager CLI, to initiate the SSH and SFTP sessions in verbose mode, and by reviewing NSX-T Manager logs, we were able identify root cause of the “backup operation unknown error”, as reported by the UI.