Vmware vCenter: ESXi Server required, Remove form Inventory rather Reconnecting

Not sure why.

Might be because of the Site-to-Site VPN. 

However, the bandwidth is OK.

Network Monitor & SSH showing the ESXi Server, which is on another site, normally.

On the vCenter, few time required to  Remove form Inventory  rather Reconnecting. 

Such as patching it.

   

Connection Connect 10.0.0.1 Reconnecting a host will override any resource management changes that were made directly on the host while it was disconnected. To keep these changes, remove host from the inventory and add it again. Reconnect host?

Remove from Inventory Remove Host 10.0.0.1 Removing a host also removes all of its virtual machines, resource pools and vApps from the inventory of vCenter Server. In addition, past performance data, host level permissions, user created alarms, and the values of custom attributes will be deleted. Any vApps currently on the host will turn into resource pools. Remove this host?

Update