When you have a Azure Hybrid Connection setup the wrong way, you can delete the connection trough the UI but then the Hybrid Connection Manager still knows the connection.
If you for example are moving things around and you would like to keep the connection within the UI you will have to use PowerShell to alter the connections.
All connection strings needed can be found within the Azure Portal
Adding a Connection to the Hybrid Connection Manager
Add-HybridConnection -ConnectionString [ConnectionString]
Get all configured connections
or with specific properties
Get-HybridConnection | select -Property Uri
Get a specific connection
Get-HybridConnection -Uri [URI]
Get-HybridConnection -ConnectionString [ConnectionString]
Remove a specific connection
Remove-HybridConnection -Uri [URI]
Remove-HybridConnection -ConnectionString [ConnectionString]
Update connection string
Update-HybridConnection -ConnectionString [ConnectionString]