Showing posts with label ashwinpawar. Show all posts
Showing posts with label ashwinpawar. Show all posts

Saturday, 25 March 2017

Could not open an NDMP connection to host. Please verify the NDMP server is running on that host when adding NAS iDA to CommVault v10 & 11.

Following error is seen when adding a NAS iDA NetApp 7-mode client in the CommCell v10.



Customer reported : Telnet to Port 10000 on the NAS Host works.

Telnet to Port 10000 is a good troubleshooting step, however in cases like this, it isn't helpful.

Reason : Telnet to Port '10000' only signifies NDMP server is listening, but we need to find out why is it refusing to speak, and reporting this error - 'Please verify NDMP server is running on that host'.

Cause: NDMP server [FILER] is definitely listening on Port 10000, but it is refusing connection requests, b'cos its NDMP memory pool is Full.

Initial troubleshooting steps:
Ensure NDMP is turned on.
FILER>options ndmpd.enable
ndmpd.enable           on

That indicates, NDMP server is definitely listening, however there is something which is preventing the communication.

First clue, as reported in EvMgrS.log: Connection refused by the NDMP server



This is the first clue, which kind of give us an indication that the NDMP server is unable to accept any more connections.

Second clue, look for stale NDMP sessions on the filer:
FILER>backup status
This will tell you if there is any existing NDMP backups running on the filer. In this case, there were none.
FILER>ndmpd status

In this particular case, we found there were lot of stale NDMP sessions sitting idle and doing nothing, in other words, they were simply holding on to NDMP memory and hence stopping NDMP to accept new connection.

Please note, If there is nothing in the backup status output, then you can safely kill all the stale ndmp sessions that are shown via ndmpd status output.

Solution:
1. Kill all the stale sessions.
FILER>ndmpd killall

2. Turn off and turn On NDMP [This step may be required, but not necessary]
FILER>options ndmpd.enable off
FILER>options ndmpd.enable on

3. Try to add the NDMP NAS iDA client once again, this time it should succeed.


Please Note: For more information on available ndmpd commands, simply type 'ndmpd' on the putty console for 7-mode;
FILER> ndmpd
usage:  ndmpd [on|off|status|probe [<session #>]|kill <session #>|killall|

Ports usage during Out of Place restore using CommVault IntelliSnap & NDMP for v10 & v11

NetApp CDOT: [IntelliSnap Restore]

Out-of-place IntelliSnap restore to 'LINUX', log files to look at:

On CommServe:
File    : CVNasSnapRestore.log
File    : fsIndexedRestore.log
File    : JobManager.log
File    : StartRestore.log


On LINUX BOX:
File    : CVNRDS.log
Location: /opt/simpana/iDataAgent/CVNRDS

Source & destination machines during Intelli Snap restore: Out of place
Source: FILER
Dest  :   LINUX

CVNasSnapRestore.log: Ports during Data connection establishment
6968  187c  03/30 21:30:22 9471 ndmp_v4.cpp 2532 NDMP_DATA_LISTEN: successful
6968  187c  03/30 21:30:22 9471 Connect Data Servers() - Sending CAB prepare to destination ...
6968  187c  03/30 21:30:22 9471 Connect Data Servers() - Sending data_connect to destination ...
6968  187c  03/30 21:30:22 9471 ndmp_v4.cpp 2143 NDMP_DATA_CONNECT(tcp):
6968  187c  03/30 21:30:22 9471 ndmp_v4.cpp 2147 --- address[0xc0a80064] port[18601] <---- On FILER
6968  187c  03/30 21:30:22 9471 ndmp_v4.cpp 2147 --- address[0xc0a8003c] port[18601] <---- On FILER

[root@redhatcentos Log_Files]# netstat -anp | grep 192.168.0.10
tcp        0      0 192.168.0.25:55564          192.168.0.100:18601         ESTABLISHED 13609/CVNRDS    <-----On LINUX

-----------------------------------------------------------------------------------------------------
NetApp CDOT: [NDMP Restore]

Out-of-place NDMP restore to 'LINUX', log files to look at:

On CommServ:
File    : CVD.log
File    : CVNdmpRemoteServer.log
File    : fsIndexedRestore.log
File    : JobManager.log
File    : MediaManager.log
File    : nasRestore.log
File    : StartRestore.log

On LINUX BOX:
File    : CVNRDS.log
Location: /opt/simpana/iDataAgent/CVNRDS

Source & destination machines during NDMP restore: Out of place
Source: MA
Dest  :  LINUX  

nasrestore.log: Ports during DATA connection establishment:
1436  1338  03/30 21:11:40 9470 ndmp_v4.cpp 2735 NDMP_MOVER_LISTEN: successful
1436  1338  03/30 21:11:40 9470 ndmp_v4.cpp 2742 --- address[0xc0a8000a] port[60700] <--- On MA
1436  1338  03/30 21:11:40 9470 ndmp_v4.cpp 2143 NDMP_DATA_CONNECT(tcp):
1436  1338  03/30 21:11:40 9470 ndmp_v4.cpp 2147 --- address[0xc0a8000a] port[60700] <----On MA
1436  1338  03/30 21:11:40 9470 ndmp_v4.cpp 2192 NDMP_DATA_CONNECT: successful

Ports on  MA:192.168.0.10
C:\Windows\system32>netstat -anp tcp | findstr 192.168.0.25
  TCP    192.168.0.10:59117     192.168.0.25:42372     ESTABLISHED
  TCP    192.168.0.10:60341     192.168.0.25:8402      ESTABLISHED
  TCP    192.168.0.10:60523     192.168.0.25:57990     ESTABLISHED
  TCP    192.168.0.10:60539     192.168.0.25:59445     ESTABLISHED
  TCP    192.168.0.10:60610     192.168.0.25:34059     ESTABLISHED
  TCP    192.168.0.10:60700     192.168.0.25:40163     TIME_WAIT   <-----Data Pipe connection

Ports on Linux:192.168.0.25
[root@redhatcentos Log_Files]# netstat -anp | grep 192.168.0.10
tcp        0      0 192.168.0.25:8400           192.168.0.10:60655          TIME_WAIT   -                  
tcp        0      0 192.168.0.25:34059          192.168.0.10:60610          ESTABLISHED 3134/cvd           
tcp        0      0 192.168.0.25:40163          192.168.0.10:60700          ESTABLISHED 12117/CVNRDS    <-----Data Pipe connection