Monday, March 4, 2013

Killing a Windows Service that Seems to be Hanged on Stopping or Starting

Windows Services can sometimes stuck in the stopping or starting state. I have faced this issue with IBM Datacap Server services in development environment.  Rebooting workstation clears the problem. But following can be done to kill  hanged service and avoid reboot workstation or server.


  • Get the Service Name. Go to Services(Run--> services.msc)
       


  • Find out the Process ID using sc queryex on the Windows Service.
                    C:\Users\Admin>sc queryex DCTMS
    SERVICE_NAME: DCTMS
                    TYPE                  : 10  WIN32_OWN_PROCESS
                    STATE               : 3  STOP_PENDING
                                                      (STOPPABLE, PAUSABLE, IGNORES_SHUTDOWN)
                   WIN32_EXIT_CODE      : 0  (0x0)
                   SERVICE_EXIT_CODE  : 0  (0x0)
                   CHECKPOINT        : 0x0
                   WAIT_HINT          : 0x0
                   PID                           : 4716
                   FLAGS                     :
  • The Process ID is 4716.  So now we can issue the taskkill /F command on the process.
         Note: in Windows 2008, if we are not running the command prompt with ‘run as administrator’, we  
         will get an access denied error on the delete.
            
              C:\Users\Admin>taskkill /f /pid 4716
   SUCCESS: The process with PID 4716 has been terminated.

         

Friday, March 1, 2013

Unable to stop WAS application server


Problem

IBM WebSphere application Server(WAS) failed to stop. When you stop the Websphere server by clicking stop server shortcut on desktop, it looks like server is stopped. But you still see the server1.pid file at below location - E:\Program Files\IBM\WebSphere\AppServer\profiles\AppSrv01\logs\server1. That means server is still running.


The following errors are logged in stopServer.log, which is WAS log file.

[23/09/11 16:39:40:751 CEST] 0000000a WsServerStop  E   ADMU3002E: Exception attempting to process server server1
[23/09/11 16:39:40:751 CEST] 0000000a WsServerStop  E   ADMU3007E: Exception com.ibm.websphere.management.exception.ConnectorException: ADMC0053E: The system cannot create a SOAP connector to connect to host <ServerName> at port 9080with SOAP connector security enabled.
[23/09/11 16:39:40:751 CEST] 0000000a WsServerStop  A   ADMU3007E: Exception com.ibm.websphere.management.exception.ConnectorException: ADMC0053E: The system cannot create a SOAP connector to connect to host <ServerName> at port 9080 with SOAP connector security enabled.
                at com.ibm.websphere.management.AdminClientFactory.createAdminClient(AdminClientFactory.java:486)
                …
                ... 27 more
Caused by: com.ibm.websphere.management.exception.ConnectorNotAvailableException: [SOAPException: faultCode=SOAP-ENV:Client; msg=Error opening socket: javax.net.ssl.SSLHandshakeException: com.ibm.jsse2.util.h: No trusted certificate found; targetException=java.lang.IllegalArgumentException: Error opening socket: javax.net.ssl.SSLHandshakeException: com.ibm.jsse2.util.h: No trusted certificate found]
                …
                ... 32 more
Caused by: [SOAPException: faultCode=SOAP-ENV:Client; msg=Error opening socket: javax.net.ssl.SSLHandshakeException: com.ibm.jsse2.util.h: No trusted certificate found; targetException=java.lang.IllegalArgumentException: Error opening socket: javax.net.ssl.SSLHandshakeException: com.ibm.jsse2.util.h: No trusted certificate found]
                …
                ... 33 more

[23/09/11 16:39:40:767 CEST] 0000000a AdminTool     A   ADMU0509I: The server "server1" cannot be reached. It appears to be stopped.
[23/09/11 16:39:40:767 CEST] 0000000a AdminTool     A   ADMU0211I: Error details may be seen in the file: E:\Program Files\IBM\WebSphere\AppServer\profiles\AppSrv01\logs\server1\stopServer.log
[23/09/11 16:39:40:939 CEST] 00000016 WindowsServic 3   inputRec == Service failed to stop.  stopServer return code -10
[23/09/11 16:39:40:939 CEST] 0000000a WindowsServic 1   retCode from childProcess.waitFor() is -10
[23/09/11 16:39:40:939 CEST] 0000000a WindowsServic 3   -- interrupt the thread that is collecting stdout
[23/09/11 16:39:41:939 CEST] 00000014 WindowsServic 3   readAndDisplayServiceLogFile(), caught InterruptedException
[23/09/11 16:39:41:939 CEST] 0000000a WindowsServic 3   exitCode = -10
[23/09/11 16:39:41:939 CEST] 0000000a WindowsServic 3   Service failed to stop.  stopServer return code -10



Cause

With WAS 6.1 or later, self-signed certificate will automatically be updated, and when user executes any administration commands after the update, the following message will be displayed.

----------------------------------------------------------------
*** SSL SIGNER EXCHANGE PROMPT ***
SSL signer from target host testserver.ibm.com is not found in trust
store /opt/IBM/WebSphere/AppServer/profiles/Dmgr01/etc/trust.p12.

Here is the signer information (verify the digest value matches what is
displayed at the server):

Subject DN: CN=testserver.ibm.com, O=IBM, C=US
Issuer DN: CN=testserver.ibm.com, O=IBM, C=US
Serial number: 1192512730
Expires: Wed Oct 15 14:32:10 JST 2008
SHA-1 Digest:
D3:4F:52:C1:17:54:C3:58:0E:07:D3:BD:4E:0E:5A:ED:4A:F8:59:08
MD5 Digest: 7B:2F:22:F6:85:EF:CE:08:D6:83:62:AB:61:E1:EC:BF

Add signer to the trust store now? (y/n)
----------------------------------------------------------------

If user manually executes stopServer command on DOS console to stop WAS application server, since he can see the message, he would be able to stop the application server by answering "y" to the message.
However if user tried to stop using Windows' "Service" GUI or a bach program which calls stopServer.bat so on, he didn't have a chance to see the message.


Resolving the problem

Run one of the following WAS admin commands to display the message so that you can accept to add signer to the trust store. After that, the message will not be displayed.
  • serverStatus
  • stopServer
  • stopManager
  • stopNode
  • wsadmin

1.      Open command prompt and go to following location.
E:\Program Files\IBM\WebSphere\AppServer\profiles\AppSrv01\bin
2.      Then execute following batch file
serverStatus.bat server1
3.      Pop will come up asking for log in credentials for WAS Admin user


Followings is complete output on command prompt


E:\Program Files\IBM\WebSphere\AppServer\profiles\AppSrv01\bin>serverStatus.bat
server1
ADMU0116I: Tool information is being logged in file E:\Program
           Files\IBM\WebSphere\AppServer\profiles\AppSrv01\logs\server1\serverSt
atus.log
ADMU0128I: Starting tool with the AppSrv01 profile
ADMU0500I: Retrieving server status for server1

*** SSL SIGNER EXCHANGE PROMPT ***
SSL signer from target host null is not found in trust store E:/Program Files/IB
M/WebSphere/AppServer/profiles/AppSrv01/etc/trust.p12.

Here is the signer information (verify the digest value matches what is displaye
d at the server):

Subject DN:    CN=<servername.Domain.comt, O=IBM, C=US
Issuer DN:     CN=<servername.Domain.comt, O=IBM, C=US
Serial number: 1315078236
Expires:       Sun Sep 02 21:30:36 CEST 2012
SHA-1 Digest:  AB:F9:9A:64:97:00:58:BC:B3:BE:1F:81:72:46:D5:38:00:A4:A6:B9
MD5 Digest:    CE:5C:DF:20:B5:E4:C9:64:F3:91:70:00:BF:E2:BF:27

Add signer to the trust store now? (y/n) y
A retry of the request may need to occur if the socket times out while waiting f
or a prompt response.  If the retry is required, note that the prompt will not b
e redisplayed if (y) is entered, which indicates the signer has already been add
ed to the trust store.
ADMU0508I: The Application Server "server1" is STARTED

E:\Program Files\IBM\WebSphere\AppServer\profiles\AppSrv01\bin>