I'm getting several errors in the logs every couple of days on a Windows 2003 Server that has to be rebooted to resolve this issue. The system cannot be rebuilt at this moment but will eventually once projects and time are available for the development team to reconfigure all of the IBM software I have a fresh VM waiting for that day.
A little background on the server:
It was a physical machine that was converted P2V and is now a guest on Hyper-V 2012 R2.
The errors occur at random, sometimes twice in the same day but usually days to a week or so in between.
The previous PM had capacitors bulge on the SCSI controller and is now completely dead. The machine was converted prior to me working at the company. It used to be on a domain several years ago but was removed due to restructuring and has been on a workgroup since then. I have made several backups of the machine throughout my cleaning and update processes.
Here's the weird part: Sometimes when it gets errors I can RDP to the VM but can't ping out or access anything outside of the machine and the windows firewall is already off. Other times I can't reach the VM via RDP and have to access it via the host machine to reboot it. When I reboot it I have to wait for CICSPRNT.exe to start running and then kill the process and run the developers script to start the CICSPRNT process again.
The system is running Windows Server 2003 R2 standard edition SP 2 and also has integration services installed for Hyper-V. It is running IBM CICS Transaction Gateway v6.0, websphere express 6, websphere developer for zseries 6.0 EGL COBOL Gen, IBM Rational Application Developer V6.0, IBM Rational Agent Controller, toolkit for Oracle 1.2.4, Opttech Sort Version 2003a, pervasive data integrator, TX series for Multiplatforms v6.1
What I've done so far:
scanned VM with multiple AV/Malware/Spyware products and the system is clean.
sfc /scannow did not find any corrupted system files
chkdsk didn't find any issues
Since it looked like the virtual NIC's had limited connectivity I wrote a script to rebuild winsock and networking. (I'll post this towards the bottom.)
repaired and cleaned the registry using Glary Utilities (Yes I have a backup of before)
installed all available Windows updates and Java updates.
Below are the errors and interesting info from the event viewer:
ERRORS
The description for Event ID ( 5555 ) in Source ( CICSTS ) cannot be found. The local computer may not have the necessary registry information or message DLL files to display messages from a remote computer. You may be able to use the /AUXSOURCE= flag to retrieve this description; see Help and Support for details. The following information is part of the event: ERZ025001E/0081 1/28/2014 04:29:51.890625000 PROD1 5216/0001 : Unsuccessful open of file: '/.:/cics/sfs/DELLAZ5'/'dfhcfil', index: 'cfilkey', SFS code - 1997905970 'ENC-sfs-0050: File by this name does not exist.'.
The description for Event ID ( 5555 ) in Source ( CICSTS ) cannot be found. The local computer may not have the necessary registry information or message DLL files to display messages from a remote computer. You may be able to use the /AUXSOURCE= flag to retrieve this description; see Help and Support for details. The following information is part of the event: ERZ080005E/0801 1/28/2014 00:00:15.421875000 DEV 9284/0001 : Abnormal termination U8005. XA_OPEN returned a Resource Manager error when opening 'Oracle Single Phase' using XA_OPEN string 'Oracle_XA+Acc=P/riuser/######+SesTm=20+SqlNet=TSIRPDB+DbgFl=15'. ' SQLCODE -12560, ORA-12560: TNS:protocol adapter error
Event ID 7034
The IBM CICS Transaction Gateway service terminated unexpectedly. It has done this 1 time(s).
For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
INFO
The description for Event ID ( 5557 ) in Source ( CICSTS ) cannot be found. The local computer may not have the necessary registry information or message DLL files to display messages from a remote computer. You may be able to use the /AUXSOURCE= flag to retrieve this description; see Help and Support for details. The following information is part of the event: ERZ058413I/0001 1/28/2014 04:29:52 PROD1 : Service 'cics.PROD1' has started successfully..
The description for Event ID ( 0 ) in Source ( SymTrackService ) cannot be found. The local computer may not have the necessary registry information or message DLL files to display messages from a remote computer. You may be able to use the /AUXSOURCE= flag to retrieve this description; see Help and Support for details. The following information is part of the event: Service started/resumed.
The description for Event ID ( 28712 ) in Source ( Encina ) cannot be found. The local computer may not have the necessary registry information or message DLL files to display messages from a remote computer. You may be able to use the /AUXSOURCE= flag to retrieve this description; see Help and Support for details. The following information is part of the event: 1 00868 14/01/28-04:29:43.140625 502c5448 A Initialized ... 1/28/2014 4:29:43 AMdev
Script to rebuild winsock and networking
color EC
cls
@echo off
echo Be sure to write down your network configuration and reconfigure it before rebooting!
echo If you need to do this now then either minimize or close this window and start it later after doing so.
pause
echo restarting winsock
echo For more information of what this script is doing please visit: support.microsoft.com/kb/811259
netsh winsock reset
pause
echo Resetting TCP/IP
echo For more information of what this script is doing please visit: support.microsoft.com/kb/31758
echo A log file will be created at the location "c:\IPResetLog.txt"
netsh int ip reset c:\IPResetLog.txt
echo Keep this window open and reconfigure your Network settings NOW!!!
pause
echo creating ARP table log in c:\ARPLog.txt
arp -a > c:\ARPLog.txt
echo flushing ARP tables
arp -d
echo creating NBTStat log with ALL current info in c:\NBTStat.txt
nbtstat -a > c:\NBTStat.txt
echo "!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!" >> c:\NBTStat.txt
nbtstat -A >> c:\NBTStat.txt
echo "!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!" >> c:\NBTStat.txt
nbtstat -r >> c:\NBTStat.txt
echo "!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!" >> c:\NBTStat.txt
nbtstat -s >> c:\NBTStat.txt
echo "!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!" >> c:\NBTStat.txt
nbtstat -S >> c:\NBTStat.txt
echo Purging and reloading the remote cache name table
nbtstat -R
echo Sending Name Release packets to WINS and then, starts Refresh
nbtstat -RR
echo Purging the DNS Resolver cache
ipconfig /flushdns
echo Refreshing all DHCP leases and re-registering DNS names
ipconfig /registerdns
echo REMEMBER to re-configure all network settings before rebooting or
echo if you are remotely working on this box you may lose connectivity and need physical access to this box
pause
echo This computer will reboot if you press any key, if you don't want to reboot now close this window.
shutdown /r /d p:0:0
