Earlier this evening, nearly all servers went down and our whole dedicated server would not respond to remote desktop, however I could ping it fine. In the end we had to get the box rebooted. After looking through the panel monitor logs I saw this:
Code: Select all
03/01/2009 22:49:56 System resource exceeded.
at System.Data.OleDb.OleDbCommand.ExecuteCommandTextForSingleResult(tagDBPARAMS dbParams, Object& executeResult)
at System.Data.OleDb.OleDbCommand.ExecuteCommandText(Object& executeResult)
at System.Data.OleDb.OleDbCommand.ExecuteCommand(CommandBehavior behavior, Object& executeResult)
at System.Data.OleDb.OleDbCommand.ExecuteReaderInternal(CommandBehavior behavior, String method)
at System.Data.OleDb.OleDbCommand.ExecuteNonQuery()
at MashToolsSharp.oledbClass.update_data(Object table, Object field, Object data, Object wherefld, Object wherematch, Object wherefld2, Object wherefld2match)
at UGCCMon.dbinterface.update_data(String table, String field, String data, String wherefld, String wherematch, String wherefld2, String wherefld2match) in C:\ugccmon\dbinterface.cs:line 77
at UGCCMon.Mon.check(dbinterface& sql, Boolean boolRemote, String sdefid, Boolean firstboot) in C:\ugccmon\Mon.cs:line 93
at UGCCMon.UGCCMon.monitor() in C:\ugccmon\Service1.cs:line 335
03/01/2009 22:50:56 System resource exceeded.
at System.Data.OleDb.OleDbCommand.ExecuteCommandTextForSingleResult(tagDBPARAMS dbParams, Object& executeResult)
at System.Data.OleDb.OleDbCommand.ExecuteCommandText(Object& executeResult)
at System.Data.OleDb.OleDbCommand.ExecuteCommand(CommandBehavior behavior, Object& executeResult)
at System.Data.OleDb.OleDbCommand.ExecuteReaderInternal(CommandBehavior behavior, String method)
at System.Data.OleDb.OleDbCommand.ExecuteReader(CommandBehavior behavior)
at System.Data.OleDb.OleDbCommand.ExecuteReader()
at MashToolsSharp.oledbClass.get_multidata(Object table, Object returnfld, Object wherefld, Object wherefldmatch, Object wherefld2, Object wherefldmatch2, Object sortfld, Boolean sortdesc, Int32 limit)
at UGCCMon.dbinterface.get_multidata(String table, String returnfld, String wherefld, String wherefldmatch, String wherefld2, String wherefldmatch2, String sortfld, Boolean sortdesc, Int32 limit) in C:\ugccmon\dbinterface.cs:line 40
at UGCCMon.Mon.check(dbinterface& sql, Boolean boolRemote, String sdefid, Boolean firstboot) in C:\ugccmon\Mon.cs:line 23
at UGCCMon.UGCCMon.monitor() in C:\ugccmon\Service1.cs:line 335
03/01/2009 22:51:56 System resource exceeded.
at System.Data.OleDb.OleDbCommand.ExecuteCommandTextForSingleResult(tagDBPARAMS dbParams, Object& executeResult)
at System.Data.OleDb.OleDbCommand.ExecuteCommandText(Object& executeResult)
at System.Data.OleDb.OleDbCommand.ExecuteCommand(CommandBehavior behavior, Object& executeResult)
at System.Data.OleDb.OleDbCommand.ExecuteReaderInternal(CommandBehavior behavior, String method)
at System.Data.OleDb.OleDbCommand.ExecuteReader(CommandBehavior behavior)
at System.Data.OleDb.OleDbCommand.ExecuteReader()
at MashToolsSharp.oledbClass.get_multidata(Object table, Object returnfld, Object wherefld, Object wherefldmatch, Object wherefld2, Object wherefldmatch2, Object sortfld, Boolean sortdesc, Int32 limit)
at UGCCMon.dbinterface.get_multidata(String table, String returnfld, String wherefld, String wherefldmatch, String wherefld2, String wherefldmatch2, String sortfld, Boolean sortdesc, Int32 limit) in C:\ugccmon\dbinterface.cs:line 40
at UGCCMon.Mon.check(dbinterface& sql, Boolean boolRemote, String sdefid, Boolean firstboot) in C:\ugccmon\Mon.cs:line 23
at UGCCMon.UGCCMon.monitor() in C:\ugccmon\Service1.cs:line 335
03/01/2009 22:52:56 System resource exceeded.
at System.Data.OleDb.OleDbCommand.ExecuteCommandTextForSingleResult(tagDBPARAMS dbParams, Object& executeResult)
at System.Data.OleDb.OleDbCommand.ExecuteCommandText(Object& executeResult)
at System.Data.OleDb.OleDbCommand.ExecuteCommand(CommandBehavior behavior, Object& executeResult)
at System.Data.OleDb.OleDbCommand.ExecuteReaderInternal(CommandBehavior behavior, String method)
at System.Data.OleDb.OleDbCommand.ExecuteReader(CommandBehavior behavior)
at System.Data.OleDb.OleDbCommand.ExecuteReader()
at MashToolsSharp.oledbClass.get_multidata(Object table, Object returnfld, Object wherefld, Object wherefldmatch, Object wherefld2, Object wherefldmatch2, Object sortfld, Boolean sortdesc, Int32 limit)
at UGCCMon.dbinterface.get_multidata(String table, String returnfld, String wherefld, String wherefldmatch, String wherefld2, String wherefldmatch2, String sortfld, Boolean sortdesc, Int32 limit) in C:\ugccmon\dbinterface.cs:line 40
at UGCCMon.Mon.check(dbinterface& sql, Boolean boolRemote, String sdefid, Boolean firstboot) in C:\ugccmon\Mon.cs:line 23
at UGCCMon.UGCCMon.monitor() in C:\ugccmon\Service1.cs:line 335
03/01/2009 22:53:56 System resource exceeded.
at System.Data.OleDb.OleDbConnectionInternal..ctor(OleDbConnectionString constr, OleDbConnection connection)
at System.Data.OleDb.OleDbConnectionFactory.CreateConnection(DbConnectionOptions options, Object poolGroupProviderInfo, DbConnectionPool pool, DbConnection owningObject)
at System.Data.ProviderBase.DbConnectionFactory.CreateNonPooledConnection(DbConnection owningConnection, DbConnectionPoolGroup poolGroup)
at System.Data.ProviderBase.DbConnectionFactory.GetConnection(DbConnection owningConnection)
at System.Data.ProviderBase.DbConnectionClosed.OpenConnection(DbConnection outerConnection, DbConnectionFactory connectionFactory)
at System.Data.OleDb.OleDbConnection.Open()
at MashToolsSharp.oledbClass..ctor(String ConnectString)
at UGCCMon.dbinterface..ctor(String dbtypeo, String connectstr) in C:\ugccmon\dbinterface.cs:line 23
at UGCCMon.UGCCMon.monitor() in C:\ugccmon\Service1.cs:line 333
03/01/2009 22:54:56 System resource exceeded.
at System.Data.OleDb.OleDbCommand.ExecuteCommandTextForSingleResult(tagDBPARAMS dbParams, Object& executeResult)
at System.Data.OleDb.OleDbCommand.ExecuteCommandText(Object& executeResult)
at System.Data.OleDb.OleDbCommand.ExecuteCommand(CommandBehavior behavior, Object& executeResult)
at System.Data.OleDb.OleDbCommand.ExecuteReaderInternal(CommandBehavior behavior, String method)
at System.Data.OleDb.OleDbCommand.ExecuteReader(CommandBehavior behavior)
at System.Data.OleDb.OleDbCommand.ExecuteReader()
at MashToolsSharp.oledbClass.get_multidata(Object table, Object returnfld, Object wherefld, Object wherefldmatch, Object wherefld2, Object wherefldmatch2, Object sortfld, Boolean sortdesc, Int32 limit)
at UGCCMon.dbinterface.get_multidata(String table, String returnfld, String wherefld, String wherefldmatch, String wherefld2, String wherefldmatch2, String sortfld, Boolean sortdesc, Int32 limit) in C:\ugccmon\dbinterface.cs:line 40
at UGCCMon.Mon.check(dbinterface& sql, Boolean boolRemote, String sdefid, Boolean firstboot) in C:\ugccmon\Mon.cs:line 23
at UGCCMon.UGCCMon.monitor() in C:\ugccmon\Service1.cs:line 335
03/01/2009 22:55:56 System resource exceeded.
at System.Data.OleDb.OleDbConnectionInternal..ctor(OleDbConnectionString constr, OleDbConnection connection)
at System.Data.OleDb.OleDbConnectionFactory.CreateConnection(DbConnectionOptions options, Object poolGroupProviderInfo, DbConnectionPool pool, DbConnection owningObject)
at System.Data.ProviderBase.DbConnectionFactory.CreateNonPooledConnection(DbConnection owningConnection, DbConnectionPoolGroup poolGroup)
at System.Data.ProviderBase.DbConnectionFactory.GetConnection(DbConnection owningConnection)
at System.Data.ProviderBase.DbConnectionClosed.OpenConnection(DbConnection outerConnection, DbConnectionFactory connectionFactory)
at System.Data.OleDb.OleDbConnection.Open()
at MashToolsSharp.oledbClass..ctor(String ConnectString)
at UGCCMon.dbinterface..ctor(String dbtypeo, String connectstr) in C:\ugccmon\dbinterface.cs:line 23
at UGCCMon.UGCCMon.monitor() in C:\ugccmon\Service1.cs:line 333
03/01/2009 22:56:56 System resource exceeded.
at System.Data.OleDb.OleDbConnectionInternal..ctor(OleDbConnectionString constr, OleDbConnection connection)
at System.Data.OleDb.OleDbConnectionFactory.CreateConnection(DbConnectionOptions options, Object poolGroupProviderInfo, DbConnectionPool pool, DbConnection owningObject)
at System.Data.ProviderBase.DbConnectionFactory.CreateNonPooledConnection(DbConnection owningConnection, DbConnectionPoolGroup poolGroup)
at System.Data.ProviderBase.DbConnectionFactory.GetConnection(DbConnection owningConnection)
at System.Data.ProviderBase.DbConnectionClosed.OpenConnection(DbConnection outerConnection, DbConnectionFactory connectionFactory)
at System.Data.OleDb.OleDbConnection.Open()
at MashToolsSharp.oledbClass..ctor(String ConnectString)
at UGCCMon.dbinterface..ctor(String dbtypeo, String connectstr) in C:\ugccmon\dbinterface.cs:line 23
at UGCCMon.UGCCMon.monitor() in C:\ugccmon\Service1.cs:line 333
03/01/2009 22:57:56 System resource exceeded.
at System.Data.OleDb.OleDbCommand.ExecuteCommandTextForSingleResult(tagDBPARAMS dbParams, Object& executeResult)
at System.Data.OleDb.OleDbCommand.ExecuteCommandText(Object& executeResult)
at System.Data.OleDb.OleDbCommand.ExecuteCommand(CommandBehavior behavior, Object& executeResult)
at System.Data.OleDb.OleDbCommand.ExecuteReaderInternal(CommandBehavior behavior, String method)
at System.Data.OleDb.OleDbCommand.ExecuteReader(CommandBehavior behavior)
at System.Data.OleDb.OleDbCommand.ExecuteReader()
at MashToolsSharp.oledbClass.get_multidata(Object table, Object returnfld, Object wherefld, Object wherefldmatch, Object wherefld2, Object wherefldmatch2, Object sortfld, Boolean sortdesc, Int32 limit)
at UGCCMon.dbinterface.get_multidata(String table, String returnfld, String wherefld, String wherefldmatch, String wherefld2, String wherefldmatch2, String sortfld, Boolean sortdesc, Int32 limit) in C:\ugccmon\dbinterface.cs:line 40
at UGCCMon.Mon.check(dbinterface& sql, Boolean boolRemote, String sdefid, Boolean firstboot) in C:\ugccmon\Mon.cs:line 23
at UGCCMon.UGCCMon.monitor() in C:\ugccmon\Service1.cs:line 335
03/01/2009 22:58:56 System resource exceeded.
at System.Data.OleDb.OleDbConnectionInternal..ctor(OleDbConnectionString constr, OleDbConnection connection)
at System.Data.OleDb.OleDbConnectionFactory.CreateConnection(DbConnectionOptions options, Object poolGroupProviderInfo, DbConnectionPool pool, DbConnection owningObject)
at System.Data.ProviderBase.DbConnectionFactory.CreateNonPooledConnection(DbConnection owningConnection, DbConnectionPoolGroup poolGroup)
at System.Data.ProviderBase.DbConnectionFactory.GetConnection(DbConnection owningConnection)
at System.Data.ProviderBase.DbConnectionClosed.OpenConnection(DbConnection outerConnection, DbConnectionFactory connectionFactory)
at System.Data.OleDb.OleDbConnection.Open()
at MashToolsSharp.oledbClass..ctor(String ConnectString)
at UGCCMon.dbinterface..ctor(String dbtypeo, String connectstr) in C:\ugccmon\dbinterface.cs:line 23
at UGCCMon.UGCCMon.monitor() in C:\ugccmon\Service1.cs:line 333
03/01/2009 22:59:56 System resource exceeded.
at System.Data.OleDb.OleDbConnectionInternal..ctor(OleDbConnectionString constr, OleDbConnection connection)
at System.Data.OleDb.OleDbConnectionFactory.CreateConnection(DbConnectionOptions options, Object poolGroupProviderInfo, DbConnectionPool pool, DbConnection owningObject)
at System.Data.ProviderBase.DbConnectionFactory.CreateNonPooledConnection(DbConnection owningConnection, DbConnectionPoolGroup poolGroup)
at System.Data.ProviderBase.DbConnectionFactory.GetConnection(DbConnection owningConnection)
at System.Data.ProviderBase.DbConnectionClosed.OpenConnection(DbConnection outerConnection, DbConnectionFactory connectionFactory)
at System.Data.OleDb.OleDbConnection.Open()
at MashToolsSharp.oledbClass..ctor(String ConnectString)
at UGCCMon.dbinterface..ctor(String dbtypeo, String connectstr) in C:\ugccmon\dbinterface.cs:line 23
at UGCCMon.UGCCMon.monitor() in C:\ugccmon\Service1.cs:line 333
03/01/2009 23:00:56 System resource exceeded.
at System.Data.OleDb.OleDbConnectionInternal..ctor(OleDbConnectionString constr, OleDbConnection connection)
at System.Data.OleDb.OleDbConnectionFactory.CreateConnection(DbConnectionOptions options, Object poolGroupProviderInfo, DbConnectionPool pool, DbConnection owningObject)
at System.Data.ProviderBase.DbConnectionFactory.CreateNonPooledConnection(DbConnection owningConnection, DbConnectionPoolGroup poolGroup)
at System.Data.ProviderBase.DbConnectionFactory.GetConnection(DbConnection owningConnection)
at System.Data.ProviderBase.DbConnectionClosed.OpenConnection(DbConnection outerConnection, DbConnectionFactory connectionFactory)
at System.Data.OleDb.OleDbConnection.Open()
at MashToolsSharp.oledbClass..ctor(String ConnectString)
at UGCCMon.dbinterface..ctor(String dbtypeo, String connectstr) in C:\ugccmon\dbinterface.cs:line 23
at UGCCMon.UGCCMon.monitor() in C:\ugccmon\Service1.cs:line 333
03/01/2009 23:01:56 System resource exceeded.
at System.Data.OleDb.OleDbConnectionInternal..ctor(OleDbConnectionString constr, OleDbConnection connection)
at System.Data.OleDb.OleDbConnectionFactory.CreateConnection(DbConnectionOptions options, Object poolGroupProviderInfo, DbConnectionPool pool, DbConnection owningObject)
at System.Data.ProviderBase.DbConnectionFactory.CreateNonPooledConnection(DbConnection owningConnection, DbConnectionPoolGroup poolGroup)
at System.Data.ProviderBase.DbConnectionFactory.GetConnection(DbConnection owningConnection)
at System.Data.ProviderBase.DbConnectionClosed.OpenConnection(DbConnection outerConnection, DbConnectionFactory connectionFactory)
at System.Data.OleDb.OleDbConnection.Open()
at MashToolsSharp.oledbClass..ctor(String ConnectString)
at UGCCMon.dbinterface..ctor(String dbtypeo, String connectstr) in C:\ugccmon\dbinterface.cs:line 23
at UGCCMon.UGCCMon.monitor() in C:\ugccmon\Service1.cs:line 333
03/01/2009 23:01:59 System resource exceeded.
03/01/2009 23:01:59 at System.Data.OleDb.OleDbConnectionInternal..ctor(OleDbConnectionString constr, OleDbConnection connection)
at System.Data.OleDb.OleDbConnectionFactory.CreateConnection(DbConnectionOptions options, Object poolGroupProviderInfo, DbConnectionPool pool, DbConnection owningObject)
at System.Data.ProviderBase.DbConnectionFactory.CreateNonPooledConnection(DbConnection owningConnection, DbConnectionPoolGroup poolGroup)
at System.Data.ProviderBase.DbConnectionFactory.GetConnection(DbConnection owningConnection)
at System.Data.ProviderBase.DbConnectionClosed.OpenConnection(DbConnection outerConnection, DbConnectionFactory connectionFactory)
at System.Data.OleDb.OleDbConnection.Open()
at MashToolsSharp.oledbClass..ctor(String ConnectString)
at UGCCMon.dbinterface..ctor(String dbtypeo, String connectstr) in C:\ugccmon\dbinterface.cs:line 23
at UGCCMon.UGCCMon.timedrestarter() in C:\ugccmon\Service1.cs:line 513
Second Problem
From looking in the logs I have also noticed the panel quering the servers and reporting that the server has crashed (which it has) and says server restarting and it was successful but the server doesn't come back online!
Example on TF2 server:
Code: Select all
03/01/2009 10:03:25 Server id49 has been non-responsive for 4 queries--restarting.
03/01/2009 10:03:25 Server dead (TF2#1), attempt to restart...SUCCESS
03/01/2009 10:08:29 Server id49 has been non-responsive for 4 queries--restarting.
03/01/2009 10:08:29 Server dead (TF2#1), attempt to restart...SUCCESS
03/01/2009 10:13:34 Server id49 has been non-responsive for 4 queries--restarting.
03/01/2009 10:13:34 Server dead (TF2#1), attempt to restart...SUCCESS
03/01/2009 10:18:39 Server id49 has been non-responsive for 4 queries--restarting.
03/01/2009 10:18:39 Server dead (TF2#1), attempt to restart...SUCCESS
03/01/2009 10:23:44 Server id49 has been non-responsive for 4 queries--restarting.
03/01/2009 10:23:44 Server dead (TF2#1), attempt to restart...SUCCESS
03/01/2009 10:28:49 Server id49 has been non-responsive for 4 queries--restarting.
03/01/2009 10:28:49 Server dead (TF2#1), attempt to restart...SUCCESS
03/01/2009 10:33:54 Server id49 has been non-responsive for 4 queries--restarting.
03/01/2009 10:33:54 Server dead (TF2#1), attempt to restart...SUCCESS
03/01/2009 10:38:59 Server id49 has been non-responsive for 4 queries--restarting.
03/01/2009 10:38:59 Server dead (TF2#1), attempt to restart...SUCCESS
03/01/2009 10:44:03 Server id49 has been non-responsive for 4 queries--restarting.
03/01/2009 10:44:04 Server dead (TF2#1), attempt to restart...SUCCESS
03/01/2009 10:49:08 Server id49 has been non-responsive for 4 queries--restarting.
03/01/2009 10:49:08 Server dead (TF2#1), attempt to restart...SUCCESS
03/01/2009 10:54:13 Server id49 has been non-responsive for 4 queries--restarting.
03/01/2009 10:54:13 Server dead (TF2#1), attempt to restart...SUCCESS
03/01/2009 10:59:18 Server id49 has been non-responsive for 4 queries--restarting.
03/01/2009 10:59:18 Server dead (TF2#1), attempt to restart...SUCCESS