Random Server Crashes..

General Comments, Questions about all things OmnipotentS that don't go in other topics/forums
User avatar
pooty
Posts: 4358
Joined: Sat Apr 03, 2021 10:22 am
Location: Michigan
Server Sponsor: Yes
Server Admin: Yes

Re: Random Server Crashes..

Post by pooty »

I wonder since, we don't use (nor would we ever I think) use EvenMatch to balance teams while playing I could comment out the whole notify_logout since it doesn't matter or just do this: [Note none of this applies to manually shuffling teams, or to the numbers balancing by UAdminMod]. Although we've been using this EvenMatch code and UAdminMod for years and didn't have crashes like this...

Code: Select all

function NotifyLogout(Controller Exiting)
{
	
	Super.NotifyLogout(Exiting);

     if (bBalanceTeamsWhilePlaying ) {
	if (PlayerController(Exiting) != None && Exiting.PlayerReplicationInfo != None ) {
		if (!Exiting.PlayerReplicationInfo.bOnlySpectator) 
		   if (bDebug) log("DEBUG: " $ Exiting.GetHumanReadableName() $ " disconnected", 'EvenMatchDebug_NotifyLogout');
		else if (bDebug) log("DEBUG: " $ Exiting.GetHumanReadableName() $ " Became a Spectator", 'EvenMatchDebug_NotifyLogout'); 
		// FYI I don't think NotifyLogout happens for specs  They aren't Logout, they just spec
		Rules.SetTimer(0.0, false);
		CheckBalance(PlayerController(Exiting), True);
	}
   }	
}

function CheckBalance(PlayerController Player, bool bIsLeaving)
{
	if (!bBalanceTeamsWhilePlaying || (Level.Game.bOverTime && !bBalanceTeamsDuringOvertime))
		return;

	ActuallyCheckBalance(Player, bIsLeaving);
}
User avatar
captainsnarf
Posts: 2632
Joined: Tue Jul 06, 2021 1:51 pm
Location: Washington
Server Sponsor: Yes
Server Admin: Yes
Contact:

Re: Random Server Crashes..

Post by captainsnarf »

Yeah that might do it pooty. Putting the if(bBalanceTeamsWhilePlaying) at the top avoids a lot of crap. I still think it's a timer call somewhere. Rules.SetTimer(0, false) might be it, and that check you suggest avoids it.

Code: Select all

Warning: Invalid object index 32071688
Guaranteed crash when you see that message.
User avatar
pooty
Posts: 4358
Joined: Sat Apr 03, 2021 10:22 am
Location: Michigan
Server Sponsor: Yes
Server Admin: Yes

Re: Random Server Crashes..

Post by pooty »

Updated EvenMatch 3.64 on server. Includes the change detailed below on NotifyLogout.
if (bBalanceTeamsWhilePlaying ) {
User avatar
pooty
Posts: 4358
Joined: Sat Apr 03, 2021 10:22 am
Location: Michigan
Server Sponsor: Yes
Server Admin: Yes

Re: Random Server Crashes..

Post by pooty »

Nope no dice. Crashed again when a bunch left at the end of the round.

Code: Select all

Warning: ONSMapListOnslaught ONS-GuitarShop-v3.ONSMapListOnslaught (Function Engine.MapList.PostBeginPlay:0030) ONSMapListOnslaught has no maps configured!
Error: OLServQuery ONS-GuitarShop-v3.OLServQuery (Function OLStats.OLServQuery.GetGamestatus:0209) Accessed array 'Maps' out of bounds (0/0)
EvenMatchDebug: 174.90 BIG_FloPPies switched to Red
EvenMatchDebug: 183.15 702j switched to Blue
EvenMatchDebug: 189.09 702j switched to Blue
EvenMatchDebug: 189.84 YEAAAHHHHHHHHHHHH switched to Red
EvenMatchDebug: 190.20 Saving PPH data...
EvenMatchDebug: 212.32 Singularity switched to Blue
EvenMatchDebug: 216.04 EbNo switched to Red
EvenMatchDebug: 217.70 Singularity switched to Blue
NetComeGo: Close TcpipConnection AnonIP:10171 03/21/23 22:40:05
EvenMatchDebug: 279.21 Saving PPH data...
Warning: UTComp_xPawn ONS-GuitarShop-v3.UTComp_xPawn (Function XGame.xPawn.ServerChangedWeapon:00EC) Accessed None 'Weapon'
Warning: SpawnActor failed because class Pawn is abstract
Warning: UTComp_xPawn ONS-GuitarShop-v3.UTComp_xPawn (Function XGame.xPawn.ServerChangedWeapon:00EC) Accessed None 'Weapon'
Warning: SpawnActor failed because class Pawn is abstract
ScriptLog: Couldn't spawn player of type UTCompOmni.UTComp_xPawn at ONS-GuitarShop-v3.PlayerStart33
Warning: SpawnActor failed because class Pawn is abstract
ScriptLog: Couldn't spawn player of type UTCompOmni.UTComp_xPawn at ONS-GuitarShop-v3.PlayerStart16
ScriptLog: YEAAAHHHHHHHHHHHH logged in as yeah.
Log: Opening user log ..\UserLogs\AntiTCC_2023-03-21_22-36_ONSOnslaughtGame_ONS-GuitarShop-v3.log
Warning: SpawnActor failed because class Pawn is abstract
ScriptLog: Couldn't spawn player of type UTCompOmni.UTComp_xPawn at ONS-GuitarShop-v3.PlayerStart55
AntiTCC: Saving Anti TCC log ..\UserLogs\AntiTCC_2023-03-21_22-36_ONSOnslaughtGame_ONS-GuitarShop-v3.log
UTCompOmni_ONSGameRules_CheckScore: Finished Resetting ClientViews
NetComeGo: Close TcpipConnection BigFloppiesIP:55817 03/21/23 22:48:51
UTCompOmni_ONSGameRules_CheckScore: Finished Resetting ClientViews
Log: Opening user log ..\UserLogs\AntiTCC_2023-03-21_22-36_ONSOnslaughtGame_ONS-GuitarShop-v3.log
NetComeGo: Open myLevel 03/21/23 22:48:52  BigFloppiesIP:55817
NetComeGo: Close TcpipConnection SingularityIP:52890 03/21/23 22:48:55
UTCompOmni_ONSGameRules_CheckScore: Finished Resetting ClientViews
NetComeGo: Close TcpipConnection AnkeedoIP:65155 03/21/23 22:48:55
UTCompOmni_ONSGameRules_CheckScore: Finished Resetting ClientViews
ScriptLog: Reset Onslaught.
Log: Collecting garbage
Log: Purging garbage
Log: Garbage: objects: 79028->76110; refs: 1077982
Warning: ONSOnslaughtGame ONS-GuitarShop-v3.ONSOnslaughtGame (Function Engine.GameInfo.DiscardInventory:0007) Accessed None 'Other'
Warning: ONSOnslaughtGame ONS-GuitarShop-v3.ONSOnslaughtGame (Function Engine.GameInfo.DiscardInventory:000F) Attempt to assign variable through None
Warning: ONSOnslaughtGame ONS-GuitarShop-v3.ONSOnslaughtGame (Function Engine.GameInfo.DiscardInventory:0017) Accessed None 'Other'
Warning: ONSOnslaughtGame ONS-GuitarShop-v3.ONSOnslaughtGame (Function Engine.GameInfo.DiscardInventory:001F) Attempt to assign variable through None
Warning: ONSOnslaughtGame ONS-GuitarShop-v3.ONSOnslaughtGame (Function Engine.GameInfo.DiscardInventory:002A) Accessed None 'Other'
Warning: Invalid object index 288174088
Warning: This is: TcpipConnection Package.TcpipConnection
Critical: Assertion failed: IsValid() [File:.\UnObj.cpp] [Line: 464]
Exit: Executing UObject::StaticShutdownAfterError
Critical: UObject::~UObject
Critical: UNetDriver::TickDispatch
Critical: UTcpNetDriver::TickDispatch
Critical: UpdatePreNet
Critical: ULevel::Tick
Critical: (NetMode=1)
Critical: TickLevel
Critical: UGameEngine::Tick
Critical: Level ONS-GuitarShop
Critical: UpdateWorld
Critical: UServerCommandlet::Main
Exit: Exiting.
Log: FileManager: Reading 2 GByte 4175 MByte 4194351 KByte 518 Bytes from HD took 6.436857 seconds (5.062850 reading, 1.374008 seeking).
Log: FileManager: 2.226020 seconds spent with m
So this is odd:
NetComeGo: Close TcpipConnection BigFloppiesIP:55817 03/21/23 22:48:51
UTCompOmni_ONSGameRules_CheckScore: Finished Resetting ClientViews
Log: Opening user log ..\UserLogs\AntiTCC_2023-03-21_22-36_ONSOnslaughtGame_ONS-GuitarShop-v3.log
NetComeGo: Open myLevel 03/21/23 22:48:52 BigFloppiesIP:55817

Why is it opening myLevel for BigFloppies if he left?

And our friendly CheckScore is right in the middle of it. I think we need to comment function out entirely... and figure out why the end of round viewsetting normally isn't working. Set the flag to disable check score function. Lets see if that stops the crashes at least.

And what's with these 'Maps' errors:
Warning: ONSMapListOnslaught ONS-GuitarShop-v3.ONSMapListOnslaught (Function Engine.MapList.PostBeginPlay:0030) ONSMapListOnslaught has no maps configured!
Error: OLServQuery ONS-GuitarShop-v3.OLServQuery (Function OLStats.OLServQuery.GetGamestatus:0209) Accessed array 'Maps' out of bounds (0/0)
User avatar
pooty
Posts: 4358
Joined: Sat Apr 03, 2021 10:22 am
Location: Michigan
Server Sponsor: Yes
Server Admin: Yes

Re: Random Server Crashes..

Post by pooty »

Code: Select all

UTCompOmni_ONSGameRules_CheckScore: Finished Resetting ClientViews
Log: Opening user log ..\UserLogs\AntiTCC_2023-03-21_22-36_ONSOnslaughtGame_ONS-GuitarShop-v3.log
NetComeGo: Open myLevel 03/21/23 22:48:52  BigFloppiesIP:55817
NetComeGo: Close TcpipConnection SingularityIP:52890 03/21/23 22:48:55
UTCompOmni_ONSGameRules_CheckScore: Finished Resetting ClientViews
NetComeGo: Close TcpipConnection AnkeedoIP:65155 03/21/23 22:48:55
UTCompOmni_ONSGameRules_CheckScore: Finished Resetting ClientViews
Snarf, why would CheckScore get called more than once at end of round? It should reset the views once and done.
User avatar
captainsnarf
Posts: 2632
Joined: Tue Jul 06, 2021 1:51 pm
Location: Washington
Server Sponsor: Yes
Server Admin: Yes
Contact:

Re: Random Server Crashes..

Post by captainsnarf »

pooty wrote: Wed Mar 22, 2023 9:56 am

Code: Select all

UTCompOmni_ONSGameRules_CheckScore: Finished Resetting ClientViews
Log: Opening user log ..\UserLogs\AntiTCC_2023-03-21_22-36_ONSOnslaughtGame_ONS-GuitarShop-v3.log
NetComeGo: Open myLevel 03/21/23 22:48:52  BigFloppiesIP:55817
NetComeGo: Close TcpipConnection SingularityIP:52890 03/21/23 22:48:55
UTCompOmni_ONSGameRules_CheckScore: Finished Resetting ClientViews
NetComeGo: Close TcpipConnection AnkeedoIP:65155 03/21/23 22:48:55
UTCompOmni_ONSGameRules_CheckScore: Finished Resetting ClientViews
Snarf, why would CheckScore get called more than once at end of round? It should reset the views once and done.
I think checkscore is called every time a score (kill, node destroyed, etc) is made. A death match game is over when score is 25 by default, so that checkscore is probably simple like 'if(score >=25) return true;'

Since you get points for shooting the core, but not until the core dies, maybe everybody that shot the core is causing checkscore to get called? Not really sure. We might need to return true in that method when we see one of the cores is dead (health <= 0).
User avatar
pooty
Posts: 4358
Joined: Sat Apr 03, 2021 10:22 am
Location: Michigan
Server Sponsor: Yes
Server Admin: Yes

Re: Random Server Crashes..

Post by pooty »

Code: Select all

function bool CheckScore(PlayerReplicationInfo Scorer)
{
    local PlayerController PC;
    local Controller C;
    local ONSOnslaughtGame ONS;
    local int deadCore;
    //local bool retval;

    if (bEnableEndRoundCheckScore) {

			Global.timer(); //clears all timers
			
	    deadCore = -1;
	    ONS = ONSOnslaughtGame(Level.Game);
	    if(ONS != none && ONS.PowerCores[ONS.FinalCore[0]].Health <= 0)
	        deadCore = 0;
	    else if(ONS != none && ONS.PowerCores[ONS.FinalCore[1]].Health <= 0)
	        deadCore = 1;

	        //round has ended
	    if(deadCore >= 0)
	    {
	        if(Level != None)
	            C = Level.ControllerList;

	// lots of checking on C since there's been a few crashes when someone leaves
	// maybe this catches it.
	        while(C != None)
	        {
	            if(C != None) PC = PlayerController(C);
	            if (PC != None) PC.ClientSetBehindView(true);
	            if (PC != None) PC.ClientSetViewTarget(ONS.PowerCores[ONS.FinalCore[deadCore]]);
	            if (PC != None) PC.SetViewTarget(ONS.PowerCores[ONS.FinalCore[deadCore]]);
	            if (PC != None) PC.ClientRoundEnded();
	            

	            if(C != None) C.RoundHasEnded();  // We know round has ended just set client views commented out. 03/2023 pooty

	            if(C != None) C = C.NextController;
	        } // while
	        if (bDebugRules) log("Finished Resetting ClientViews",'UTCompOmni_ONSGameRules_CheckScore');
	    } // dead core
	  }  // end bEnableEndRoundCheckScore

// Moved here to match other gamerules. other examples always show this at the end 03/2023 pooty
 if ( NextGameRules != None )
        return NextGameRules.CheckScore(Scorer);

    return false;
}
It only prints the "Finished Resetting ClientViews" IF dead core. So it shouldn't be getting called multiple times, once is enough. Maybe we need Mut level variable at the end of the while loop bClientViewsReset = True and add a check if (!bClientViewsReset) to skip if the round is over. Likely need to interrogate deadcore and move that into Mut Scope.

Either way, I've set the flag to skip this function so lets see if it still crashes...but we will probably get the weird end of round not seeing the right core/spot thing -- which might be the root of this whole mess.
User avatar
pooty
Posts: 4358
Joined: Sat Apr 03, 2021 10:22 am
Location: Michigan
Server Sponsor: Yes
Server Admin: Yes

Re: Random Server Crashes..

Post by pooty »

Boom. Not CheckScore. Crashed again tonight with CheckScore disabled.

Code: Select all

EvenMatchOmni: ModifyLogin: Red:2294.07 Blue:2293.17 - snarf going to BLUE
EvenMatch: Overriding player team preference team=255 with team=1
ScriptLog: Found new security actor ONS-DesertJunkYard-TMU-Randomizer-V12-Nova.AntiTCCSecurity for ONS-DesertJunkYard-TMU-Randomizer-V12-Nova.BS_xPlayer
ScriptLog: New Player snarf
PlayerJoinLog: [PLAYER_JOIN] 2023/3/22 22:8:3	snarf		10000	
Zound: 22:08:03:763 SetClientClass for: snarf
Log: Opening user log ..\UserLogs\AntiTCC_2023-03-22_21-45_ONSOnslaughtGame_ONS-DesertJunkYard-TMU-Randomizer-V12-Nova.log
EvenMatchDebug: 1523.21 Saving PPH data...
AntiTCC: Saving Anti TCC log ..\UserLogs\AntiTCC_2023-03-22_21-45_ONSOnslaughtGame_ONS-DesertJunkYard-TMU-Randomizer-V12-Nova.log
Warning: ONSOnslaughtGame ONS-DesertJunkYard-TMU-Randomizer-V12-Nova.ONSOnslaughtGame (Function Onslaught.ONSOnslaughtGame.MainCoreDestroyed:020B) Accessed None 'C'
NetComeGo: Close TcpipConnection RETURNER_IP:57460 03/22/23 22:09:19
ScriptLog: Reset Onslaught.
Log: Collecting garbage
Log: Purging garbage
Log: Garbage: objects: 103764->101932; refs: 1432755
Warning: ONSVehicleRandomizer ONS-DesertJunkYard-TMU-Randomizer-V12-Nova.ONSVehicleRandomizer1 (Function ONS-DesertJunkYard-TMU-Randomizer-V12-Nova.ONSVehicleRandomizer.RandomizeVehicles:00D5) ** ONSVehicleRandomizer: No MusicReplicator found in level (must be present to change songs randomly)
Warning: ONSOnslaughtGame ONS-DesertJunkYard-TMU-Randomizer-V12-Nova.ONSOnslaughtGame (Function Engine.GameInfo.DiscardInventory:0007) Accessed None 'Other'
Warning: ONSOnslaughtGame ONS-DesertJunkYard-TMU-Randomizer-V12-Nova.ONSOnslaughtGame (Function Engine.GameInfo.DiscardInventory:000F) Attempt to assign variable through None
Warning: ONSOnslaughtGame ONS-DesertJunkYard-TMU-Randomizer-V12-Nova.ONSOnslaughtGame (Function Engine.GameInfo.DiscardInventory:0017) Accessed None 'Other'
Warning: ONSOnslaughtGame ONS-DesertJunkYard-TMU-Randomizer-V12-Nova.ONSOnslaughtGame (Function Engine.GameInfo.DiscardInventory:001F) Attempt to assign variable through None
Warning: ONSOnslaughtGame ONS-DesertJunkYard-TMU-Randomizer-V12-Nova.ONSOnslaughtGame (Function Engine.GameInfo.DiscardInventory:002A) Accessed None 'Other'
Log: Opening user log ..\UserLogs\AntiTCC_2023-03-22_21-45_ONSOnslaughtGame_ONS-DesertJunkYard-TMU-Randomizer-V12-Nova.log
Critical: UObject::Destroy
Critical: UObject::GetFullName
Critical: UPlayer::Destroy
Critical: UNet
Its definitely crashing on the start of the new round, not the end of the last round.
1. ScriptLog: Reset Onslaught. (Starts new round)
2. With CheckScore fix off -- that fixes the focus on the cores and you can hear the countdown before the crash.

So, that does narrow it down, but fundamentally what in UTCompOmni is messing up the end of round, I think that's missing some "cleanup" resulting in the crash on the restart. These warnings here:
Warning: ONSOnslaughtGame ONS-DesertJunkYard-TMU-Randomizer-V12-Nova.ONSOnslaughtGame (Function Engine.GameInfo.DiscardInventory:002A) Accessed None 'Other'
Never showed up prior to these crashes.
User avatar
pooty
Posts: 4358
Joined: Sat Apr 03, 2021 10:22 am
Location: Michigan
Server Sponsor: Yes
Server Admin: Yes

Re: Random Server Crashes..

Post by pooty »

Code: Select all

/* Discard a player's inventory after he dies.
*/
function DiscardInventory( Pawn Other )
{
    Other.Weapon = None;
    Other.SelectedItem = None;
    while ( Other.Inventory != None )
        Other.Inventory.Destroy();
}
and

Code: Select all

function MainCoreDestroyed(byte T)
{
    local Controller C;
    local PlayerController PC;
    local int Score;

    if (bOverTime)
        Score = 1;
    else
        Score = 2;

    if (T == 1)
    {
        BroadcastLocalizedMessage( class'ONSOnslaughtMessage', 0);
        TeamScoreEvent(0, Score, "enemy_core_destroyed");
        Teams[0].Score += Score;
        Teams[0].NetUpdateTime = Level.TimeSeconds - 1;
       CheckScore(PowerCores[FinalCore[1]].LastDamagedBy);
    }
    else
    {
        BroadcastLocalizedMessage( class'ONSOnslaughtMessage', 1);
        TeamScoreEvent(1, Score, "enemy_core_destroyed");
        Teams[1].Score += Score;
        Teams[1].NetUpdateTime = Level.TimeSeconds - 1;
        CheckScore(PowerCores[FinalCore[0]].LastDamagedBy);
    }

    //round has ended
    for (C = Level.ControllerList; C != None; C = C.NextController)
    {
        PC = PlayerController(C);
        if (PC != None)
        {
            PC.ClientSetBehindView(true);
            PC.ClientSetViewTarget(PowerCores[FinalCore[T]]);
            PC.SetViewTarget(PowerCores[FinalCore[T]]);
            if (!bGameEnded)
                PC.ClientRoundEnded();
        }
        if (!bGameEnded)
            C.RoundHasEnded();
    }

    ResetCountDown = ResetTimeDelay;
}
I think I am getting somewhere. This is the function here:
Warning: ONSOnslaughtGame ONS-DesertJunkYard-TMU-Randomizer-V12-Nova.ONSOnslaughtGame (Function Onslaught.ONSOnslaughtGame.MainCoreDestroyed:020B) Accessed None 'C'

Which that warning should never occur.
That's the issue with the funky end of round all the stuff past //round has ended is what sets the view to the exploding core.
However we had to override the scoring for no 2 pt victories -- perhaps that's part of the issue. Maybe we just make non-OT a 2pt victory -- if you can win in 4 mins or less, then teams are fubar'd and move on (or we could mulligan it...), I'd take that over these mid match crashes.
User avatar
YEAAAHHHHHHHHHH
Posts: 1059
Joined: Tue Jun 08, 2021 3:03 pm
Server Sponsor: Yes
Server Admin: Yes

Re: Random Server Crashes..

Post by YEAAAHHHHHHHHHH »

Maybe we just make non-OT a 2pt victory -- if you can win in 4 mins or less, then teams are fubar'd and move on (or we could mulligan it...), I'd take that over these mid match crashes.
I'd be cool with that. I think we have enough admins active that we can intervene early enough if needed. And if we blow through a map quickly then so what.
Post Reply