Random Server Crashes..

General Comments, Questions about all things OmnipotentS that don't go in other topics/forums
User avatar
McLovin
Posts: 1165
Joined: Sat Apr 03, 2021 12:54 pm
Location: Salt Lake City, Utah
Server Sponsor: Yes
Server Admin: Yes

Re: Random Server Crashes..

Post by McLovin »

Server crash:
Map: ONS-MinusBadgerMeUp-Beta23
Time: Wed, 20230315 at 1815 MDT

Nobody on server except me, sitting idle. Not at end of round.
MasterBath loaded.

Funny thing: Before the reboot, the spectator list would show. After the reboot, the spectator list is gone from the lower right corner.
User avatar
pooty
Posts: 4358
Joined: Sat Apr 03, 2021 10:22 am
Location: Michigan
Server Sponsor: Yes
Server Admin: Yes

Re: Random Server Crashes..

Post by pooty »

McLovin that was me, you were idle and I had new stuff to load.

But it didn't fix the crashes.

Code: Select all

PlayerJoinLog: [PLAYER_JOIN] 2023/3/15 21:25:41	TITAN_LOGIC	REDACTEDIP:49760	9636	
Zound: 21:25:41:809 SetClientClass for: TITAN_LOGIC
Log: Opening user log ..\UserLogs\AntiTCC_2023-03-15_21-14_ONSOnslaughtGame_ONS-Dria-Randomizer-V22.log
AntiTCC: Saving Anti TCC log ..\UserLogs\AntiTCC_2023-03-15_21-14_ONSOnslaughtGame_ONS-Dria-Randomizer-V22.log
Warning: Reaper ONS-Dria-Randomizer-V22.Reaper (Function CSapveriv.Reaper.NotifyEnemyLockedOn:0009) Accessed None 'Controller'
EvenMatchDebug: 801.00 Saving PPH data...
Warning: ONSOnslaughtGame ONS-Dria-Randomizer-V22.ONSOnslaughtGame (Function Onslaught.ONSOnslaughtGame.MainCoreDestroyed:020B) Accessed None 'C'
NetComeGo: Close TcpipConnection REDACTEDIP(TITAN_LOGIC):49760 03/15/23 21:27:15
ScriptLog: Reset Onslaught.
Log: Collecting garbage
Log: Purging garbage
Log: Garbage: objects: 117439->114238; refs: 1605909
Warning: ONSVehicleRandomizer ONS-Dria-Randomizer-V22.ONSVehicleRandomizer1 (Function ONS-Dria-
So Titan_logic left at the end of the round, and boom.
I see this, looks familiar
(Function Onslaught.ONSOnslaughtGame.MainCoreDestroyed:020B) Accessed None 'C'

At this point I think we need to turn off evenmatch or UTComp and see if it still crashes.. not sure which one is easier to decouple.
User avatar
pooty
Posts: 4358
Joined: Sat Apr 03, 2021 10:22 am
Location: Michigan
Server Sponsor: Yes
Server Admin: Yes

Re: Random Server Crashes..

Post by pooty »

Funny thing: Before the reboot, the spectator list would show. After the reboot, the spectator list is gone from the lower right corner.
I disabled the UTComp Scoreboard as the default server setting...I think that's fixed the bug of people getting the too short UTComp one when they don't want it. Maybe we should just make the UTComp one hold 16v16.


And what at least is somewhat interesting is this line
Warning: ONSVehicleRandomizer ONS-Dria-Randomizer-V22.ONSVehicleRandomizer1 (Function ONS-Dria-

That's this warning, but cut off.
Warning: ONSVehicleRandomizer ONS-Dria-Randomizer-V22.ONSVehicleRandomizer1 (Function ONS-Dria-Randomizer-V22.ONSVehicleRandomizer.RandomizeVehicles:00D5) ** ONSVehicleRandomizer: No MusicReplicator found in level (must be present to change songs randomly)

But that warning fires when the new level starts, and the Randomizer begins...so maybe the crashes are happening in Start of Next Round, and the previous round completes OK.
User avatar
McLovin
Posts: 1165
Joined: Sat Apr 03, 2021 12:54 pm
Location: Salt Lake City, Utah
Server Sponsor: Yes
Server Admin: Yes

Re: Random Server Crashes..

Post by McLovin »

pooty wrote: Wed Mar 15, 2023 9:32 pm McLovin that was me, you were idle and I had new stuff to load.
...
Oh good then. When I'm waiting for pick-up play I don't constantly monitor the game screen. I'm off surfing the internet on a 2nd monitor. I usually find out somebody joined when I get kicked for being idle.
User avatar
pooty
Posts: 4358
Joined: Sat Apr 03, 2021 10:22 am
Location: Michigan
Server Sponsor: Yes
Server Admin: Yes

Re: Random Server Crashes..

Post by pooty »

So a couple of things, since we still crashed.

- Going to put back the line C.RoundHasEnded(), as it still doesn't seem to point you to the proper core. Although I'd like to figure out why UTComp causes the need for this in the first place

- I think the scoreboard thing is fixed, but now we've lost spec list and colored names...I'll look at reverting it.

- Adding more log statements around end of round/start of round processing to hopefully narrow it down, as to whether its end of round, or start of next round where it booms. I am hoping to avoid just turning one or the other off during primetime (maybe if we get some testing volunteers (*hint* *hint*), plus it might be an interaction between EvenMatch and UTComp (which also has some ugly ONSPlus code).

- Moved the GameRulesNext processing to the end of the functions, to match examples in base UT source code.

I AM going to fix this damn crash. :twisted:
User avatar
captainsnarf
Posts: 2632
Joined: Tue Jul 06, 2021 1:51 pm
Location: Washington
Server Sponsor: Yes
Server Admin: Yes
Contact:

Re: Random Server Crashes..

Post by captainsnarf »

I still think we should try removing UTCompGameRules.CheckScore function and see if crashing goes away, or at least put a config variable around it.

Code: Select all

var config bool bEnableCheckScore;
function bool CheckScore(...)
{
    if(!bEnableCheckScore)
        return false;

}

defaultproperties
{
    bEnableCheckScore=true
}
then can be set in ut2004.ini

Code: Select all

[UTCompOmni.UTCompGameRules]
bEnableCheckScore=false
Another thing to try would be to add a call to 'Global.Timer()' at the beginning of CheckScore. My hunch is there is a timer going off that is modifying the controllerlist. It's that or garbage collection doing it. Somewhere somebody is calling Destroy on a controller :? A timer or garbage collection are the only things that could do it asynchronously. The game is not threaded.
User avatar
Enyo
Posts: 1626
Joined: Mon Apr 05, 2021 11:27 pm
Server Sponsor: Yes
Server Admin: Yes

Re: Random Server Crashes..

Post by Enyo »

FYI, we had a mid-round crash while playing Denum last night. We played through round 1, round 2 started and we got about 5 or 6 min in and it just randomly crashed during the 2nd round. There were only maybe 12-14 total players, but it didn’t get logged into the statsDB since the match didn’t finish.

Just thought you should know since this one had nothing to do with players joining or leaving between rounds.
“Never argue with stupid people, they will drag you down to their level and then beat you with experience.”
― Mark Twain
User avatar
pooty
Posts: 4358
Joined: Sat Apr 03, 2021 10:22 am
Location: Michigan
Server Sponsor: Yes
Server Admin: Yes

Re: Random Server Crashes..

Post by pooty »

Yeah I looked at the log for Denum, seemed to be vehicle / mesh interaction related. Not sure why it crashed, but it wasn't same type of end of round crash.

So I put some logging around UTCompGameRules.CheckScore

I think though its actually crashing on initializing the new round..

Code: Select all

NetComeGo: Close TcpipConnection REDACTEDIP(TITAN_LOGIC):49760 03/15/23 21:27:15
ScriptLog: Reset Onslaught.
Log: Collecting garbage
Log: Purging garbage
Log: Garbage: objects: 117439->114238; refs: 1605909
Warning: ONSVehicleRandomizer ONS-Dria-Randomizer-V22.ONSVehicleRandomizer1 (Function ONS-Dria-
The Reset Onslaught entry is the end of the round, resetting for the new one, at least based on watching the local log as things transpire.

I'll add the EnableEndRoundCheckScore, I was already adding some specific debug flags (for rules in general and the damage points). Damage points debug gets really chatty and we don't really need it, but its there in case.
User avatar
pooty
Posts: 4358
Joined: Sat Apr 03, 2021 10:22 am
Location: Michigan
Server Sponsor: Yes
Server Admin: Yes

Re: Random Server Crashes..

Post by pooty »

Updated version on the server.
User avatar
pooty
Posts: 4358
Joined: Sat Apr 03, 2021 10:22 am
Location: Michigan
Server Sponsor: Yes
Server Admin: Yes

Re: Random Server Crashes..

Post by pooty »

Last night, 0 crashes (at least from 1:50pm Eastern) when I loaded the updates on the server.
So keeping my fingers crossed.

And yes the scoreboard glitch is still there, not sure why but will work on it as I get time, or maybe snarf knows... I wonder if we should just make a ONS UTComp scoreboard that shows PPH and shows 16v16.
Post Reply