Random Server Crashes..

[Super Sanka]

What happened today? Haven't had that much lag since...never. I thought they were throwing a party at server hq. Although my downloads didn't help heh.

[Enyo]

What happened today? Haven't had that much lag since...never. I thought they were throwing a party at server hq. Although my downloads didn't help heh.

Yeah, the severe lag on MTMU was the worst and only significant lag I’ve experienced since they moved us to the new hardware.

[pooty]

Lag must have been network related, though I didn't see any record of any DDOS or network outages. Might have been outside of NFO's control too...

Between new hardware and some of the vehicle changes I made, it seems CPU is really good. For last week peak CPU was 52, and several times we had 32 players!

Just need to figure out the crashes... do we have any volunteers for perhaps this weekend for testing? We could potentially use not only the server, but we could maybe use Anon's or Roger's to get perhaps some additional details since we have direct access to those. Or even I could spin up a server on my desktop and open it up for testing.

[pooty]

Snarf, need your eyes on this.
For GameRules we have UTCompGameRules.CheckScore

Code: Select all

function bool CheckScore(PlayerReplicationInfo Scorer)
{
	...

    retval = false;
    if(NextGameRules != none)
        retval = NextGameRules.CheckScore(Scorer);
...

However, we also have set of rules in EvenMatchOmni

Code: Select all

function bool CheckScore(PlayerReplicationInfo Scorer)
{
	local int i;

  if (bBalancing) return True;
  // if balancing return, we don't need to do anything else.
    if (EvenMatchMutator.bCustomScoring && Scorer != None) CustomScore(Scorer);
  //if (bBalancing || Super.CheckScore(Scorer)) {  [i](Wonder if we really need that Super.CheckScore(Scorer)? Its commented out )
[/i]
  ...... bunch of Evenmatch code
  
  // should we check for Scorer = None?
  if ( NextGameRules != none )
	{
		return NextGameRules.CheckScore( Scorer );
  }
    
	return false;
  
  

Question is we don't know which order the NextGameRules.CheckScore gets called and potentially, if the rules are like this [EvenMatchOmni, UTCompOmni], then it isn't going to call UTCompOmniRules.CheckScore because bBalancing = True (edit I think this only gets true during mid round shuffle so I probably doesn't matter ), and it will never call:
// should we check for Scorer = None?

Code: Select all

if ( NextGameRules != none )
	{
		return NextGameRules.CheckScore( Scorer );
  }
    
	return false;

On a related note: What do we really need UTCompOmniRules.CheckScore... its noted that this was to fix end of round shenanigans.. but I don't recall why we should even need that? All the client focus should get set by Onslaught.ONSOnslaughtGame.MainCoreDestroyed?

Also, I know we changed the above loop in UTCompOmniRules.CheckScore from for to while loop, but all core UT code seems to use a for loop instead, so maybe should change it back. I realize it shouldn't make any difference but who knows.

And one other thing, could the crash be somehow related to the scoreboard bug? Looked at the code, not sure how it would get to UTComp scoreboard as default

[pooty]

Hmm from Engine.GameRules

Code: Select all

* CheckScore()
see if this score means the game ends
return true to override gameinfo checkscore, or if game was ended (with a call to Level.Game.EndGame() )
*/
function bool CheckScore(PlayerReplicationInfo Scorer)
{
    if ( NextGameRules != None )
        return NextGameRules.CheckScore(Scorer);

    return false;
}

I wonder if this
if (bBalancing) return True; //doesn't check for game end because return true makes it skip the check.

And/Or bBalancing is a local Var/Prop to EvenMatchOmni, but never gets set unless it mulligans... so first run through technically its undefined (default to false or true). Could it be that it calls this and returns true, never gets to gameinforules that techincally ends the round/game?

Maybe should add
bBalancing=False to default properties

[captainsnarf]

Yeah, it might be something with that return value.

maybe it should be

Code: Select all

if (bBalancing) return false; 

?? This whole mechanism is to allow for mutators to alter when the round/game ends. I don't think we really want to do that, so we should probably never return true here. We are just hooking into the CheckScore function to do some other stuff but not actually end the round prematurely.


I think as test, we might want to completely comment out the UTCompGameRules.CheckScore function and see if the crashing goes away.

Players may be able to move around and shoot after core dies but maybe their won't be any crashes anymore. It's hard to say for certain. There is no way to really test this without actual players.

[pooty]

There is no way to really test this without actual players.

Nope, I can kind of test with running Server and 64-bit client. But can't make it crash, but I can test the mutator and the end of round stuff.

[pooty]

Do we need this in UTComp CheckScores:

if(C != None) C.RoundHasEnded();

We know Round has ended do we need to call this?

[captainsnarf]

Do we need this in UTComp CheckScores:

if(C != None) C.RoundHasEnded();

We know Round has ended do we need to call this?

Yeah I believe that is a part of the fix. That's what keeps them from running around after the core is destroyed.

[pooty]

Hmm... I tested it without C.RoundHasEnded() and its seems to not let me run around at the end (whereas if I comment out the whole CheckScores I can run around).

I may try it. I have a new version of both UTCompOmni (that's the change there) and EvenMatch, some clean up there as well. (mulligan set as default)

I'll try to push them to the server... at best it fixes it, at worst they might still run around the cores... I tried several times to crash it locally and nothing... I did on prior version get it to happen, but the log showed nothing (but I had an orphan UT2004 process). I added some additional debugging to EvenMatch.