Recent Lag ... and what's being done.

[pooty]

Probably better to just ask them in a support ticket. I'd hate to mess with it and have it mess up some of their automation and leave it in an unusable state.

I opened a ticket and asked.

CPU is bit higher over the last few months, probably due to tick rate increase.... maybe we need to reduce it a bit

[pooty]

Per NFO
"This is an old game and runs well already, so I do not believe there would be a benefit to experimenting with the 64-bit version, even if it offers higher performance. The CPU usage of your server has peaked at 79% over the last month, which also suggests that it is not CPU-constrained. "

So not sure what we'd get other than perhaps some new bugs on 64-bit version...

[Anonymous.]

Per NFO
"This is an old game and runs well already, so I do not believe there would be a benefit to experimenting with the 64-bit version, even if it offers higher performance. The CPU usage of your server has peaked at 79% over the last month, which also suggests that it is not CPU-constrained. "

So not sure what we'd get other than perhaps some new bugs on 64-bit version...

ucc.exe/ucc-win64.exe after 30 minutes of Torlan with 32 bots on identical inis, this time 30 seconds of video footage: https://streamable.com/bxdk0g

And another test, Minus-KitchenSink with 32 bots, 40 minutes in: https://streamable.com/yj1acq; 66% higher overall CPU time on x86 vs. x64, running average closer to 33%.



I've never seen a server provider turn down free 33%+ performance in my life, but it's NFO after all. They believe their at best once-per-minute CPU usage log over players reporting seconds of server pausing due to CPU maxing out.

[pooty]

I can share that with them.

We could also spin up a trial instance (I think you get a week) and try renaming/moving the .exes and see what happens. I don't want to do that on the main server in case something really gets screwed up.

I let them know and NFO said

I haven't seen a game yet that uses appreciably less CPU with 64-bit code, unfortunately. Often, CPU usage is higher, since 64-bit code requires more memory and leads to additional cacheline bouncing.

But Anon's testing shows otherwise. But humans put much more load on the game than bots....so it'd be an interesting test with human players.

And the other thing is we could lower the tick rate (I think its at 50, back to 45 or 40...) which really reduces server load.

All that being said, I still suspect networking, even though UT bandwidth is pretty smalll..

[Anonymous.]

And the other thing is we could lower the tick rate (I think its at 50, back to 45 or 40...) which really reduces server load.

The point of x64 is we can keep the tick rate higher without the lagouts from max utilization. But as it stands right now, 50 is too high with enough going on. Hell, you can even raise it and see what happens to reproduce it (with some backlash ). It's changeable via set command in real time.

Code: Select all

admin set IpDrv.TcpNetDriver NetServerMaxTickRate #

Include admin only if you're doing it from the in-game console (vs. webadmin).

[pooty]

So the last couple of "lag" I think has been the logging.

Code: Select all

Log: Warning: attempt to set blending parameters for base stage in meshinstance [SkeletalMeshInstance]
Log: Warning: attempt to set blending parameters for base stage in meshinstance [SkeletalMeshInstance]
Log: Warning: attempt to set blending parameters for base stage in meshinstance [SkeletalMeshInstance]
Log: Warning: attempt to set blending parameters for base stage in meshinstance [SkeletalMeshInstance]

For example tonight on Pyroclastic, that message appears in the log like seriously 100,000 times. Yes 100k of log lines in the file with that error, not sure what's causing it. I bet the game lags when the log buffer's full and the single threaded game writes the log. Normally we wouldn't see 20000 lines a night and to see all that in one match.
Be nice to show which SkeletalMeshInstance bad fucking epic coders..... I've seen this message before so I wonder if its a vehicle and not the map itself?

[pooty]

Maps that had it (There's more but that was today's log)
ONS-Minus-McLovins-MEGA-ARENA-V8
ONS-Pyroclastic-Inferno-V5

[pooty]

I lowered the tick rate to 40 during the lag MB... no more lag after that.
I think 50 tick rate is too high for 32 players.

[pooty]

So that log spamming is coming from the FireBlade.... Thank goodness for MEGA-Arena made that easy to find.

Not sure what the fix is, but something in the FireBlade does it. Maybe I should just rework it, I think it needs a hit box fix anyway.

[vorpal]

I bet the game lags when the log buffer's full and the single threaded game writes the log. Normally we wouldn't see 20000 lines a night and to see all that in one match.

Do we know approximately how many lines are written to the log each evening? Even on a dedicated server, if NFO are using a congested or misconfigured SAN, it could cause hiccups.

To test this hypothesis, we could ask NFO to set up the Windows equivalent of a tmpfs partition (i.e. RAMdrive) and configure UT to write the log there instead of to disk. It would be interesting to see whether this would have an impact.