Server down

General Comments, Questions about all things OmnipotentS that don't go in other topics/forums
FuriousRabbit
Posts: 126
Joined: Sat Jun 04, 2022 12:28 am

Server down

Post by FuriousRabbit »

This is intolerable for a Friday. My mental health depends on this game.
User avatar
YEAAAHHHHHHHHHH
Posts: 1059
Joined: Tue Jun 08, 2021 3:03 pm
Server Sponsor: Yes
Server Admin: Yes

Re: Server down

Post by YEAAAHHHHHHHHHH »

DW was a fun nostalgia ride. Didn't cut it. I need Omni direct into my veins.
User avatar
pooty
Posts: 4358
Joined: Sat Apr 03, 2021 10:22 am
Location: Michigan
Server Sponsor: Yes
Server Admin: Yes

Re: Server down

Post by pooty »

Working on it. It appears to be an NFO problem. I have a ticket open with them. They are usually pretty good about responding.

Error:
Log: Defaulting to false
Log: FUdpLink::BindPort: Failed to bind UDP socket to port with error 10013
Critical: FUdpLink::BindPort: Failed to bind UDP socket to port 7778
Exit: Executing UObject::StaticShutdownAfterError
Critical: FUdpLink::BindPort
Critical: FMasterServerUplinkLink::FMasterServerUplinkLink
Critical: UObject::ProcessEvent
Critical: (OLSQMSUplink ONS-MasterBath-Renovation-v4-OS.OLSQMSUplink, Function OLStats.OLSQMSUplink.BeginPlay)
Critical: BeginPlay
Critical: UGameEngine::LoadMap
Critical: LocalMapURL
Critical: UGameEngine::Browse
Critical: UGameEngine::Init
Critical: UServerCommandlet::Main
Exit: Exiting.
User avatar
pooty
Posts: 4358
Joined: Sat Apr 03, 2021 10:22 am
Location: Michigan
Server Sponsor: Yes
Server Admin: Yes

Re: Server down

Post by pooty »

I've got it back up! Hooray!

Traced it to the OLStats mutator, we've been running that for over a year no issues, but its definitely that. If I back it out of UT2004.ini server starts up, if its on it fails. Perhaps it creates a zombie process on those UDP ports?

So the StatsDB logging is off. Curiously Enyo had noted it wasn't getting the right scores. This is for StatsDB (https://www.omnipotents.com/utstats/) which presently stopped logging on 2.9.2023? Keep in mind this is different from the scores the balancer uses..it has its own recordkeeping.

Since it won't start with it, and there's clearly a scoring bug with it. Its off until further notice, not sure its worth the time to fix it. Its a nice to have but it might have some interaction with the older UT master servers? Or somehow it crashed on our server? But at least the server is back up.
User avatar
infinitecat
Posts: 141
Joined: Tue Jun 08, 2021 3:02 pm
Location: Dallas

Re: Server down

Post by infinitecat »

Thanks, Pooty! Hope you had a good Vegas vacay.
FuriousRabbit
Posts: 126
Joined: Sat Jun 04, 2022 12:28 am

Re: Server down

Post by FuriousRabbit »

Awesome news. Thanks for the effort. Come to think of it, the server has had good uptime. I can't even remember the last time it was down.
User avatar
Enyo
Posts: 1626
Joined: Mon Apr 05, 2021 11:27 pm
Server Sponsor: Yes
Server Admin: Yes

Re: Server down

Post by Enyo »

pooty wrote: Sat Feb 11, 2023 7:53 am I've got it back up! Hooray!

Traced it to the OLStats mutator, we've been running that for over a year no issues, but its definitely that. If I back it out of UT2004.ini server starts up, if its on it fails. Perhaps it creates a zombie process on those UDP ports?

So the StatsDB logging is off. Curiously Enyo had noted it wasn't getting the right scores. This is for StatsDB (https://www.omnipotents.com/utstats/) which presently stopped logging on 2.9.2023? Keep in mind this is different from the scores the balancer uses..it has its own recordkeeping.

Since it won't start with it, and there's clearly a scoring bug with it. Its off until further notice, not sure its worth the time to fix it. Its a nice to have but it might have some interaction with the older UT master servers? Or somehow it crashed on our server? But at least the server is back up.
I suspect statsDB being broken, and subsequently the conflict causing the server to be down, has something to do with the recent damage points fix. StatsDB was fine before that, but I'd rather have correct scoring for the balancer to work off than have a historical record of stats just for shits and giggles. Or, it could've been that update you had to do to make the statsDB work again after the PHP update.

BTW, one of my web developers says updating to PHP 8 is not necessary and is causing a ton of issues with existing sites. He's advising all our customers to stay on 7.4, all the warnings that it's not secure are BS. Not sure we have a choice here though.
“Never argue with stupid people, they will drag you down to their level and then beat you with experience.”
― Mark Twain
User avatar
pooty
Posts: 4358
Joined: Sat Apr 03, 2021 10:22 am
Location: Michigan
Server Sponsor: Yes
Server Admin: Yes

Re: Server down

Post by pooty »

I suspect statsDB being broken, and subsequently the conflict causing the server to be down, has something to do with the recent damage points fix. StatsDB was fine before that, but I'd rather have correct scoring for the balancer to work off than have a historical record of stats just for shits and giggles. Or, it could've been that update you had to do to make the statsDB work again after the PHP update.

BTW, one of my web developers says updating to PHP 8 is not necessary and is causing a ton of issues with existing sites. He's advising all our customers to stay on 7.4, all the warnings that it's not secure are BS. Not sure we have a choice here though.
So there's two basically seperate components:
1. OLStats, this is the mutator that runs on the server that collects the raw stats. Its written in UTScript. This was the culprit behind the crashes. It was trying to bind to some network ports that was failing. Basically it writes out a stats file, and then it gets FTP'd automatically over to the StatsDB which runs on a different machine than the game.. So the damage points could be affecting that depending on how the hooks are written, but damage points wouldn't cause the network fails we saw in the errors. There also could have been something in OLStats that used the old Epic Master Servers which are now DOA -- the newer Openspy/333Networks don't have all the Stats functionality that epic ones did...

2. StatsDB, this is where the file is picked up and parsed, then subsequently loaded into the mySQL database. That's all done in PHP, same with the web front end. This was where I had to make all the changes for the PHP 8 upgrade, which apparently has some security features etc. (https://wisdmlabs.com/blog/php-8-upgrade/ ) Note I am not a PHP expert, but when you have to manage multiple machines and keep your support up to date, you need to upgrade. Mostly code changes were just fixing deprecated functions. Any changes here have no effect on the game server.

Perhaps one day I'll take a look at the OLStats, but its as pointed out "just for shits and giggles" so there isn't a good payback for the hours spent vs. say map edits or server stability.
User avatar
captainsnarf
Posts: 2632
Joined: Tue Jul 06, 2021 1:51 pm
Location: Washington
Server Sponsor: Yes
Server Admin: Yes
Contact:

Re: Server down

Post by captainsnarf »

Oh no! I will forever be 3rd place behind Leon and Enyo :shock: 8-) Pooty will forever be ... Player 19 :lol:
User avatar
pooty
Posts: 4358
Joined: Sat Apr 03, 2021 10:22 am
Location: Michigan
Server Sponsor: Yes
Server Admin: Yes

Re: Server down

Post by pooty »

Player19 kicks ass. lolz

So I took a quick look through the OLStats code, it extends MasterServerGameStats and MasterServerUplink both which connect and report to Epic's (Gamespy) master servers for stats. I suspect that with NO connection its handled, but it has a connection to OpenSpy but Openspy doesn't have a stats implementation... so it thinks there's a master stats server but not really. BTW it did have a class called OLStatsAccuracyCounter which seems to Calc fire accuracy (which based on values in DB isn't right) which means it tracking/logging all the shots -- seems like could be pretty CPU heavy. As far as Damage points it looks like it has a hook when player score changes, but it might require more event logging since damage points aren't added to normal scoring activity (kills, linking etc.).

But TL;DR is a bunch of code to wade through and debug, doesn't seem to be an easy fix. I tried setting a bunch of settings around sending stats to master to false, but it still doesn't work, server wont' start, there's something hard coded in how its calling the master server.
Post Reply