FYI. Might see some network P/L today..

[pooty]

From NFO
Re: Chicago issues
Jan 14 2024 11:48:10 AM PT The issues in Chicago are actually due to components in our routers overheating, due to a major datacenter outage with their cooling system. We are unable to stop the packet loss for now as a result. We are actively monitoring for updates from the facility on a fix.
Upstream link issue

[captainsnarf]

HOC, Landords, and Omni are currently down. SoL is still up, come play

[pooty]

Omni is up... I was just loading a new map, so its up and running, but maybe the network is playing games.

[McLovin]

1800 MST - Unplayable

[ankeedo]

routers overheating ?? call LEON the overheating expert

[leon]

Put it outdoor. It's - 4 in Chicago right now.

[California]

During this time of crisis we should come together as a community and watch Videodrome and perhaps play Overwatch. Stay safe out there kings.

[ankeedo]

[McLovin]

PERFECTION! This artwork needs to be in a game on a wall somewhere.

[pooty]

routers overheating ?? call LEON the overheating expert

Lolz!

I could connect but won't stay connected.

From NFO

Code: Select all

The issues in Chicago are actually due to components in our routers overheating, due to a major datacenter outage with their cooling system. We are unable to stop the packet loss for now as a result. We are actively monitoring for updates from the facility on a fix.

Update @ 3:48pm CST: The facility (Equinix) reports that they are continuing to try to recover from multiple failed chillers, which went down due to the low temperatures. Currently, temperatures are still rising in the facility, wreaking havoc on our equipment, including our routers. Our last look at our main router showed an internal case temperature of around 160 degrees F.

We have tried workarounds, but both of our redundant routers are seeing serious problems, including failed power supplies, due to the heat issue, and all of our upstream connections are intermittently offline because the network cards and cables are so hot. Only when they have brought temperatures down again will we be able to assess what is permanently broken and adapt to it.

Update @ 6:31pm CST: The facility still has not brought down temperatures, though they are trying with fans. Our primary router seems to have entirely failed, likely because it overheated; we are transitioning to exclusive use of our secondary router, but do not know how long it will hold out. Its upstream links are also still frequently going down.

Update @ 6:53pm CST: As the facility continues to have temperature rises, we are seeing switches and machines fail. We will have to assess these when the cooling is restored. We hope that there are no permanent hardware failures. 

Put it outdoor. It's - 4 in Chicago right now.

No shit, just open a window its so cold out (and dry) it'd cool that right down. See Leon has it handled!