Unidentified Server's Crash (Shutdown)

[kung foo man]

Normally you should be able to just copy the working libmysqlclient.so over next to your cod2_lnxded binary and then "include" it with LD_PRELOAD_PATH=. (IIRC, did this stuff years ago)

This one is from 2009, might still do it's job great on your Ubuntu 12.04: http://killtube.org/downloads/libcod/libmysqlclient.so (or take the one from your working distribution)

OR:

Rename the default one to /usr/lib/i386-linux-gnu/libmysqlclient.so.18_backup and upload the working one from your working distribution (hacky hacky..)

The command "ldd cod2_lnxded" is pretty helpful to see which .so would be loaded.

Also try adding LD_DEBUG=libs for more debugging: http://www.bnikolic.co.uk/blog/linux-ld-debug.html


About the printf debug... I meant to add printf's in the libcod source code and recompiling it, but seems like we already know that it is the MySQL lib version, so no need for that anymore.

[Whiskas]

I'm still fighting with this problem :P.

What I've done so far:

1). Worked with Iznogod and his libcod fixes (sadly, it didn't fix the problem :|)
2). Upgraded Ubuntu to 14.04
3). Updated linux-libc-dev i386 to version 3.13.0-85.129 (Dunno if it's even related but it was outdated and I was updating everything I could)
4). Added LD_DEBUG=all to start script.
5). It seems that after the dist upgrade segfaults aren't saving core dumps. Tried ulimit -c unlimited, but it resets later (Checking it with ulimit -a). Any other way to save core dumps?
6). Before last segfault server skipped query (maybe it is related to the problem?)

Client 47 connecting with 250 challenge ping from 37.238.162.56:-31071
Going from CS_FREE to CS_CONNECTED for  (num 7 guid 0)
clientDownload: 7 : begining "whiskys.lair/zzz_modz.iwd"
clientDownload: 7 : file "whiskys.lair/zzz_modz.iwd" completed
clientDownload: 7 : begining "whiskys.lair/zzz_all_rifles_v1.5.iwd"
clientDownload: 7 : file "whiskys.lair/zzz_all_rifles_v1.5.iwd" completed
SELECT `ip`, `name`, `reason`, `whobanned` FROM `bans` WHERE `ip` = "37.238.162.56" LIMIT 1
SELECT `ip`, `name`, `reason`, `whobanned` FROM `bans` WHERE `ip` = "37.238.162.56" LIMIT 1
Sending heartbeat to cod2master.activision.com
3: EXE_DISCONNECTED
^9(GAME_AXIS)I<3evil.Server 445^7: QUICKMESSAGE_MOVE_IN
Client 48 connecting with 150 challenge ping from 78.38.19.196:10039
Going from CS_FREE to CS_CONNECTED for  (num 3 guid 0)
4: EXE_DISCONNECTED
Client 49 connecting with 250 challenge ping from 41.225.131.123:28960
Going from CS_FREE to CS_CONNECTED for  (num 16 guid 0)
13: EXE_DISCONNECTED
SELECT `ip`, `name`, `reason`, `whobanned` FROM `bans` WHERE `ip` = "41.225.131.123" LIMIT 1
SELECT `ip`, `name`, `reason`, `whobanned` FROM `bans` WHERE `ip` = "41.225.131.123" LIMIT 1
Rcon from 89.36.219.214:-10854:
say
Segmentation fault (core dumped)

		SearchForBan()
{
    myIP = self getIP();
    if(!isDefined(myIP))
        return;
    printf("SELECT `ip`, `name`, `reason`, `whobanned` FROM `bans` WHERE `ip` = \"%\" LIMIT 1\n", myIP);  
    maps\mp\gametypes\_mysql::asyncQuery("SELECT `ip`, `name`, `reason`, `whobanned` FROM `bans` WHERE `ip` = \"" + myIP + "\" LIMIT 1", ::checkBan);
    printf("SELECT `ip`, `name`, `reason`, `whobanned` FROM `bans` WHERE `ip` = \"%\" LIMIT 1\n", myIP);    
} 
	

Seems like myIP wasn't defined, from now I've added printf before return.

Btw. maybe check if isDefined(self) before getting IP?

Attaching logfile with LD_DEBUG=all https://www.sendspace.com/file/kq3tie

Thanks again!

[Whiskas]

Let me drop the big UPDATE here:

1). I am retarded.
2). I've been backtracing the old core dump from wrong mysql initiating (which was later resolved).
3). New backtracing look like this:

#0  0x080e983d in ?? ()
(gdb) bt
#0  0x080e983d in ?? ()
#1  0x0808f697 in ?? ()
#2  0x0808fa6f in ?? ()
#3  0x080943e1 in ?? ()
#4  0x08061779 in ?? ()
#5  0x08062451 in ?? ()
#6  0x0806251d in ?? ()
#7  0x080d2b22 in ?? ()
#8  0xb738aa83 in __libc_start_main (main=0x80d2990, argc=12, argv=0xbfd22604,
    init=0x8049b90, fini=0x8139b50, rtld_fini=0xb7722180 <_dl_fini>,
    stack_end=0xbfd225fc) at libc-start.c:287
#9  0x0804a4d1 in ?? ()

4). Thanks to Kung's lessons I've managed to check the code behind those addresess:
Behind #1 0x0808f697 in ?? () there is:

		int __cdecl sub_808F510(int a1, int a2, int a3)
{
  int result; // eax@4
  char v4; // [sp+4h] [bp-3D4h]@0
  int v5; // [sp+14h] [bp-3C4h]@9
  char *v6; // [sp+18h] [bp-3C0h]@9
  char *v7; // [sp+1Ch] [bp-3BCh]@11
  char v8[896]; // [sp+20h] [bp-3B8h]@11
  char s; // [sp+3A0h] [bp-38h]@9
  int v10; // [sp+3C4h] [bp-14h]@5
  int v11; // [sp+3C8h] [bp-10h]@9
  int i; // [sp+3CCh] [bp-Ch]@9
  char v13[8]; // [sp+3D0h] [bp-8h]@25

  if ( a3 )
    *(_DWORD *)(a1 + 134408) = *(_DWORD *)(a1 + 133144);
  else
    *(_DWORD *)(a1 + 134408) = -1;
  result = *(_DWORD *)(a1 + 133132) - *(_DWORD *)(a1 + 133136);
  if ( result <= 127 )
  {
    v10 = sub_8067EE8(a2);
    if ( v10 > 0 )
    {
      if ( v10 <= 32 )
      {
        v11 = dword_842BC8C;
        v11 = *(_DWORD *)(a1 + 133144) ^ dword_842BC8C;
        v11 ^= sub_80AA70C(a1 + 1032 * (*(_DWORD *)(a1 + 133136) & 0x7F) + 1036, 32);
        v5 = sub_808FE02(-1653759219 * ((a1 - (signed int)dword_841FB0C) >> 2));
        sub_80684E6(v5, &s);
        v6 = &s;
        for ( i = 0; i < v10; ++i )
        {
          v7 = &v8[28 * i];
          sub_8068ADE(a2, v11, v6, v7);
          if ( !(unsigned __int8)sub_80E97F0(v5, (unsigned __int8)v7[8]) )
            v7[8] = *(_BYTE *)(v5 + 212);
          if ( !(unsigned __int8)sub_80E97F0(v5, (unsigned __int8)v7[9]) )
            v7[9] = *(_BYTE *)(v5 + 208);
          v6 = v7;
        }
        *(_DWORD *)(a1 + 9924 * (*(_DWORD *)(a1 + 133144) & 0x1F) + 144352) = dword_841FB04;
        if ( *(_DWORD *)a1 == 3 )
          sub_808E1F0(a1, v8);
        if ( !*(_BYTE *)(dword_848B200 + 8) || *(_DWORD *)(a1 + 452016) )
        {
          if ( *(_DWORD *)a1 == 4 )
          {
            for ( i = 0; ; ++i )
            {
              result = i;
              if ( i >= v10 )
                break;
              if ( *(_DWORD *)&v13[28 * i - 944] <= *(_DWORD *)&v13[28 * v10 - 972]
                && *(_DWORD *)&v13[28 * i - 944] > *(_DWORD *)(a1 + 133156) )
              {
                sub_808F488(a1, &v8[28 * i]);
              }
            }
          }
          else
          {
            result = a1;
            *(_DWORD *)(a1 + 134408) = -1;
          }
        }
        else
        {
          result = sub_808DC8C(a1, "EXE_CANNOTVALIDATEPURECLIENT");
        }
      }
      else
      {
        result = sub_8060B2C("cmdCount > MAX_PACKET_USERCMDS\n", v4);
      }
    }
    else
    {
      result = sub_8060B2C("cmdCount < 1\n", v4);
    }
  }
  return result;
} 
	

5). Today on the server which partly uses my scripts I've noticed this:
http://i.imgur.com/HVVa5hn.jpg
It was spamming the hell out of the console.
6). thOuMta (server's admin) did not see it (only me).
7). To stop this spam I've used the cl_maxpackets "30" command. (this server had set 20 at default).
8). Kung foo man has pointed out the problem with "cmdCount > MAX_PACKET_USERCMDS".

Not sure what to do about it, Should I patch libcod with this? https://killtube.org/showthread.php?...tch-must-apply or just parse cl_maxpackets?
Or.. maybe as it was said in this thread https://killtube.org/showthread.php?...ACKET_USERCMDS I have "wait without ending on disconnect"
Could my self SearchForBan(); be treated as a wait and I should move it somewhere?

		SearchForBan()
{
    if(!isDefined(self))
    {
        printf("\n Self wasn't defined :(");
        return;
    }
    myIP = self getIP();
    if(!isDefined(myIP))
    {
        printf("\n myIP wasnt defined! \n");
        return;
    }
    printf("SELECT `ip`, `name`, `reason`, `whobanned` FROM `bans` WHERE `ip` = \"%\" LIMIT 1\n", myIP);  
    maps\mp\gametypes\_mysql::asyncQuery("SELECT `ip`, `name`, `reason`, `whobanned` FROM `bans` WHERE `ip` = \"" + myIP + "\" LIMIT 1", ::checkBan);
    printf("SELECT `ip`, `name`, `reason`, `whobanned` FROM `bans` WHERE `ip` = \"%\" LIMIT 1\n", myIP);    
}

checkBan(rows, args)
{
    
   if(isDefined(rows) && isDefined(rows[0]) && isDefined(rows[0]["ip"]) && isDefined(self))
   {
           kickmsg = rows[0]["name"] + "^7, You're banned by " + rows[0]["whobanned"] + "^7, because of reason: " + rows[0]["reason"] + ". Check www.whiskaskitler.xyz to get unbanned :) " + ". You will need your IP: " + rows[0]["ip"];
          kick2(self getEntityNumber(), kickmsg);
   }
}

Callback_PlayerConnect()
{
    thread dummy();
    
    self.statusicon = "hud_status_connecting";
    self waittill("begin");
    self.statusicon = "";
    level notify("connected", self);
    
    if(!level.splitscreen)
        iprintln(&"MP_CONNECTED", self);
    lpselfnum = self getEntityNumber();
    lpGuid = self getGuid();
    logPrint("J;" + lpGuid + ";" + lpselfnum + ";" + self.name + "\n");

    if(game["state"] == "intermission")
    {
        spawnIntermission();
        return;
    } 
    level endon("intermission");
    
    self SearchForBan();     

    if(level.splitscreen)
        scriptMainMenu = game["menu_ingame_spectator"];
    else
        scriptMainMenu = game["menu_ingame"];

    if(isDefined(self.pers["team"]) && self.pers["team"] != "spectator")
    {
        self setClientCvar("ui_allow_weaponchange", "1");

        if(self.pers["team"] == "allies")
            self.sessionteam = "allies";
        else
            self.sessionteam = "axis";

        if(isDefined(self.pers["weapon"]))
            spawnPlayer();
        else
        {
            spawnSpectator();

            if(self.pers["team"] == "allies")
            {
                self openMenu(game["menu_weapon_allies"]);
                scriptMainMenu = game["menu_weapon_allies"];
            }
            else
            {
                self openMenu(game["menu_weapon_axis"]);
                scriptMainMenu = game["menu_weapon_axis"];
            }
        }
    }
    else
    {
        self setClientCvar("ui_allow_weaponchange", "0");

        if(!isDefined(self.pers["skipserverinfo"]))
            self openMenu(game["menu_team"]);

        self.pers["team"] = "spectator";
        self.sessionteam = "spectator";

        spawnSpectator();
    }

    self setClientCvar("g_scriptMainMenu", scriptMainMenu);
} 
	

Sorry for the long post. I've wanted to make it with as many details as (I thought) it needs.

Thank you.

Edit: used cl_maxpackets "20" on my server and also got spammed.

[Mitch]

The rate limit patch is not related to crashes. It is for preventing people from generating too much traffic on your server.

I never found why the server crashes when you are seeing a few cmdCount > MAX_PACKET_USERCMDS in your log.
How often does this crash happen and are you able to replicate the crash? Replicating the crash will make it easier to test and fix the issue.

I don't think cmdCount > MAX_PACKET_USERCMDS (32) is the cause of the crash.
https://github.com/id-Software/Quake...er/sv_client.c

#0 0x080e983d traces back to:

		int __cdecl sub_80E97F0(int a1, int a2)
{
  int v3; // [sp+Ch] [bp-Ch]@5
  char v4; // [sp+17h] [bp-1h]@1

  v4 = 1;
  if ( !(unsigned __int8)sub_80E9758(a2) )
    v4 = 0;
  if ( !(unsigned __int8)sub_80D9E84(a1 + 1348, a2) )
    v4 = 0;
  v3 = sub_80E9270(a2); // weapon = getweapon(index)
  if ( !*(_DWORD *)(v3 + 132) // = 0x080e983d = weapon offhandClass
    && *(_BYTE *)(a1 + 1365) != a2
    && *(_BYTE *)(a1 + 1366) != a2
    && *(_DWORD *)(v3 + 876) != a2 )
    v4 = 0;
  return (unsigned __int8)v4;
} 
	

		v7 = &v8[28 * i];
sub_8068ADE(a2, v11, v6, &v8[28 * i]);
if ( !(unsigned __int8)sub_80E97F0(v5, (unsigned __int8)v7[8]) ) // probably player entity, weapon index 
	

It could that for some reason the weapon index is out of range.
One way to found out is to hook the sub_80E9270 call and print a message when the index is invalid.

[Whiskas]

How often does this crash happen and are you able to replicate the crash? Replicating the crash will make it easier to test and fix the issue.

Could be 10 crashes per day. Usually one by one, then peace for next ~14 hours. I didn't tried to replicate it .

#0 0x080e983d traces back to:

		int __cdecl sub_80E97F0(int a1, int a2)
{
  int v3; // [sp+Ch] [bp-Ch]@5
  char v4; // [sp+17h] [bp-1h]@1

  v4 = 1;
  if ( !(unsigned __int8)sub_80E9758(a2) )
    v4 = 0;
  if ( !(unsigned __int8)sub_80D9E84(a1 + 1348, a2) )
    v4 = 0;
  v3 = sub_80E9270(a2); // weapon = getweapon(index)
  if ( !*(_DWORD *)(v3 + 132) // = 0x080e983d = weapon offhandClass
    && *(_BYTE *)(a1 + 1365) != a2
    && *(_BYTE *)(a1 + 1366) != a2
    && *(_DWORD *)(v3 + 876) != a2 )
    v4 = 0;
  return (unsigned __int8)v4;
} 
	

		v7 = &v8[28 * i];
sub_8068ADE(a2, v11, v6, &v8[28 * i]);
if ( !(unsigned __int8)sub_80E97F0(v5, (unsigned __int8)v7[8]) ) // probably player entity, weapon index 
	

Yeah, Kung foo man suggested to hook up the address into libcod:

		#if COD_VERSION == COD2_1_0
            if (0)
                cracking_hook_function(0x08092D5C, (int)SV_AddServerCommand);
            if (0)
                cracking_hook_function(0x0809301C, (int)SV_SendServerCommand);

            cracking_hook_call(0x0808F134, (int)hook_ClientUserinfoChanged);
            cracking_hook_call(0x0807059F, (int)Scr_GetCustomFunction);
            cracking_hook_call(0x080707C3, (int)Scr_GetCustomMethod);
            cracking_hook_call(0x08098CD0, (int)custom_SV_WriteDownloadToClient);
            cracking_hook_call(0x080DFF66, (int)hook_player_setmovespeed);
            cracking_hook_call(0x080F50AB, (int)hook_player_g_speed);
            cracking_hook_call(0x080E9524, (int)hook_findWeaponIndex);
            cracking_hook_call(0x080E983D, (int)hook_findProblem); // This address 
	

and add printf with parameters in gsc_player.cpp

		int hook_findProblem(int a1, int a2) { 
    printf("findProblem: %d %d\n", a1, a2);
    return 1; 
} 
	

Also added function to gsc_player.hpp

		int hook_findProblem(int a1, int a2); 
	

We hoped that it fill print arguments before server crash. But at the next ~10 segfaults there was no output from this address. So I've started to check out previous addresses and as I found addresses with MAX_PACKET kung suggested to check out previous threads on the forum.

It could that for some reason the weapon index is out of range.
One way to found out is to hook the sub_80E9270 call and print a message when the index is invalid.

Will do this, but yesterday I've implemented your libcod version, parsed cl_maxpackets on PlayersConnects and moved SearchForBan(); at the end of the Callback_PlayerConnect(). For now I didn't have any segfaults which is record for my server. If segmentation fault will came back I let you know with provided printf of sub_80E9270. Thanks for your time!

Whisky