Unidentified Server's Crash (Shutdown)

[Whiskas]

Hey, I'm hosting a server with some custom mods. I had many problems with server's crash. There were always an error saying why does the server crashed.
Now I'm struggling with server's "shutdown", but there is no any error which could help me specify the problem.

Here are the latest lines of the console_mp_server.log file (from the 3 different crashes (more like "shutdowns", because whole procces was killed o.O))

1). Today's first crash:

cmd 1703: 33027950: G 173
cmd 1704: 33029300: "MP_JOINED_ALLIES^^0Scorpion^5@^^0Your ^^0Ego^7"
cmd 1705: 33030000: G 174
cmd 1706: 33031300: G 175
cmd 1707: 33032500: v ui_bolt_fraggrenades1 "0"
cmd 1708: 33032500: v ui_bolt_fraggrenades2 "0"
cmd 1709: 33032500: v ui_bolt_fraggrenades3 "0"
cmd 1710: 33032500: v ui_semi_fraggrenades1 "0"
cmd 1711: 33032500: v ui_semi_fraggrenades2 "0"
cmd 1712: 33032500: v ui_semi_fraggrenades3 "0"
cmd 1713: 33034150: G 176
cmd 1714: 33034350: G 177
cmd 1715: 33036000: G 178
cmd 1716: 33037300: H 160
cmd 1717: 33037500: v ui_bolt_fraggrenades1 "0"
cmd 1718: 33037500: v ui_bolt_fraggrenades2 "0"
cmd 1719: 33037500: v ui_bolt_fraggrenades3 "0"
cmd 1720: 33037500: v ui_semi_fraggrenades1 "0"
cmd 1721: 33037500: v ui_semi_fraggrenades2 "0"
cmd 1722: 33037500: v ui_semi_fraggrenades3 "0"
cmd 1723: 33038700: G 179
cmd 1724: 33039000: v developer "0"
cmd 1725: 33039000: v r_xdebug "0"
cmd 1726: 33039000: I "1"
cmd 1727: 33039000: I "1"
cmd 1728: 33039000: v cg_objectiveText "MP_GAIN_POINTS_BY_ELIMINATING1350"
cmd 1729: 33039250: H 161
cmd 1730: 33041300: H 162
cmd 1731: 33042500: v ui_bolt_fraggrenades1 "0"
cmd 1732: 33042500: v ui_bolt_fraggrenades2 "0"
cmd 1733: 33042500: v ui_bolt_fraggrenades3 "0"
cmd 1734: 33042500: v ui_semi_fraggrenades1 "0"
cmd 1735: 33042500: v ui_semi_fraggrenades2 "0"
cmd 1736: 33042500: v ui_semi_fraggrenades3 "0"
cmd 1737: 33047500: v ui_bolt_fraggrenades1 "0"
cmd 1738: 33047500: v ui_bolt_fraggrenades2 "0"
cmd 1739: 33047500: v ui_bolt_fraggrenades3 "0"
cmd 1740: 33047500: v ui_semi_fraggrenades1 "0"
cmd 1741: 33047500: v ui_semi_fraggrenades2 "0"
cmd 1742: 33047500: v ui_semi_fraggrenades3 "0"
cmd 1743: 33052500: v ui_bolt_fraggrenades1 "0"
cmd 1744: 33052500: v ui_bolt_fraggrenades2 "0"
cmd 1745: 33052500: v ui_bolt_fraggrenades3 "0"
cmd 1746: 33052500: v ui_semi_fraggrenades1 "0"
cmd 1747: 33052500: v ui_semi_fraggrenades2 "0"
cmd 1748: 33052500: v ui_semi_fraggrenades3 "0"
cmd 1749: 33057500: v ui_bolt_fraggrenades1 "0"
cmd 1750: 33057500: v ui_bolt_fraggrenades2 "0"
cmd 1751: 33057500: v ui_bolt_fraggrenades3 "0"
cmd 1752: 33057500: v ui_semi_fraggrenades1 "0"
cmd 1753: 33057500: v ui_semi_fraggrenades2 "0"
cmd 1754: 33057500: v ui_semi_fraggrenades3 "0"
cmd 1755: 33062500: v ui_bolt_fraggrenades1 "0"
cmd 1756: 33062500: v ui_bolt_fraggrenades2 "0"
cmd 1757: 33062500: v ui_bolt_fraggrenades3 "0"
cmd 1758: 33062500: v ui_semi_fraggrenades1 "0"
cmd 1759: 33062500: v ui_semi_fraggrenades2 "0"
cmd 1760: 33062500: v ui_semi_fraggrenades3 "0"
cmd 1761: 33067500: v ui_bolt_fraggrenades1 "0"
cmd 1762: 33067500: v ui_bolt_fraggrenades2 "0"
cmd 1763: 33067500: v ui_bolt_fraggrenades3 "0"
cmd 1764: 33067500: v ui_semi_fraggrenades1 "0"
cmd 1765: 33067500: v ui_semi_fraggrenades2 "0"
cmd 1766: 33067500: v ui_semi_fraggrenades3 "0"
cmd 1767: 33072500: v ui_bolt_fraggrenades1 "0"
cmd 1768: 33072500: v ui_bolt_fraggrenades2 "0"
cmd 1769: 33072500: v ui_bolt_fraggrenades3 "0"
cmd 1770: 33072500: v ui_semi_fraggrenades1 "0"
cmd 1771: 33072500: v ui_semi_fraggrenades2 "0"
cmd 1772: 33072500: v ui_semi_fraggrenades3 "0"
cmd 1773: 33077500: v ui_bolt_fraggrenades1 "0"
cmd 1774: 33077500: v ui_bolt_fraggrenades2 "0"
cmd 1775: 33077500: v ui_bolt_fraggrenades3 "0"
cmd 1776: 33077500: v ui_semi_fraggrenades1 "0"
cmd 1777: 33077500: v ui_semi_fraggrenades2 "0"
cmd 1778: 33077500: v ui_semi_fraggrenades3 "0"
cmd 1779: 33082500: v ui_bolt_fraggrenades1 "0"
cmd 1780: 33082500: v ui_bolt_fraggrenades2 "0"
cmd 1781: 33082500: v ui_bolt_fraggrenades3 "0"
cmd 1782: 33082500: v ui_semi_fraggrenades1 "0"
cmd 1783: 33082500: v ui_semi_fraggrenades2 "0"
cmd 1784: 33082500: v ui_semi_fraggrenades3 "0"
cmd 1785: 33087500: v ui_bolt_fraggrenades1 "0"
cmd 1786: 33087500: v ui_bolt_fraggrenades2 "0"
cmd 1787: 33087500: v ui_bolt_fraggrenades3 "0"
cmd 1788: 33087500: v ui_semi_fraggrenades1 "0"
cmd 1789: 33087500: v ui_semi_fraggrenades2 "0"
cmd 1790: 33087500: v ui_semi_fraggrenades3 "0"
cmd 1791: 33092500: v ui_bolt_fraggrenades1 "0"
cmd 1792: 33092500: v ui_bolt_fraggrenades2 "0"
cmd 1793: 33092500: v ui_bolt_fraggrenades3 "0"
cmd 1794: 33092500: v ui_semi_fraggrenades1 "0"
cmd 1795: 33092500: v ui_semi_fraggrenades2 "0"
cmd 1796: 33092500: v ui_semi_fraggrenades3 "0"
cmd 1797: 33093100: v developer "0"
cmd 1798: 33011950: H 157
cmd 1799: 33093100: v r_xdebug "0"
Unknown Soldier: Delta request from out of date packet.
Going to CS_ZOMBIE for
10: EXE_SERVERCOMMANDOVERFLOW
Going from CS_ZOMBIE to CS_FREE for
Going from CS_PRIMED to CS_ACTIVE for ^7I^1<3^9evil^1.^7Server 338

2). Second:


Client 55 connecting with 50 challenge ping from //HERE WAS SOME IP:28960
Going from CS_FREE to CS_CONNECTED for (num 6 guid 0)
clientDownload: 6 : begining "whiskys.lair/zzz_all_rifles_v1.5.iwd"
clientDownload: 6 : file "whiskys.lair/zzz_all_rifles_v1.5.iwd" completed
^9(GAME_AXIS) ^7: QUICKMESSAGE_NO_SIR
^8^3?^7: yes
6: EXE_DISCONNECTED
^9(GAME_AXIS) ^7: QUICKMESSAGE_NO_SIR
^9(GAME_AXIS) ^7: QUICKMESSAGE_NO_SIR


3). Third:

Client 98 connecting with 100 challenge ping from //HERE WAS SOME IP:28960
Going from CS_FREE to CS_CONNECTED for (num 1 guid 0)
Client 99 connecting with 100 challenge ping from //HERE WAS SOME IP:28960
Going from CS_FREE to CS_CONNECTED for (num 5 guid 0)
clientDownload: 1 : begining "whiskys.lair/zzz_all_rifles_v1.5.iwd"
clientDownload: 1 : file "whiskys.lair/zzz_all_rifles_v1.5.iwd" completed
Rcon from //HERE WAS MY SERVERS IP:-25423:
say
^9(GAME_AXIS)^4Balbazar^7: QUICKMESSAGE_SORRY
^9(GAME_AXIS)^4Balbazar^7: QUICKMESSAGE_SORRY
^8(GAME_ALLIES)gAy-bIrD...^7: QUICKMESSAGE_SORRY
^8gAy-bIrD...^7: ownd
^8(GAME_ALLIES)gAy-bIrD...^7: QUICKMESSAGE_SORRY
^8(GAME_ALLIES)gAy-bIrD...^7: QUICKMESSAGE_GREAT_SHOT
Rcon from //HERE WAS MY SERVERS IP:-17623:
say
^9(GAME_AXIS)^4Balbazar^7: QUICKMESSAGE_SORRY
^9(GAME_AXIS)^4Balbazar^7: QUICKMESSAGE_SORRY
4: EXE_DISCONNECTED
Sending heartbeat to cod2master.activision.com
Client 100 connecting with 150 challenge ping from //HERE WAS SOME IP
Going from CS_FREE to CS_CONNECTED for (num 4 guid 0)
Rcon from //HERE WAS SOME IP:-11712:
say
^8(GAME_ALLIES)gAy-bIrD...^7: QUICKMESSAGE_SORRY
^8(GAME_ALLIES)gAy-bIrD...^7: QUICKMESSAGE_SORRY
Client 101 connecting with 50 challenge ping from //HERE WAS SOME IP
Going from CS_FREE to CS_CONNECTED for (num 7 guid 0)
^9(GAME_AXIS)^4Balbazar^7: QUICKMESSAGE_SORRY


Also I have to say that I'm using TMUX to run server in the background and usually when server has crash (cause of some script error), the tmux process will stay alive. But in this case, whole (TMUX's) process(or session?) is killed.

I will provide any suspicious script.

Thank you for your help (or any tip) in advance.

[kung foo man]

1) Does it happen with screen too?

2) How many servers are you running on that box? When it's the only CoD2 server on a VPS box, maybe it was a forced restart from host, which would explain the killed tmux session.

Check last reboot:



root@euve78301:~# last reboot

wtmp begins Sat Mar 5 02:12:23 2016


3) Otherwise probably google "tmux log crash" ^^

[voron00]

I just dropped your server by using one of the 'well known nukes' (the good old va() bug), you need to use a patched binary from here: http://killtube.org/showthread.php?2128-Combine-all-fixes-into-1-binary&p=11345&viewfull=1#post11345

Edit: That proably not the actual thing with the crashes but still, some people are abusing this stuff.

[Whiskas]

1) Does it happen with screen too?


I will test it now.



2) How many servers are you running on that box? When it's the only CoD2 server on a VPS box, maybe it was a forced restart from host, which would explain the killed tmux session.


Two servers, and it's always only one of them that is "shutting down". Also I have auto start with reboot.



Check last reboot:



root@euve78301:~# last reboot

wtmp begins Sat Mar 5 02:12:23 2016



who -b
system boot 2016-03-11 04:00

But it's my "forced" reboot, as I set it in the CronTab. 1st server is dead, thanks to Voron :P, but the second still works.



3) Otherwise probably google "tmux log crash" ^^

I'll try this https://github.com/tmux-plugins/tmux-logging . Cause I'm can't find any log.
First server will be turned on with screen and I let second live on TMUX, but with logging. Let's see what happens. Thanks!


I just dropped your server by using one of the 'well known nukes' (the good old va() bug), you need to use a patched binary from here: http://killtube.org/showthread.php?2128-Combine-all-fixes-into-1-binary&p=11345&viewfull=1#post11345

Edit: That proably not the actual thing with the crashes but still, some people are abusing this stuff.

Will use this from now, thanks. I think there was one time I've seen in some crash logs this NAKAttempt.


P.S I think (not sure) I witnessed this "shutdown". Server seem to work but when I disconnected I couldn't reconnect back there, and the server wasn't listed on servers list.

Console log:

cmd 1480: 1090200: v ui_bolt_fraggrenades1 "0"
cmd 1481: 1090200: v ui_bolt_fraggrenades2 "0"
cmd 1482: 1090200: v ui_bolt_fraggrenades3 "0"
cmd 1483: 1090200: v ui_semi_fraggrenades1 "0"
cmd 1484: 992000: G 29
cmd 1485: 1090200: v ui_semi_fraggrenades2 "0"
1: EXE_SERVERCOMMANDOVERFLOW
Sending heartbeat to cod2master.activision.com
0: EXE_DISCONNECTED
Sending heartbeat to cod2master.activision.com

And the output from screen's window:


File Handles:
handle 1: console_mp_server.log
----------------------
42143 files in iwd files
------- Game Initialization -------
gamename: Call of Duty 2
gamedate: Oct 24 2005
-----------------------------------
scriptengine> Async mysql already initialized. Returning before adding additional connections
-----------------------------------
Hitch warning: 1974 msec frame time
Sending heartbeat to cod2master.activision.com

[Whiskas]

Update:

These shutdown were when I wasn't back home. I sometimes check GamesTracker website to check if my server is alive. If it's DEAD I also check console log and type this command:


echo -e '\xff\xff\xff\xffgetstatus' | netcat -u 89.36.219.214 28960

If it's gave me no output at all, I suggested that server was dead and I remotely restart it.

Today I had some time to play. I was playing for about 30 mins, checked gamestracker and it says that server is down and that the last scan was 10 mins ago. I was quite suprised so typed above command and yeah, it gave me nothing despite I was playing server with other players and more has came. Server was listed in call of duty's server list and disconnecting/connecting went without any problems.

Who knows, maybe there were no shutdowns at all and I was remotely restarting the server because of getstatus null response :P

Is there any known getstatus bug?

Edit:
This http://killtube.org/showthread.php?2147-Receiving-server-info&p=11571&viewfull=1#post11571 might be related if I use something like this? :

numberofplayers.sh

#!/bin/bash

/home/codserver2/testscript
/home/codserver/testscript



testscript

#!/bin/bash

#Execute starttest (tmux script) which uses echo getstatus of 'spected' server
#It is executed as linux user "codserver"
su --login codserver2 --command "/home/codserver/starttest"

#starttest has an infinite loop
#Added short time sleep to make sure starttest made whole dump
sleep 2

#Kill this infinite loop for fuck sake!
su --login codserver2 --command "/home/codserver/stoptest"

#Our starttest saved his output to an 'output' file
#Here we are couting lines in output file and saving as variable
players="$(wc -l /home/codserver2/output | awk '{print $1}')"

#Executing php file with our variable as a parameter
php /home/codserver2/players_num.php $players

#Delete this temporary file
#rm /home/codserver2/output

starttest

tmux new-session -d -s codcheck2 "echo -e '\xff\xff\xff\xffgetstatus' | netcat -u 89.36.219.214 28960 > /home/codserver2/output"

stoptest

tmux kill-session -t codcheck2

players_num.php

<?php
$number = $argv[1];
$server_ip = '89.36.219.214';
$server_port = '28961';
$server_timeout = 10;
$number2 = (int)$number;
$number2 = $number2 - 2;



$server_addr = "udp://" . $server_ip;
@$connect = fsockopen($server_addr, $server_port, $re, $errstr, $server_timeout);
if (! $connect)
{
echo 'Not working 1';
}
socket_set_timeout ($connect, $server_timeout);

$server_rconpass = 'HEREISMYRCONPASS';
$server_extra_wait = false;
$server_extra_footer = true;
$server_buffer_results = 2048; // enter a number of bytes; decrease if you receive only a part of returned results, increase to speed up
$server_buffer_cur = $server_buffer_results;
$cmd = "say Server Status: ^9crack_hj_tdm^6!^3.. ^7" . $number2 . "/20 players.";

//global $server_rconpass, $connect, $server_extra_wait, $server_buffer_cur, $server_extra_footer;
$send = "\xff\xff\xff\xff" . 'rcon "' . $server_rconpass . '" '.$cmd.(($server_extra_footer)?"\n":'');
fwrite($connect, $send);

$output = (($server_extra_wait)?(fread ($connect, 1)):'');
do {
$status_pre = socket_get_status ($connect);
if ((($server_extra_wait) && ($output != '')) || (! $server_extra_wait))
$output .= fread ($connect, $server_buffer_cur);
$status_post = socket_get_status ($connect);
} while ($status_pre['unread_bytes'] != $status_post['unread_bytes']);

return $output;
fclose($connect);
?>

It's added in crobtab and spams nearly every 3 mins. Maybe GameTracker's getstatus is sometimes executed in same time as my numberofplayers.sh script?

[Whiskas]

Sorry for triple posting, but I wanted to say that I let live my server that had "getstatuts?" bug. But screen's session died later anyway (same as tmux was dying).
I've turned off any bash/php script related to my server and we will see if anything happends.

[IzNoGoD]

next time your getstatus doesnt return anything: check net_port on your server. Sometimes the port is "in use" on mapchange, causing cod to go to port+1 and itll keep running from there

[maxdamage99]

My server crash error:


WARNING: bad command byte 7 for client 8
WARNING: bad command byte 7 for client 8
WARNING: bad command byte 7 for client 8

help me pls

[Mitch]

My server crash error:


WARNING: bad command byte 7 for client 8
WARNING: bad command byte 7 for client 8
WARNING: bad command byte 7 for client 8

help me pls

Maybe start your own thread? Because your problem doesn't seem related to the crashes in this thread.
Also which cod2 version? Libcod? Which libcod version/build? Is there anything else that you think we should know?

[maxdamage99]

libcod2_1_0, cod2_lnxded 1.0
(libcod last version, without grenade_damage)

[Whiskas]

I had no downtimes until one player came... I'm not sure how, but he makes Segmentation Fault on connect.

8: EXE_DISCONNECTED
Rcon from 89.36.219.214:-8170:
say
12: EXE_DISCONNECTED
Client 362 connecting with 200 challenge ping from 118.107.128.42:28960
Going from CS_FREE to CS_CONNECTED for (num 8 guid 0)
Segmentation fault (core dumped)
./cod2_lnxded_1_0: /usr/lib/i386-linux-gnu/libmysqlclient.so.18: no version information available (required by ./libcod2_1_0.so)
> [INFO] Compiled for: CoD2 1.0
Compiled: Jan 29 2015 02:23:00
> [INFO] value of closer=08111d32
> [INFO] value of download=0808e508
> [PLUGIN LOADED]
CoD2 MP 1.0 build linux-i386 Oct 24 2005
----- FS_Startup -----
Current search path:
/home/codserver/.callofduty2/whiskys.lair
/home/codserver/whiskys.lair/zzz_modz.iwd (3 files)
/home/codserver/whiskys.lair/zzz_all_rifles_v1.5.iwd (18 files)
/home/codserver/whiskys.lair
/home/codserver/.callofduty2/main
/home/codserver/main/iw_14.iwd (4038 files)
/home/codserver/main/iw_13.iwd (22624 files)
/home/codserver/main/iw_12.iwd (1016 files)
/home/codserver/main/iw_11.iwd (1462 files)
/home/codserver/main/iw_10.iwd (1936 files)
/home/codserver/main/iw_09.iwd (2142 files)
/home/codserver/main/iw_08.iwd (2723 files)
/home/codserver/main/iw_07.iwd (3384 files)
/home/codserver/main/iw_06.iwd (990 files)
/home/codserver/main/iw_05.iwd (928 files)
/home/codserver/main/iw_04.iwd (698 files)
/home/codserver/main/iw_03.iwd (26 files)
/home/codserver/main/iw_02.iwd (40 files)
/home/codserver/main/iw_01.iwd (16 files)
/home/codserver/main/iw_00.iwd (102 files)
/home/codserver/main
/home/codserver/.callofduty2/raw
/home/codserver/.callofduty2/raw_shared
/home/codserver/.callofduty2/devraw
/home/codserver/.callofduty2/devraw_shared
/home/codserver/raw
/home/codserver/raw_shared
/home/codserver/devraw
/home/codserver/devraw_shared

File Handles:
----------------------
42146 files in iwd files
execing default_mp.cfg
couldn't exec language.cfg
execing config_mp_server.cfg
dedicated is read only.
fs_game is write protected.
Opening IP socket: localhost:28960
Hostname: localhost.domain
Alias: localhost
Alias: whiskers
Alias: whiskers
IP: 127.0.0.1
IP: 89.36.219.214
--- Common Initialization Complete ---
net_port is write protected.
dedicated is read only.
fs_game is write protected.
execing server.cfg
net_ip is write protected.
net_port is write protected.
dedicated is read only.
sv_cheats is write protected.
map_rotate...

"sv_mapRotation" is:"gametype tdm map mp_toujane gametype tdm map mp_carentan gametype tdm map mp_trainstation gametype tdm map mp_decoy gametype tdm map mp_farmhouse gametype tdm map mp_burgundy gametype tdm map mp_brecourt gametype tdm map mp_railyard gametype tdm map mp_downtown gametype tdm map mp_leningrad gametype tdm map mp_dawnville gametype tdm map mp_matmata gametype tdm map mp_breakout"

"sv_mapRotationCurrent" is:""

Setting g_gametype: tdm.
Setting map: mp_toujane.
manymaps> map=mp_toujane sv_iwdNames:
manymaps> link src=/home/codserver/.callofduty2/whiskys.lair/Library/mp_toujane.iwd dst=/home/codserver/.callofduty2/whiskys.lair/mp_toujane.iwd
------ Server Initialization ------
Server: mp_toujane
----- FS_Startup -----
Current search path:
/home/codserver/.callofduty2/whiskys.lair
/home/codserver/whiskys.lair/zzz_modz.iwd (3 files)
/home/codserver/whiskys.lair/zzz_all_rifles_v1.5.iwd (18 files)
/home/codserver/whiskys.lair
/home/codserver/.callofduty2/main
/home/codserver/main/iw_14.iwd (4038 files)
/home/codserver/main/iw_13.iwd (22624 files)
/home/codserver/main/iw_12.iwd (1016 files)
/home/codserver/main/iw_11.iwd (1462 files)
/home/codserver/main/iw_10.iwd (1936 files)
/home/codserver/main/iw_09.iwd (2142 files)
/home/codserver/main/iw_08.iwd (2723 files)
/home/codserver/main/iw_07.iwd (3384 files)
/home/codserver/main/iw_06.iwd (990 files)
/home/codserver/main/iw_05.iwd (928 files)
/home/codserver/main/iw_04.iwd (698 files)
/home/codserver/main/iw_03.iwd (26 files)
/home/codserver/main/iw_02.iwd (40 files)
/home/codserver/main/iw_01.iwd (16 files)
/home/codserver/main/iw_00.iwd (102 files)
/home/codserver/main
/home/codserver/.callofduty2/raw
/home/codserver/.callofduty2/raw_shared
/home/codserver/.callofduty2/devraw
/home/codserver/.callofduty2/devraw_shared
/home/codserver/raw
/home/codserver/raw_shared
/home/codserver/devraw
/home/codserver/devraw_shared

File Handles:
----------------------
42146 files in iwd files
------- Game Initialization -------
gamename: Call of Duty 2
gamedate: Oct 24 2005
-----------------------------------
-----------------------------------
Hitch warning: 2621 msec frame time
Resolving cod2master.activision.com
cod2master.activision.com resolved to 185.34.104.231:20710
Sending heartbeat to cod2master.activision.com
Resolving cod2master.activision.com
cod2master.activision.com resolved to 185.34.104.231:20700
Client 0 connecting with 50 challenge ping from 188.127.12.144:28960
Going from CS_FREE to CS_CONNECTED for (num 0 guid 735737)
Sending heartbeat to cod2master.activision.com
Client 1 connecting with 200 challenge ping from 118.107.128.42:28960
Going from CS_FREE to CS_CONNECTED for (num 1 guid 0)
Segmentation fault (core dumped)

In tdm.gsc

Callback_PlayerConnect()
{
thread dummy();

self.statusicon = "hud_status_connecting";
self waittill("begin");
self.statusicon = "";
level notify("connected", self);

if(!level.splitscreen)
iprintln(&"MP_CONNECTED", self);
lpselfnum = self getEntityNumber();
lpGuid = self getGuid();
logPrint("J;" + lpGuid + ";" + lpselfnum + ";" + self.name + "\n");

self SearchForBan();

if(game["state"] == "intermission")
{
spawnIntermission();
return;
}
level endon("intermission");

if(level.splitscreen)
scriptMainMenu = game["menu_ingame_spectator"];
else
scriptMainMenu = game["menu_ingame"];

if(isDefined(self.pers["team"]) && self.pers["team"] != "spectator")
{
self setClientCvar("ui_allow_weaponchange", "1");

if(self.pers["team"] == "allies")
self.sessionteam = "allies";
else
self.sessionteam = "axis";

if(isDefined(self.pers["weapon"]))
spawnPlayer();
else
{
spawnSpectator();

if(self.pers["team"] == "allies")
{
self openMenu(game["menu_weapon_allies"]);
scriptMainMenu = game["menu_weapon_allies"];
}
else
{
self openMenu(game["menu_weapon_axis"]);
scriptMainMenu = game["menu_weapon_axis"];
}
}
}
else
{
self setClientCvar("ui_allow_weaponchange", "0");

if(!isDefined(self.pers["skipserverinfo"]))
self openMenu(game["menu_team"]);

self.pers["team"] = "spectator";
self.sessionteam = "spectator";

spawnSpectator();
}

self setClientCvar("g_scriptMainMenu", scriptMainMenu);
}

Added functioncs

SearchForBan()
{
myIP = self getIP();
maps\mp\gametypes\_mysql::asyncQuery("SELECT `ip`, `name`, `reason`, `whobanned` FROM `bans` WHERE `ip` = \"" + myIP + "\" LIMIT 1", ::checkBan);
}

checkBan(rows, args)
{

if(isDefined(rows) && isDefined(rows[0]) && isDefined(rows[0]["ip"]))
{
kickmsg = rows[0]["name"] + "^7, You're banned by " + rows[0]["whobanned"] + "^7, because of reason: " + rows[0]["reason"] + ". Check www.whiskaskitler.xyz to get unbanned :) " + ". You will need your IP: " + rows[0]["ip"];
kick2(self getEntityNumber(), kickmsg);
}
}

Also _callbacksetup.gsc

CodeCallback_PlayerConnect()
{
self endon("disconnect");
[[level.callbackPlayerConnect]]();
self.pers["admin"] = 0;
}

Anything that could make this problem? Should I add Ip of this player to iptables to prevent new downtimes?

[kung foo man]

Load the core dumb in gdb and show the trace, iirc just type "bt" for backtrace

[Whiskas]

Program terminated with signal 11, Segmentation fault.
#0 0xb716ab4b in mysql_options ()
from /usr/lib/i386-linux-gnu/libmysqlclient.so.18
(gdb) bt
#0 0xb716ab4b in mysql_options ()
from /usr/lib/i386-linux-gnu/libmysqlclient.so.18
#1 0xb773ccb1 in ?? ()
#2 0x00000000 in ?? ()


Does Core store only new dump?

Also providing _mysql.gsc file:

init()
{
//get your host, user, pass, db, port here
level.JH_mysql = mysql_reuse_connection();
if(!isDefined(level.JH_mysql))
level.JH_mysql = initMySQL("localhost", "root", "passhere", "cod_db", 3306);
initAsyncMySQL("localhost", "root", "passhere", "cod_db", 3306);
}

query(query)
{
if(!isDefined(level.JH_mysql))
return undefined;
result = mysql_query(level.JH_mysql, query);
resettimeout();
if(result)
{
printf("Error in " + query + "\n");
return undefined;
}
result = mysql_store_result(level.JH_mysql);
rows = getRows(result);
return rows;
}

queryNosave(query)
{
if(!isDefined(level.JH_mysql))
return undefined;
result = mysql_query(level.JH_mysql, query);
resettimeout();
if(result)
{
printf("Error in " + query + "\n");
return undefined;
}
return [];
}

getRows(result)
{
if(!isDefined(result))
return [];
rowcount = mysql_num_rows(result);
fields = [];
field = mysql_fetch_field(result);
while(isDefined(field))
{
fields[fields.size] = field;
field = mysql_fetch_field(result);
}
rows = [];
for(i = 0; i < rowcount; i++)
{
row = mysql_fetch_row(result);
rows[rows.size] = [];
for(j = 0; j < fields.size; j++)
rows[rows.size - 1][fields[j]] = row[j];
}
mysql_free_result(result);
return rows;
}

asyncQuery(query, function, args)
{
id = mysql_async_create_Query(query);
task = spawnstruct();
task.query = query;
task.invoker = self;
task.function = function;
task.args = args;
level.JH_mysqlAsync["" + id] = task;
}

asyncQueryNosave(query, function, args)
{
id = mysql_async_create_Query_Nosave(query);
task = spawnstruct();
task.query = query;
task.invoker = self;
task.function = function;
task.args = args;
level.JH_mysqlAsync["" + id] = task;
}

initAsyncMySQL(host, user, pass, db, port)
{
mysql_async_initializer(host, user, pass, db, port, 4);
level.JH_mysqlAsync = [];
thread loopAsyncMySQL();
}

loopAsyncMySQL()
{
while(true)
{
list = mysql_async_GetDone_List();
for(i = 0; i < list.size; i++)
{
result = mysql_async_getresult_and_free(list[i]);
if(!isdefined(result))
continue;
if(result == 0)
result = undefined;
task = "" + list[i];
if(isDefined(level.JH_mysqlAsync[task]))
{
if(isDefined(level.JH_mysqlAsync[task].function))
{
if(isDefined(level.JH_mysqlAsync[task].invoker))
{
rows = getRows(result);
level.JH_mysqlAsync[task].invoker thread [[level.JH_mysqlAsync[task].function]](rows, level.JH_mysqlAsync[task].args);
}
else if(isDefined(result))
mysql_free_result(result);
}
else if(isDefined(result))
mysql_free_result(result);
}
else if(isDefined(result))
mysql_free_result(result);
level.JH_mysqlAsync[task] = undefined;
}
wait .05;
}
}

initMySQL(host, user, pass, db, port)
{
mysql = mysql_init();
ret = mysql_real_connect(mysql, host, user, pass, db, port);
if(!ret)
{
printf("errno=" + mysql_errno(mysql) + " error= " + mysql_error(mysql) + "\n");
mysql_close(mysql);
return undefined;
}
return mysql;
}
stripString(string)
{
if(isDefined(level.JH_mysql))
return mysql_real_escape_string(level.JH_mysql, string);
return "";
}

I also have to admit that this player IP is not banned (it's not in the mysql table).

Thanks for your time!

[IzNoGoD]

Please printf the query

[maxdamage99]

I make my ban system, Whiskas check for all queryS if(isDefined..
sometime player disconnect and cod2 make query for null login and server crash by error: Segmentation Fault

[Whiskas]

printf supports more than one argument, in the form of printf("Hello %\n", "world"); and as such supports all types of secondary arguments. All % will be replaced by additional input arguments. If no arguments are given, the % will be removed. To print a % to console, use printf("this works 100%%");


Is it what you said, IzNoGoD?:


SearchForBan()
{
myIP = self getIP();
maps\mp\gametypes\_mysql::asyncQuery(printf("SELECT `ip`, `name`, `reason`, `whobanned` FROM `bans` WHERE `ip` = \"%\" LIMIT 1", myIP), ::checkBan);
}


maxdamage99
I make my ban system, Whiskas check for all queryS if(isDefined..
sometime player disconnect and cod2 make query for null login and server crash by error: Segmentation Fault

You mean to check if any row from query is defined? I'm not sure what you mean.

[IzNoGoD]

no, do a printf("select etcetera") BEFORE you do the asyncquery line

[Whiskas]

SearchForBan()
{
myIP = self getIP();
printf("SELECT `ip`, `name`, `reason`, `whobanned` FROM `bans` WHERE `ip` = \"%\" LIMIT 1", myIP);
maps\mp\gametypes\_mysql::asyncQuery("SELECT `ip`, `name`, `reason`, `whobanned` FROM `bans` WHERE `ip` = \"" + myIP + "\" LIMIT 1", ::checkBan);
}

This? But what it's going on here? :eek:

[IzNoGoD]

I'm not sure your query is actually defined, if myip is undefined, the whole server might crash.

If nothing else, it can show you the last query done before crashing

You should also add a printf when 1) starting to parse the results of the query and 2) when you're done parsing the results of the query

[maxdamage99]

callback_PlayerConnect()
{
/*....*/
//CHECK_BAN
login="";
if(isDefined(self.stats["login"])) login=self.stats["login"]; //without if, sometimes server crash :(
if(isBan(self getip()))
{
ban=info_ipban(self getip());
saychat(-1, "^7: Player: "+self.name+" ^7banned from this server. ^5Reason^7: "+ban["reason"]+"^7; ^1ID^7: "+ban["id"]+"^4.");
self setClientCvar("com_errorTitle", "^7You are ^1BANNED^7!!!");
self setClientCvar("com_errorMessage", self.name + " \n^4|| ^7BAN ID^5: ^3"+ban["id"]+" ^4||\n^4|| ^7ADMIN^5:^7 "+ban["admin"]+" ^4||\n^4|| ^7Date (Moscow)^5:^7 "+ban["date"]+"^4 ||\n^4|| ^7Reason^5:^7 "+ban["reason"]+" ^4||\n");
self closeMenu();
self closeInGameMenu();
msgconsole(self.name+" KICKED BY IP_BAN "+self getip()+" | "+login); //printf
self exec("disconnect");
return;
}
//CHECK_BAN
///////////////////////
isBan(ip)
{
if(!isDefined(ip)) return; //if make query by argument ip (NULL) server crash
sql = "SELECT * FROM `table_bans` WHERE `ban_ip`='" + ip + "'";
msgconsole(ip+" isBan(190) "+sql);
mysql_query(level.mysql, sql);
result = mysql_store_result(level.mysql);
if(mysql_num_rows(result) >= 1)
{mysql_free_result(result);return 1;}
mysql_free_result(result);
return 0;
}
info_ipban(ip)
{
if(!isDefined(ip)) return; //if make query by argument ip (NULL) server crash
info=[];
info["id"]=999;
info["admin"]="Unknown Soldier";
info["date"]=readRealTime();
info["reason"]="undefined";
sql = "SELECT `id`, `ban_reason`, `ban_date`, `admin_name` FROM `table_bans` WHERE `ban_ip`='" + ip + "'";
msgconsole(ip+" info_ipban(207) "+sql);
mysql_query(level.mysql, sql);
result = mysql_store_result(level.mysql);
if(!isDefined(result)) {mysql_free_result(result);return info;}
if(mysql_num_rows(result) != 1)
return info;
row = mysql_fetch_row(result);
info["id"] = int(row[0]);
info["admin"] = row[3];
info["date"] = row[2];
info["reason"] = row[1];
mysql_free_result(result);
return info;
}

P.S: dont use async query, my server is lag (VDS BY OPENVZ), i change OPEVNZ on KVM, and without async server normal job (no lag, uptime > 20 days), async - server no lag, but sometime server crash!

[kung foo man]

Does Core store only new dump?


By default the name is always "core" I think, hence overwriting old core dumps, but you can change the name pattern and add the PID for example: https://sigquit.wordpress.com/2009/03/13/the-core-pattern/

Kinda filthy to debug a C API crash, to be able to analyze those crashes in my MySQL part I added debug statements in every C glue function, like:

https://github.com/kungfooman/libcod/blob/master/gsc_mysql.cpp#L483


#if DEBUG_MYSQL
printf("gsc_mysql_num_fields(result=%d)\n", result);
#endif


If I would want to tackle the problem, I would first give every function a debug printf, starting with the functions which call mysql_options(), since that was mentioned in the gdb backtrace.

[IzNoGoD]

Stop blaming async mysql for your crashes unless you can show a stacktrace.

Async mysql is required if you run your mysql server on a different box, as network laggs transition into server laggs.

Got a box in australia that communicates with a mysql server in germany without any laggs at all, using async mysql.

[maxdamage99]

+ async: no lags
- async: my server crash by error Segmentatio Fault
PS: i remove async and make normal query (mysql_query(...)) - server stop crashing (uptime > 20 days)

[IzNoGoD]

Have you tried debugging where that async query fails? I got servers with lots of uptime using async mysql without any problems.

[Whiskas]

By default the name is always "core" I think, hence overwriting old core dumps

Thanks, I was worried it maybe error I had before.



maxdamage99
I make my ban system, Whiskas check for all queryS if(isDefined..
sometime player disconnect and cod2 make query for null login and server crash by error: Segmentation Fault

Thanks.


SearchForBan()
{
myIP = self getIP();
if(!isDefined(myIP))
return;
printf("SELECT `ip`, `name`, `reason`, `whobanned` FROM `bans` WHERE `ip` = \"%\" LIMIT 1", myIP);
maps\mp\gametypes\_mysql::asyncQuery("SELECT `ip`, `name`, `reason`, `whobanned` FROM `bans` WHERE `ip` = \"" + myIP + "\" LIMIT 1", ::checkBan);
printf("SELECT `ip`, `name`, `reason`, `whobanned` FROM `bans` WHERE `ip` = \"%\" LIMIT 1", myIP);
}


I'm not sure your query is actually defined, if myip is undefined, the whole server might crash.


Didn't know that 'myIP' can sometimes be undefined. :O


P.S: dont use async query, my server is lag (VDS BY OPENVZ), i change OPEVNZ on KVM, and without async server normal job (no lag, uptime > 20 days), async - server no lag, but sometime server crash!

I would like to stay with async query. It is superior to me. If there will be no async I think that I wouldn't add mysql at all, but thanks for your tip.

Thanks for every reply guys. You're amazing. Hope my server's uptime will be increased. :)

[Whiskas]

Today I had 2 downtimes one by one.

Client 211 connecting with 150 challenge ping from 188.215.242.10:13732
Going from CS_FREE to CS_CONNECTED for (num 9 guid 0)
clientDownload: 9 : begining "whiskys.lair/zzz_modz.iwd"
clientDownload: 9 : file "whiskys.lair/zzz_modz.iwd" completed
clientDownload: 9 : begining "whiskys.lair/zzz_all_rifles_v1.5.iwd"
Sending heartbeat to cod2master.activision.com
clientDownload: 9 : file "whiskys.lair/zzz_all_rifles_v1.5.iwd" completed
1: EXE_DISCONNECTED
Client 212 connecting with 500 challenge ping from 39.54.158.44:28960
Going from CS_FREE to CS_CONNECTED for (num 1 guid 0)
Client 213 connecting with 100 challenge ping from 213.179.252.246:26160
Going from CS_FREE to CS_CONNECTED for (num 10 guid 0)
clientDownload: 10 : begining "whiskys.lair/zzz_modz.iwd"
clientDownload: 10 : file "whiskys.lair/zzz_modz.iwd" completed
clientDownload: 10 : begining "whiskys.lair/zzz_all_rifles_v1.5.iwd"
SELECT `ip`, `name`, `reason`, `whobanned` FROM `bans` WHERE `ip` = "39.54.158.44" LIMIT 1SELECT `ip`, `name`, `reason`, `whobanned` FROM `bans` WHERE `ip` = "39.54.158.44" LIMIT 1clientDownload: 10 : file "whiskys.lair/zzz_all_rifles_v1.5.iwd" completed
SELECT `ip`, `name`, `reason`, `whobanned` FROM `bans` WHERE `ip` = "213.179.252.246" LIMIT 1SELECT `ip`, `name`, `reason`, `whobanned` FROM `bans` WHERE `ip` = "213.179.252.246" LIMIT 1Segmentation fault (core dumped)


Hitch warning: 3238 msec frame time
Resolving cod2master.activision.com
cod2master.activision.com resolved to 185.34.104.231:20710
Sending heartbeat to cod2master.activision.com
Resolving cod2master.activision.com
cod2master.activision.com resolved to 185.34.104.231:20700
Client 0 connecting with 600 challenge ping from 39.54.158.44:28960
Going from CS_FREE to CS_CONNECTED for (num 0 guid 0)
Sending heartbeat to cod2master.activision.com
SELECT `ip`, `name`, `reason`, `whobanned` FROM `bans` WHERE `ip` = "39.54.158.44" LIMIT 1SELECT `ip`, `name`, `reason`, `whobanned` FROM `bans` WHERE `ip` = "39.54.158.44" LIMIT 1Client 1 connecting with 100 challenge ping from 109.94.125.169:28960
Going from CS_FREE to CS_CONNECTED for (num 1 guid 0)
Rcon from 89.36.219.214:-24539:
say
SELECT `ip`, `name`, `reason`, `whobanned` FROM `bans` WHERE `ip` = "109.94.125.169" LIMIT 1SELECT `ip`, `name`, `reason`, `whobanned` FROM `bans` WHERE `ip` = "109.94.125.169" LIMIT 1Segmentation fault (core dumped)
Sorry for not adding new line in printf.

Got only dump from the second one:


warning: Could not load shared library symbols for ./libcod2_1_0.so.
Do you need "set solib-search-path" or "set sysroot"?
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/i386-linux-gnu/libthread_db.so.1".
Core was generated by `./cod2_lnxded +set net_port 28960 +set dedicated 2 +set fs_game whiskys.lair +e'.
Program terminated with signal 11, Segmentation fault.
#0 0xb716ab4b in mysql_options ()
from /usr/lib/i386-linux-gnu/libmysqlclient.so.18
(gdb) bt
#0 0xb716ab4b in mysql_options ()
from /usr/lib/i386-linux-gnu/libmysqlclient.so.18
#1 0xb773ccb1 in ?? ()
#2 0x00000000 in ?? ()


Any thoughts? :)

[kung foo man]

Looks like the same crash, segmentation fault 11 in some function using mysql_options(), so the code is dereferencing a NULL pointer e.g., which causes the segfault.

Probably still the easiest way to debug this:



Kinda filthy to debug a C API crash, to be able to analyze those crashes in my MySQL part I added debug statements in every C glue function, like:

https://github.com/kungfooman/libcod/blob/master/gsc_mysql.cpp#L483


#if DEBUG_MYSQL
printf("gsc_mysql_num_fields(result=%d)\n", result);
#endif


If I would want to tackle the problem, I would first give every function a debug printf, starting with the functions which call mysql_options(), since that was mentioned in the gdb backtrace.

I would printf the lines here: https://github.com/kungfooman/libcod/blob/master/gsc_mysql.cpp#L290

Googling "mysql_options seg fault" shows some problems with mysql_options() too, but that stuff seems ancient. But could still be related to your libmysqlclient version maybe.

[IzNoGoD]

./cod2_lnxded_1_0: /usr/lib/i386-linux-gnu/libmysqlclient.so.18: no version information available (required by ./libcod2_1_0.so)

seems interesting...



David, this is MySQL problem, they screwed something in their setup on Debian systems. There is lot of information on Internet about that, for example:

http://forum.directadmin.com/showthread.php?t=46807

try to follow their recommendations how to fix it.

You can just google " libmysqlclient.so.18: no version information available

[Whiskas]

Kinda filthy to debug a C API crash, to be able to analyze those crashes in my MySQL part I added debug statements in every C glue function, like:

https://github.com/kungfooman/libcod/blob/master/gsc_mysql.cpp#L483


#if DEBUG_MYSQL
printf("gsc_mysql_num_fields(result=%d)\n", result);
#endif


If I would want to tackle the problem, I would first give every function a debug printf, starting with the functions which call mysql_options(), since that was mentioned in the gdb backtrace.

I would like to adapt your idea, but I'm not sure I do this right way.



SearchForBan()
{
myIP = self getIP();
if(!isDefined(myIP))
return;
printf("SELECT `ip`, `name`, `reason`, `whobanned` FROM `bans` WHERE `ip` = \"%\" LIMIT 1\n", myIP);
maps\mp\gametypes\_mysql::asyncQuery("SELECT `ip`, `name`, `reason`, `whobanned` FROM `bans` WHERE `ip` = \"" + myIP + "\" LIMIT 1", ::checkBan);
printf("SELECT `ip`, `name`, `reason`, `whobanned` FROM `bans` WHERE `ip` = \"%\" LIMIT 1\n", myIP);
printf("gsc_mysql_num_fields(result=%d)\n", checkBan);
}





./cod2_lnxded_1_0: /usr/lib/i386-linux-gnu/libmysqlclient.so.18: no version information available (required by ./libcod2_1_0.so)

seems interesting...


David, this is MySQL problem, they screwed something in their setup on Debian systems. There is lot of information on Internet about that, for example:

http://forum.directadmin.com/showthread.php?t=46807

try to follow their recommendations how to fix it.

You can just google " libmysqlclient.so.18: no version information available


Yeah I googled it and found that AskUbuntu (http://askubuntu.com/questions/543612/usr-lib-i386-linux-gnu-libmysqlclient-so-18-no-version-information-available). I also run Ubuntu 12.04, but I have the ability to upgrade it to 14.04. Shall I do this?


Silly question:

If I run 'do-release-upgrade' to upgrade the ubuntu, I won't loose any private datas (/home/ or any other directory (/var/ is also important to me :) ) nor mysql database). ?

Sorry for being too lazy to google it. Already did.

EDIT:

Seems that the mysql problem is only related to 5.5 verison and I have (5.5.47-0ubuntu0.12.04.1) :|

[IzNoGoD]

what does apt-get dist-upgrade do for your mysql version?

[kung foo man]

Normally you should be able to just copy the working libmysqlclient.so over next to your cod2_lnxded binary and then "include" it with LD_PRELOAD_PATH=. (IIRC, did this stuff years ago)

This one is from 2009, might still do it's job great on your Ubuntu 12.04: http://killtube.org/downloads/libcod/libmysqlclient.so (or take the one from your working distribution)

OR:

Rename the default one to /usr/lib/i386-linux-gnu/libmysqlclient.so.18_backup and upload the working one from your working distribution (hacky hacky..)

The command "ldd cod2_lnxded" is pretty helpful to see which .so would be loaded.

Also try adding LD_DEBUG=libs for more debugging: http://www.bnikolic.co.uk/blog/linux-ld-debug.html


About the printf debug... I meant to add printf's in the libcod source code and recompiling it, but seems like we already know that it is the MySQL lib version, so no need for that anymore.

[Whiskas]

I'm still fighting with this problem :P.

What I've done so far:

1). Worked with Iznogod and his libcod fixes (sadly, it didn't fix the problem :|)
2). Upgraded Ubuntu to 14.04
3). Updated linux-libc-dev i386 to version 3.13.0-85.129 (Dunno if it's even related but it was outdated and I was updating everything I could)
4). Added LD_DEBUG=all to start script.
5). It seems that after the dist upgrade segfaults aren't saving core dumps. Tried ulimit -c unlimited, but it resets later (Checking it with ulimit -a). Any other way to save core dumps?
6). Before last segfault server skipped query (maybe it is related to the problem?)

Client 47 connecting with 250 challenge ping from 37.238.162.56:-31071
Going from CS_FREE to CS_CONNECTED for (num 7 guid 0)
clientDownload: 7 : begining "whiskys.lair/zzz_modz.iwd"
clientDownload: 7 : file "whiskys.lair/zzz_modz.iwd" completed
clientDownload: 7 : begining "whiskys.lair/zzz_all_rifles_v1.5.iwd"
clientDownload: 7 : file "whiskys.lair/zzz_all_rifles_v1.5.iwd" completed
SELECT `ip`, `name`, `reason`, `whobanned` FROM `bans` WHERE `ip` = "37.238.162.56" LIMIT 1
SELECT `ip`, `name`, `reason`, `whobanned` FROM `bans` WHERE `ip` = "37.238.162.56" LIMIT 1
Sending heartbeat to cod2master.activision.com
3: EXE_DISCONNECTED
^9(GAME_AXIS)I<3evil.Server 445^7: QUICKMESSAGE_MOVE_IN
Client 48 connecting with 150 challenge ping from 78.38.19.196:10039
Going from CS_FREE to CS_CONNECTED for (num 3 guid 0)
4: EXE_DISCONNECTED
Client 49 connecting with 250 challenge ping from 41.225.131.123:28960
Going from CS_FREE to CS_CONNECTED for (num 16 guid 0)
13: EXE_DISCONNECTED
SELECT `ip`, `name`, `reason`, `whobanned` FROM `bans` WHERE `ip` = "41.225.131.123" LIMIT 1
SELECT `ip`, `name`, `reason`, `whobanned` FROM `bans` WHERE `ip` = "41.225.131.123" LIMIT 1
Rcon from 89.36.219.214:-10854:
say
Segmentation fault (core dumped)


SearchForBan()
{
myIP = self getIP();
if(!isDefined(myIP))
return;
printf("SELECT `ip`, `name`, `reason`, `whobanned` FROM `bans` WHERE `ip` = \"%\" LIMIT 1\n", myIP);
maps\mp\gametypes\_mysql::asyncQuery("SELECT `ip`, `name`, `reason`, `whobanned` FROM `bans` WHERE `ip` = \"" + myIP + "\" LIMIT 1", ::checkBan);
printf("SELECT `ip`, `name`, `reason`, `whobanned` FROM `bans` WHERE `ip` = \"%\" LIMIT 1\n", myIP);
}

Seems like myIP wasn't defined, from now I've added printf before return.

Btw. maybe check if isDefined(self) before getting IP?

Attaching logfile with LD_DEBUG=all https://www.sendspace.com/file/kq3tie

Thanks again!

[Whiskas]

Let me drop the big UPDATE here:

1). I am retarded.
2). I've been backtracing the old core dump from wrong mysql initiating (which was later resolved).
3). New backtracing look like this:


#0 0x080e983d in ?? ()
(gdb) bt
#0 0x080e983d in ?? ()
#1 0x0808f697 in ?? ()
#2 0x0808fa6f in ?? ()
#3 0x080943e1 in ?? ()
#4 0x08061779 in ?? ()
#5 0x08062451 in ?? ()
#6 0x0806251d in ?? ()
#7 0x080d2b22 in ?? ()
#8 0xb738aa83 in __libc_start_main (main=0x80d2990, argc=12, argv=0xbfd22604,
init=0x8049b90, fini=0x8139b50, rtld_fini=0xb7722180 <_dl_fini>,
stack_end=0xbfd225fc) at libc-start.c:287
#9 0x0804a4d1 in ?? ()

4). Thanks to Kung's lessons I've managed to check the code behind those addresess:
Behind #1 0x0808f697 in ?? () there is:

int __cdecl sub_808F510(int a1, int a2, int a3)
{
int result; // eax@4
char v4; // [sp+4h] [bp-3D4h]@0
int v5; // [sp+14h] [bp-3C4h]@9
char *v6; // [sp+18h] [bp-3C0h]@9
char *v7; // [sp+1Ch] [bp-3BCh]@11
char v8[896]; // [sp+20h] [bp-3B8h]@11
char s; // [sp+3A0h] [bp-38h]@9
int v10; // [sp+3C4h] [bp-14h]@5
int v11; // [sp+3C8h] [bp-10h]@9
int i; // [sp+3CCh] [bp-Ch]@9
char v13[8]; // [sp+3D0h] [bp-8h]@25

if ( a3 )
*(_DWORD *)(a1 + 134408) = *(_DWORD *)(a1 + 133144);
else
*(_DWORD *)(a1 + 134408) = -1;
result = *(_DWORD *)(a1 + 133132) - *(_DWORD *)(a1 + 133136);
if ( result <= 127 )
{
v10 = sub_8067EE8(a2);
if ( v10 > 0 )
{
if ( v10 <= 32 )
{
v11 = dword_842BC8C;
v11 = *(_DWORD *)(a1 + 133144) ^ dword_842BC8C;
v11 ^= sub_80AA70C(a1 + 1032 * (*(_DWORD *)(a1 + 133136) & 0x7F) + 1036, 32);
v5 = sub_808FE02(-1653759219 * ((a1 - (signed int)dword_841FB0C) >> 2));
sub_80684E6(v5, &s);
v6 = &s;
for ( i = 0; i < v10; ++i )
{
v7 = &v8[28 * i];
sub_8068ADE(a2, v11, v6, v7);
if ( !(unsigned __int8)sub_80E97F0(v5, (unsigned __int8)v7[8]) )
v7[8] = *(_BYTE *)(v5 + 212);
if ( !(unsigned __int8)sub_80E97F0(v5, (unsigned __int8)v7[9]) )
v7[9] = *(_BYTE *)(v5 + 208);
v6 = v7;
}
*(_DWORD *)(a1 + 9924 * (*(_DWORD *)(a1 + 133144) & 0x1F) + 144352) = dword_841FB04;
if ( *(_DWORD *)a1 == 3 )
sub_808E1F0(a1, v8);
if ( !*(_BYTE *)(dword_848B200 + 8) || *(_DWORD *)(a1 + 452016) )
{
if ( *(_DWORD *)a1 == 4 )
{
for ( i = 0; ; ++i )
{
result = i;
if ( i >= v10 )
break;
if ( *(_DWORD *)&v13[28 * i - 944] <= *(_DWORD *)&v13[28 * v10 - 972]
&& *(_DWORD *)&v13[28 * i - 944] > *(_DWORD *)(a1 + 133156) )
{
sub_808F488(a1, &v8[28 * i]);
}
}
}
else
{
result = a1;
*(_DWORD *)(a1 + 134408) = -1;
}
}
else
{
result = sub_808DC8C(a1, "EXE_CANNOTVALIDATEPURECLIENT");
}
}
else
{
result = sub_8060B2C("cmdCount > MAX_PACKET_USERCMDS\n", v4);
}
}
else
{
result = sub_8060B2C("cmdCount < 1\n", v4);
}
}
return result;
}

5). Today on the server which partly uses my scripts I've noticed this:
http://i.imgur.com/HVVa5hn.jpg
It was spamming the hell out of the console.
6). thOuMta (server's admin) did not see it (only me).
7). To stop this spam I've used the cl_maxpackets "30" command. (this server had set 20 at default).
8). Kung foo man has pointed out the problem with "cmdCount > MAX_PACKET_USERCMDS".

Not sure what to do about it, Should I patch libcod with this? https://killtube.org/showthread.php?2584-Ratelimit-patch-must-apply or just parse cl_maxpackets?
Or.. maybe as it was said in this thread https://killtube.org/showthread.php?2067-Server-Crash/page3&highlight=MAX_PACKET_USERCMDS I have "wait without ending on disconnect"
Could my self SearchForBan(); be treated as a wait and I should move it somewhere?


SearchForBan()
{
if(!isDefined(self))
{
printf("\n Self wasn't defined :(");
return;
}
myIP = self getIP();
if(!isDefined(myIP))
{
printf("\n myIP wasnt defined! \n");
return;
}
printf("SELECT `ip`, `name`, `reason`, `whobanned` FROM `bans` WHERE `ip` = \"%\" LIMIT 1\n", myIP);
maps\mp\gametypes\_mysql::asyncQuery("SELECT `ip`, `name`, `reason`, `whobanned` FROM `bans` WHERE `ip` = \"" + myIP + "\" LIMIT 1", ::checkBan);
printf("SELECT `ip`, `name`, `reason`, `whobanned` FROM `bans` WHERE `ip` = \"%\" LIMIT 1\n", myIP);
}

checkBan(rows, args)
{

if(isDefined(rows) && isDefined(rows[0]) && isDefined(rows[0]["ip"]) && isDefined(self))
{
kickmsg = rows[0]["name"] + "^7, You're banned by " + rows[0]["whobanned"] + "^7, because of reason: " + rows[0]["reason"] + ". Check www.whiskaskitler.xyz to get unbanned :) " + ". You will need your IP: " + rows[0]["ip"];
kick2(self getEntityNumber(), kickmsg);
}
}

Callback_PlayerConnect()
{
thread dummy();

self.statusicon = "hud_status_connecting";
self waittill("begin");
self.statusicon = "";
level notify("connected", self);

if(!level.splitscreen)
iprintln(&"MP_CONNECTED", self);
lpselfnum = self getEntityNumber();
lpGuid = self getGuid();
logPrint("J;" + lpGuid + ";" + lpselfnum + ";" + self.name + "\n");

if(game["state"] == "intermission")
{
spawnIntermission();
return;
}
level endon("intermission");

self SearchForBan();

if(level.splitscreen)
scriptMainMenu = game["menu_ingame_spectator"];
else
scriptMainMenu = game["menu_ingame"];

if(isDefined(self.pers["team"]) && self.pers["team"] != "spectator")
{
self setClientCvar("ui_allow_weaponchange", "1");

if(self.pers["team"] == "allies")
self.sessionteam = "allies";
else
self.sessionteam = "axis";

if(isDefined(self.pers["weapon"]))
spawnPlayer();
else
{
spawnSpectator();

if(self.pers["team"] == "allies")
{
self openMenu(game["menu_weapon_allies"]);
scriptMainMenu = game["menu_weapon_allies"];
}
else
{
self openMenu(game["menu_weapon_axis"]);
scriptMainMenu = game["menu_weapon_axis"];
}
}
}
else
{
self setClientCvar("ui_allow_weaponchange", "0");

if(!isDefined(self.pers["skipserverinfo"]))
self openMenu(game["menu_team"]);

self.pers["team"] = "spectator";
self.sessionteam = "spectator";

spawnSpectator();
}

self setClientCvar("g_scriptMainMenu", scriptMainMenu);
}

Sorry for the long post. I've wanted to make it with as many details as (I thought) it needs.

Thank you.

Edit: used cl_maxpackets "20" on my server and also got spammed.

[Mitch]

The rate limit patch is not related to crashes. It is for preventing people from generating too much traffic on your server.

I never found why the server crashes when you are seeing a few cmdCount > MAX_PACKET_USERCMDS in your log.
How often does this crash happen and are you able to replicate the crash? Replicating the crash will make it easier to test and fix the issue.

I don't think cmdCount > MAX_PACKET_USERCMDS (32) is the cause of the crash.
https://github.com/id-Software/Quake-III-Arena/blob/dbe4ddb10315479fc00086f08e25d968b4b43c49/code/server/sv_client.c

#0 0x080e983d traces back to:



int __cdecl sub_80E97F0(int a1, int a2)
{
int v3; // [sp+Ch] [bp-Ch]@5
char v4; // [sp+17h] [bp-1h]@1

v4 = 1;
if ( !(unsigned __int8)sub_80E9758(a2) )
v4 = 0;
if ( !(unsigned __int8)sub_80D9E84(a1 + 1348, a2) )
v4 = 0;
v3 = sub_80E9270(a2); // weapon = getweapon(index)
if ( !*(_DWORD *)(v3 + 132) // = 0x080e983d = weapon offhandClass
&& *(_BYTE *)(a1 + 1365) != a2
&& *(_BYTE *)(a1 + 1366) != a2
&& *(_DWORD *)(v3 + 876) != a2 )
v4 = 0;
return (unsigned __int8)v4;
}




v7 = &v8[28 * i];
sub_8068ADE(a2, v11, v6, &v8[28 * i]);
if ( !(unsigned __int8)sub_80E97F0(v5, (unsigned __int8)v7[8]) ) // probably player entity, weapon index


It could that for some reason the weapon index is out of range.
One way to found out is to hook the sub_80E9270 call and print a message when the index is invalid.

[Whiskas]

How often does this crash happen and are you able to replicate the crash? Replicating the crash will make it easier to test and fix the issue.


Could be 10 crashes per day. Usually one by one, then peace for next ~14 hours. I didn't tried to replicate it :(.



#0 0x080e983d traces back to:



int __cdecl sub_80E97F0(int a1, int a2)
{
int v3; // [sp+Ch] [bp-Ch]@5
char v4; // [sp+17h] [bp-1h]@1

v4 = 1;
if ( !(unsigned __int8)sub_80E9758(a2) )
v4 = 0;
if ( !(unsigned __int8)sub_80D9E84(a1 + 1348, a2) )
v4 = 0;
v3 = sub_80E9270(a2); // weapon = getweapon(index)
if ( !*(_DWORD *)(v3 + 132) // = 0x080e983d = weapon offhandClass
&& *(_BYTE *)(a1 + 1365) != a2
&& *(_BYTE *)(a1 + 1366) != a2
&& *(_DWORD *)(v3 + 876) != a2 )
v4 = 0;
return (unsigned __int8)v4;
}




v7 = &v8[28 * i];
sub_8068ADE(a2, v11, v6, &v8[28 * i]);
if ( !(unsigned __int8)sub_80E97F0(v5, (unsigned __int8)v7[8]) ) // probably player entity, weapon index



Yeah, Kung foo man suggested to hook up the address into libcod:

#if COD_VERSION == COD2_1_0
if (0)
cracking_hook_function(0x08092D5C, (int)SV_AddServerCommand);
if (0)
cracking_hook_function(0x0809301C, (int)SV_SendServerCommand);

cracking_hook_call(0x0808F134, (int)hook_ClientUserinfoChanged);
cracking_hook_call(0x0807059F, (int)Scr_GetCustomFunction);
cracking_hook_call(0x080707C3, (int)Scr_GetCustomMethod);
cracking_hook_call(0x08098CD0, (int)custom_SV_WriteDownloadToClient);
cracking_hook_call(0x080DFF66, (int)hook_player_setmovespeed);
cracking_hook_call(0x080F50AB, (int)hook_player_g_speed);
cracking_hook_call(0x080E9524, (int)hook_findWeaponIndex);
cracking_hook_call(0x080E983D, (int)hook_findProblem); // This address

and add printf with parameters in gsc_player.cpp

int hook_findProblem(int a1, int a2) {
printf("findProblem: %d %d\n", a1, a2);
return 1;
}

Also added function to gsc_player.hpp

int hook_findProblem(int a1, int a2);

We hoped that it fill print arguments before server crash. But at the next ~10 segfaults there was no output from this address. So I've started to check out previous addresses and as I found addresses with MAX_PACKET kung suggested to check out previous threads on the forum.



It could that for some reason the weapon index is out of range.
One way to found out is to hook the sub_80E9270 call and print a message when the index is invalid.

Will do this, but yesterday I've implemented your libcod version, parsed cl_maxpackets on PlayersConnects and moved SearchForBan(); at the end of the Callback_PlayerConnect(). For now I didn't have any segfaults which is record for my server. If segmentation fault will came back I let you know with provided printf of sub_80E9270. Thanks for your time!

Whisky