Unidentified Server's Crash (Shutdown)

**Whiskas** · 10th March 2016, 22:12

Hey, I'm hosting a server with some custom mods. I had many problems with server's crash. There were always an error saying why does the server crashed.
Now I'm struggling with server's "shutdown", but there is no any error which could help me specify the problem.

Here are the latest lines of the console_mp_server.log file (from the 3 different crashes (more like "shutdowns", because whole procces was killed o.O))

1). Today's first crash:

Code:

cmd  1703: 33027950: G 173
cmd  1704: 33029300:  "MP_JOINED_ALLIES^^0Scorpion^5@^^0Your ^^0Ego^7"
cmd  1705: 33030000: G 174
cmd  1706: 33031300: G 175
cmd  1707: 33032500: v ui_bolt_fraggrenades1 "0"
cmd  1708: 33032500: v ui_bolt_fraggrenades2 "0"
cmd  1709: 33032500: v ui_bolt_fraggrenades3 "0"
cmd  1710: 33032500: v ui_semi_fraggrenades1 "0"
cmd  1711: 33032500: v ui_semi_fraggrenades2 "0"
cmd  1712: 33032500: v ui_semi_fraggrenades3 "0"
cmd  1713: 33034150: G 176
cmd  1714: 33034350: G 177
cmd  1715: 33036000: G 178
cmd  1716: 33037300: H 160
cmd  1717: 33037500: v ui_bolt_fraggrenades1 "0"
cmd  1718: 33037500: v ui_bolt_fraggrenades2 "0"
cmd  1719: 33037500: v ui_bolt_fraggrenades3 "0"
cmd  1720: 33037500: v ui_semi_fraggrenades1 "0"
cmd  1721: 33037500: v ui_semi_fraggrenades2 "0"
cmd  1722: 33037500: v ui_semi_fraggrenades3 "0"
cmd  1723: 33038700: G 179
cmd  1724: 33039000: v developer "0"
cmd  1725: 33039000: v r_xdebug "0"
cmd  1726: 33039000: I "1"
cmd  1727: 33039000: I "1"
cmd  1728: 33039000: v cg_objectiveText "MP_GAIN_POINTS_BY_ELIMINATING1350"
cmd  1729: 33039250: H 161
cmd  1730: 33041300: H 162
cmd  1731: 33042500: v ui_bolt_fraggrenades1 "0"
cmd  1732: 33042500: v ui_bolt_fraggrenades2 "0"
cmd  1733: 33042500: v ui_bolt_fraggrenades3 "0"
cmd  1734: 33042500: v ui_semi_fraggrenades1 "0"
cmd  1735: 33042500: v ui_semi_fraggrenades2 "0"
cmd  1736: 33042500: v ui_semi_fraggrenades3 "0"
cmd  1737: 33047500: v ui_bolt_fraggrenades1 "0"
cmd  1738: 33047500: v ui_bolt_fraggrenades2 "0"
cmd  1739: 33047500: v ui_bolt_fraggrenades3 "0"
cmd  1740: 33047500: v ui_semi_fraggrenades1 "0"
cmd  1741: 33047500: v ui_semi_fraggrenades2 "0"
cmd  1742: 33047500: v ui_semi_fraggrenades3 "0"
cmd  1743: 33052500: v ui_bolt_fraggrenades1 "0"
cmd  1744: 33052500: v ui_bolt_fraggrenades2 "0"
cmd  1745: 33052500: v ui_bolt_fraggrenades3 "0"
cmd  1746: 33052500: v ui_semi_fraggrenades1 "0"
cmd  1747: 33052500: v ui_semi_fraggrenades2 "0"
cmd  1748: 33052500: v ui_semi_fraggrenades3 "0"
cmd  1749: 33057500: v ui_bolt_fraggrenades1 "0"
cmd  1750: 33057500: v ui_bolt_fraggrenades2 "0"
cmd  1751: 33057500: v ui_bolt_fraggrenades3 "0"
cmd  1752: 33057500: v ui_semi_fraggrenades1 "0"
cmd  1753: 33057500: v ui_semi_fraggrenades2 "0"
cmd  1754: 33057500: v ui_semi_fraggrenades3 "0"
cmd  1755: 33062500: v ui_bolt_fraggrenades1 "0"
cmd  1756: 33062500: v ui_bolt_fraggrenades2 "0"
cmd  1757: 33062500: v ui_bolt_fraggrenades3 "0"
cmd  1758: 33062500: v ui_semi_fraggrenades1 "0"
cmd  1759: 33062500: v ui_semi_fraggrenades2 "0"
cmd  1760: 33062500: v ui_semi_fraggrenades3 "0"
cmd  1761: 33067500: v ui_bolt_fraggrenades1 "0"
cmd  1762: 33067500: v ui_bolt_fraggrenades2 "0"
cmd  1763: 33067500: v ui_bolt_fraggrenades3 "0"
cmd  1764: 33067500: v ui_semi_fraggrenades1 "0"
cmd  1765: 33067500: v ui_semi_fraggrenades2 "0"
cmd  1766: 33067500: v ui_semi_fraggrenades3 "0"
cmd  1767: 33072500: v ui_bolt_fraggrenades1 "0"
cmd  1768: 33072500: v ui_bolt_fraggrenades2 "0"
cmd  1769: 33072500: v ui_bolt_fraggrenades3 "0"
cmd  1770: 33072500: v ui_semi_fraggrenades1 "0"
cmd  1771: 33072500: v ui_semi_fraggrenades2 "0"
cmd  1772: 33072500: v ui_semi_fraggrenades3 "0"
cmd  1773: 33077500: v ui_bolt_fraggrenades1 "0"
cmd  1774: 33077500: v ui_bolt_fraggrenades2 "0"
cmd  1775: 33077500: v ui_bolt_fraggrenades3 "0"
cmd  1776: 33077500: v ui_semi_fraggrenades1 "0"
cmd  1777: 33077500: v ui_semi_fraggrenades2 "0"
cmd  1778: 33077500: v ui_semi_fraggrenades3 "0"
cmd  1779: 33082500: v ui_bolt_fraggrenades1 "0"
cmd  1780: 33082500: v ui_bolt_fraggrenades2 "0"
cmd  1781: 33082500: v ui_bolt_fraggrenades3 "0"
cmd  1782: 33082500: v ui_semi_fraggrenades1 "0"
cmd  1783: 33082500: v ui_semi_fraggrenades2 "0"
cmd  1784: 33082500: v ui_semi_fraggrenades3 "0"
cmd  1785: 33087500: v ui_bolt_fraggrenades1 "0"
cmd  1786: 33087500: v ui_bolt_fraggrenades2 "0"
cmd  1787: 33087500: v ui_bolt_fraggrenades3 "0"
cmd  1788: 33087500: v ui_semi_fraggrenades1 "0"
cmd  1789: 33087500: v ui_semi_fraggrenades2 "0"
cmd  1790: 33087500: v ui_semi_fraggrenades3 "0"
cmd  1791: 33092500: v ui_bolt_fraggrenades1 "0"
cmd  1792: 33092500: v ui_bolt_fraggrenades2 "0"
cmd  1793: 33092500: v ui_bolt_fraggrenades3 "0"
cmd  1794: 33092500: v ui_semi_fraggrenades1 "0"
cmd  1795: 33092500: v ui_semi_fraggrenades2 "0"
cmd  1796: 33092500: v ui_semi_fraggrenades3 "0"
cmd  1797: 33093100: v developer "0"
cmd  1798: 33011950: H 157
cmd  1799: 33093100: v r_xdebug "0"
Unknown Soldier: Delta request from out of date packet.
Going to CS_ZOMBIE for 
10: EXE_SERVERCOMMANDOVERFLOW
Going from CS_ZOMBIE to CS_FREE for 
Going from CS_PRIMED to CS_ACTIVE for ^7I^1<3^9evil^1.^7Server 338

2). Second:

Code:

Client 55 connecting with 50 challenge ping from //HERE WAS SOME IP:28960
Going from CS_FREE to CS_CONNECTED for  (num 6 guid 0)
clientDownload: 6 : begining "whiskys.lair/zzz_all_rifles_v1.5.iwd"
clientDownload: 6 : file "whiskys.lair/zzz_all_rifles_v1.5.iwd" completed
^9(GAME_AXIS)   ^7: QUICKMESSAGE_NO_SIR
^8^3?^7: yes
6: EXE_DISCONNECTED
^9(GAME_AXIS)   ^7: QUICKMESSAGE_NO_SIR
^9(GAME_AXIS)   ^7: QUICKMESSAGE_NO_SIR

3). Third:

Code:

Client 98 connecting with 100 challenge ping from //HERE WAS SOME IP:28960
Going from CS_FREE to CS_CONNECTED for  (num 1 guid 0)
Client 99 connecting with 100 challenge ping from //HERE WAS SOME IP:28960
Going from CS_FREE to CS_CONNECTED for  (num 5 guid 0)
clientDownload: 1 : begining "whiskys.lair/zzz_all_rifles_v1.5.iwd"
clientDownload: 1 : file "whiskys.lair/zzz_all_rifles_v1.5.iwd" completed
Rcon from //HERE WAS MY SERVERS IP:-25423:
say
^9(GAME_AXIS)^4Balbazar^7: QUICKMESSAGE_SORRY
^9(GAME_AXIS)^4Balbazar^7: QUICKMESSAGE_SORRY
^8(GAME_ALLIES)gAy-bIrD...^7: QUICKMESSAGE_SORRY
^8gAy-bIrD...^7: ownd
^8(GAME_ALLIES)gAy-bIrD...^7: QUICKMESSAGE_SORRY
^8(GAME_ALLIES)gAy-bIrD...^7: QUICKMESSAGE_GREAT_SHOT
Rcon from //HERE WAS MY SERVERS IP:-17623:
say
^9(GAME_AXIS)^4Balbazar^7: QUICKMESSAGE_SORRY
^9(GAME_AXIS)^4Balbazar^7: QUICKMESSAGE_SORRY
4: EXE_DISCONNECTED
Sending heartbeat to cod2master.activision.com
Client 100 connecting with 150 challenge ping from //HERE WAS SOME IP
Going from CS_FREE to CS_CONNECTED for  (num 4 guid 0)
Rcon from //HERE WAS SOME IP:-11712:
say
^8(GAME_ALLIES)gAy-bIrD...^7: QUICKMESSAGE_SORRY
^8(GAME_ALLIES)gAy-bIrD...^7: QUICKMESSAGE_SORRY
Client 101 connecting with 50 challenge ping from //HERE WAS SOME IP
Going from CS_FREE to CS_CONNECTED for  (num 7 guid 0)
^9(GAME_AXIS)^4Balbazar^7: QUICKMESSAGE_SORRY

Also I have to say that I'm using TMUX to run server in the background and usually when server has crash (cause of some script error), the tmux process will stay alive. But in this case, whole (TMUX's) process(or session?) is killed.

I will provide any suspicious script.

Thank you for your help (or any tip) in advance.

**kung foo man** · 11th March 2016, 04:05

1) Does it happen with screen too?

2) How many servers are you running on that box? When it's the only CoD2 server on a VPS box, maybe it was a forced restart from host, which would explain the killed tmux session.

Check last reboot:

Code:

root@euve78301:~#  last reboot

wtmp begins Sat Mar  5 02:12:23 2016

3) Otherwise probably google "tmux log crash" ^^

**voron00** · 11th March 2016, 04:15

I just dropped your server by using one of the 'well known nukes' (the good old va() bug), you need to use a patched binary from here: http://killtube.org/showthread.php?2...ll=1#post11345

Edit: That proably not the actual thing with the crashes but still, some people are abusing this stuff.

**Whiskas** · 11th March 2016, 08:49

Originally Posted by kung foo man

1) Does it happen with screen too?

I will test it now.

Originally Posted by kung foo man

2) How many servers are you running on that box? When it's the only CoD2 server on a VPS box, maybe it was a forced restart from host, which would explain the killed tmux session.

Two servers, and it's always only one of them that is "shutting down". Also I have auto start with reboot.

Originally Posted by kung foo man

Check last reboot:

Code:

root@euve78301:~#  last reboot

wtmp begins Sat Mar  5 02:12:23 2016

who -b
system boot 2016-03-11 04:00

But it's my "forced" reboot, as I set it in the CronTab. 1st server is dead, thanks to Voron :P, but the second still works.

Originally Posted by kung foo man

3) Otherwise probably google "tmux log crash" ^^

I'll try this https://github.com/tmux-plugins/tmux-logging . Cause I'm can't find any log.
First server will be turned on with screen and I let second live on TMUX, but with logging. Let's see what happens. Thanks!

Originally Posted by voron00

I just dropped your server by using one of the 'well known nukes' (the good old va() bug), you need to use a patched binary from here: http://killtube.org/showthread.php?2...ll=1#post11345

Edit: That proably not the actual thing with the crashes but still, some people are abusing this stuff.

Will use this from now, thanks. I think there was one time I've seen in some crash logs this NAKAttempt.

P.S I think (not sure) I witnessed this "shutdown". Server seem to work but when I disconnected I couldn't reconnect back there, and the server wasn't listed on servers list.

Console log:

Code:

cmd  1480:  1090200: v ui_bolt_fraggrenades1 "0"
cmd  1481:  1090200: v ui_bolt_fraggrenades2 "0"
cmd  1482:  1090200: v ui_bolt_fraggrenades3 "0"
cmd  1483:  1090200: v ui_semi_fraggrenades1 "0"
cmd  1484:   992000: G 29
cmd  1485:  1090200: v ui_semi_fraggrenades2 "0"
1: EXE_SERVERCOMMANDOVERFLOW
Sending heartbeat to cod2master.activision.com
0: EXE_DISCONNECTED
Sending heartbeat to cod2master.activision.com

And the output from screen's window:

Code:

File Handles:
handle 1: console_mp_server.log
----------------------
42143 files in iwd files
------- Game Initialization -------
gamename: Call of Duty 2
gamedate: Oct 24 2005
-----------------------------------
scriptengine> Async mysql already initialized. Returning before adding additional connections
-----------------------------------
Hitch warning: 1974 msec frame time
Sending heartbeat to cod2master.activision.com

**Whiskas** · 11th March 2016, 10:03

Update:

These shutdown were when I wasn't back home. I sometimes check GamesTracker website to check if my server is alive. If it's DEAD I also check console log and type this command:

PHP Code:


echo -e '\xff\xff\xff\xffgetstatus' | netcat -u 89.36.219.214 28960

If it's gave me no output at all, I suggested that server was dead and I remotely restart it.

Today I had some time to play. I was playing for about 30 mins, checked gamestracker and it says that server is down and that the last scan was 10 mins ago. I was quite suprised so typed above command and yeah, it gave me nothing despite I was playing server with other players and more has came. Server was listed in call of duty's server list and disconnecting/connecting went without any problems.

Who knows, maybe there were no shutdowns at all and I was remotely restarting the server because of getstatus null response :P

Is there any known getstatus bug?

Edit:
This http://killtube.org/showthread.php?2...ll=1#post11571 might be related if I use something like this? :

numberofplayers.sh

PHP Code:


#!/bin/bash



/home/codserver2/testscript

/home/codserver/testscript

testscript

PHP Code:


#!/bin/bash



#Execute starttest (tmux script) which uses echo getstatus of 'spected' server

#It is executed as linux user "codserver"

su --login codserver2 --command "/home/codserver/starttest"



#starttest has an infinite loop

#Added short time sleep to make sure starttest made whole dump

sleep 2



#Kill this infinite loop for fuck sake!

su --login codserver2 --command "/home/codserver/stoptest"



#Our starttest saved his output to an 'output' file

#Here we are couting lines in output file and saving as variable

players="$(wc -l /home/codserver2/output | awk '{print $1}')"



#Executing php file with our variable as a parameter

php /home/codserver2/players_num.php $players



#Delete this temporary file

#rm /home/codserver2/output

starttest

PHP Code:


tmux new-session -d -s codcheck2 "echo -e '\xff\xff\xff\xffgetstatus' | netcat -u 89.36.219.214 28960 > /home/codserver2/output"

stoptest

PHP Code:


tmux kill-session -t codcheck2

players_num.php

PHP Code:


<?php

    $number = $argv[1];

    $server_ip = '89.36.219.214';

    $server_port = '28961';

    $server_timeout = 10;

    $number2 = (int)$number;

    $number2 = $number2 - 2;







    $server_addr = "udp://" . $server_ip;

    @$connect = fsockopen($server_addr, $server_port, $re, $errstr, $server_timeout);

    if (! $connect) 

        { 

            echo 'Not working 1'; 

        }

    socket_set_timeout ($connect, $server_timeout);



    $server_rconpass = 'HEREISMYRCONPASS';

    $server_extra_wait = false;        

    $server_extra_footer = true;    

    $server_buffer_results = 2048;    // enter a number of bytes; decrease if you receive only a part of returned results, increase to speed up

    $server_buffer_cur = $server_buffer_results;

    $cmd = "say Server Status: ^9crack_hj_tdm^6!^3.. ^7" . $number2 . "/20 players.";



    //global $server_rconpass, $connect, $server_extra_wait, $server_buffer_cur, $server_extra_footer;

    $send = "\xff\xff\xff\xff" . 'rcon "' . $server_rconpass . '" '.$cmd.(($server_extra_footer)?"\n":'');

    fwrite($connect, $send);    



    $output = (($server_extra_wait)?(fread ($connect, 1)):'');

        do {

        $status_pre = socket_get_status ($connect);

        if ((($server_extra_wait) && ($output != '')) || (! $server_extra_wait))

            $output .= fread ($connect, $server_buffer_cur);

        $status_post = socket_get_status ($connect);

        } while ($status_pre['unread_bytes'] != $status_post['unread_bytes']);



    return $output;

    fclose($connect);

?>

It's added in crobtab and spams nearly every 3 mins. Maybe GameTracker's getstatus is sometimes executed in same time as my numberofplayers.sh script?

**Whiskas** · 12th March 2016, 08:23

Sorry for triple posting, but I wanted to say that I let live my server that had "getstatuts?" bug. But screen's session died later anyway (same as tmux was dying).
I've turned off any bash/php script related to my server and we will see if anything happends.

**IzNoGoD** · 12th March 2016, 10:40

next time your getstatus doesnt return anything: check net_port on your server. Sometimes the port is "in use" on mapchange, causing cod to go to port+1 and itll keep running from there

**maxdamage99** · 16th March 2016, 11:27

My server crash error:

Code:

WARNING: bad command byte 7 for client 8
WARNING: bad command byte 7 for client 8
WARNING: bad command byte 7 for client 8

help me pls

**Mitch** · 16th March 2016, 15:35

Originally Posted by maxdamage99

My server crash error:

Code:

WARNING: bad command byte 7 for client 8
WARNING: bad command byte 7 for client 8
WARNING: bad command byte 7 for client 8

help me pls

Maybe start your own thread? Because your problem doesn't seem related to the crashes in this thread.
Also which cod2 version? Libcod? Which libcod version/build? Is there anything else that you think we should know?

**maxdamage99** · 16th March 2016, 15:48

libcod2_1_0, cod2_lnxded 1.0
(libcod last version, without grenade_damage)

Thread: Unidentified Server's Crash (Shutdown)

Thread Tools

Search Thread

Display

Unidentified Server's Crash (Shutdown)

The Following User Says Thank You to IzNoGoD For This Useful Post:

Posting Permissions