For the past few months we have had an error like the one below appear in our ASE log. The error initially occurred once a week or so but in the last few weeks it has started happening almost daily.
00:03:00000:00078:2015/06/15 08:27:18.38 kernel Address 0x0x0000000000b7c67b (fm_unlink_frame+0x2b), siginfo (code, address) = (1, 0x0x00000000deadbabe)
00:03:00000:00078:2015/06/15 08:27:18.39 kernel ************************************
00:03:00000:00078:2015/06/15 08:27:18.39 kernel SQL causing error :
00:03:00000:00078:2015/06/15 08:27:18.39 kernel ************************************
00:03:00000:00078:2015/06/15 08:27:18.39 server SQL Text:
00:03:00000:00078:2015/06/15 08:27:18.39 kernel curdb = 1 tempdb = 0 pstat = 0x10000 p2stat = 0x141000
00:03:00000:00078:2015/06/15 08:27:18.39 kernel p3stat = 0x900 p4stat = 0x0 p5stat = 0x8 p6stat = 0x10 p7stat = 0x10000
00:03:00000:00078:2015/06/15 08:27:18.39 kernel lasterror = 0 preverror = 0 transtate = 1
00:03:00000:00078:2015/06/15 08:27:18.39 kernel curcmd = 0 program = PHP 5
00:03:00000:00078:2015/06/15 08:27:18.39 kernel extended error information: hostname: xxx.xxx.xxx login: lGbReportXXXXXXXX
00:03:00000:00078:2015/06/15 08:27:18.40 kernel pc: 0x000000000176740f pcstkwalk+0x30()
00:03:00000:00078:2015/06/15 08:27:18.40 kernel pc: 0x0000000001767205 ucstkgentrace+0x1a5()
00:03:00000:00078:2015/06/15 08:27:18.40 kernel pc: 0x0000000001706771 ucbacktrace+0x43()
00:03:00000:00078:2015/06/15 08:27:18.40 kernel pc: 0x000000000074360e terminate_process+0xd5e()
00:03:00000:00078:2015/06/15 08:27:18.40 kernel pc: 0x0000000001733d03 kisignal+0x3b3()
00:03:00000:00078:2015/06/15 08:27:18.40 kernel pc: 0x0000000000b7c67b fm_unlink_frame+0x2b()
00:03:00000:00078:2015/06/15 08:27:18.40 kernel pc: 0x0000000000b7c346 fm_free_frame+0x16()
00:03:00000:00078:2015/06/15 08:27:18.40 kernel pc: 0x0000000000b7c3c8 fm_free_list+0x18()
00:03:00000:00078:2015/06/15 08:27:18.40 kernel [Handler pc: 0x0x000000000153acde hdl_backout_msg installed by the following function:-]
00:03:00000:00078:2015/06/15 08:27:18.40 kernel pc: 0x0000000000b68d56 db__create_sysusages_map+0x126()
00:03:00000:00078:2015/06/15 08:27:18.40 kernel pc: 0x0000000000b66cf9 db_fill+0x159()
00:03:00000:00078:2015/06/15 08:27:18.40 kernel [Handler pc: 0x0x000000000153acde hdl_backout_msg installed by the following function:-]
00:03:00000:00078:2015/06/15 08:27:18.40 kernel pc: 0x0000000000b6bdfb dbt(float, int, long, long)+0x16b()
00:03:00000:00078:2015/06/15 08:27:18.40 kernel pc: 0x00000000019b0a70 dbt_get+0x440()
00:03:00000:00078:2015/06/15 08:27:18.40 kernel pc: 0x0000000000b6dd4f dbt_get_with_statuscheck+0xf()
00:03:00000:00078:2015/06/15 08:27:18.40 kernel pc: 0x0000000000b62f4a usedb_cmd+0x7a()
00:03:00000:00078:2015/06/15 08:27:18.40 kernel [Handler pc: 0x0x00000000010b644f hdl__login installed by the following function:-]
00:03:00000:00078:2015/06/15 08:27:18.40 kernel pc: 0x000000000074c7d4 login+0x734()
00:03:00000:00078:2015/06/15 08:27:18.40 kernel [Handler pc: 0x0x0000000000d4e930 ut_handle installed by the following function:-]
00:03:00000:00078:2015/06/15 08:27:18.40 kernel pc: 0x000000000191c52f conn_hdlr+0x51f()
00:03:00000:00078:2015/06/15 08:27:18.40 kernel end of stack trace, spid 78, kpid 951451747, suid 1621
After this error occurs, MAINTENANCE TOKEN processes start appearing on the server. More and more of them start, never finish, and they cannot be killed. Dataserver performance eventually degrades - the only solution is to perform a shutdown with nowait and restart the dataserver.
The error is usually like the above, but occasionally has errors like this related to memory:
00:03:00000:00127:2015/06/11 20:43:18.95 server Error: 715, Severity: 20, State: 2
00:03:00000:00127:2015/06/11 20:43:18.95 server Unable to locate a procedure cache memory control descriptor for memory '0x0x176731800' in the global list.
00:03:00000:00127:2015/06/11 20:43:18.95 server Error: 715, Severity: 20, State: 2
00:03:00000:00127:2015/06/11 20:43:18.95 server Unable to locate a procedure cache memory control descriptor for memory '0x0x16ec53000' in the global list.
00:01:00000:00127:2015/06/11 20:43:19.13 server Error: 715, Severity: 20, State: 2
00:01:00000:00127:2015/06/11 20:43:19.13 server Unable to locate a procedure cache memory control descriptor for memory '0x0x1a10ec800' in the global list.
00:01:00000:00127:2015/06/11 20:43:19.13 server Error: 715, Severity: 20, State: 2
00:01:00000:00127:2015/06/11 20:43:19.13 server Unable to locate a procedure cache memory control descriptor for memory '0x0x1b12c7000' in the global list.
00:01:00000:00127:2015/06/11 20:43:19.13 server Error: 715, Severity: 20, State: 2
00:01:00000:00127:2015/06/11 20:43:19.13 server Unable to locate a procedure cache memory control descriptor for memory '0x0x177752000' in the global list.
00:01:00000:00127:2015/06/11 20:43:19.13 server Error: 715, Severity: 20, State: 2
00:01:00000:00127:2015/06/11 20:43:19.13 server Unable to locate a procedure cache memory control descriptor for memory '0x0x18fe13000' in the global list.
00:01:00000:00127:2015/06/11 20:43:19.13 server Error: 715, Severity: 20, State: 2
00:01:00000:00127:2015/06/11 20:43:19.13 server Unable to locate a procedure cache memory control descriptor for memory '0x0x16bfe9000' in the global list.
00:01:00000:00127:2015/06/11 20:43:19.13 server Error: 714, Severity: 20, State: 2
00:01:00000:00127:2015/06/11 20:43:19.13 server Unable to locate a procedure cache memory control descriptor for memory '0x0x1b5c22800' allocated by process 127.
00:01:00000:00127:2015/06/11 20:43:19.13 server Error: 715, Severity: 20, State: 2
00:01:00000:00127:2015/06/11 20:43:19.13 server Unable to locate a procedure cache memory control descriptor for memory '0x0x1b4a92000' in the global list.
00:01:00000:00127:2015/06/11 20:43:19.13 server Error: 715, Severity: 20, State: 2
00:01:00000:00127:2015/06/11 20:43:19.13 server Unable to locate a procedure cache memory control descriptor for memory '0x0x1b64bd800' in the global list.
00:01:00000:00127:2015/06/11 20:43:19.13 server Error: 715, Severity: 20, State: 2
00:01:00000:00127:2015/06/11 20:43:19.13 server Unable to locate a procedure cache memory control descriptor for memory '0x0x1b64df000' in the global list.
00:01:00000:00127:2015/06/11 20:43:19.13 server Error: 714, Severity: 20, State: 2
00:01:00000:00127:2015/06/11 20:43:19.13 server Unable to locate a procedure cache memory control descriptor for memory '0x0x1b63bb560' allocated by process 127.
00:01:00000:00127:2015/06/11 20:43:19.13 server (mem_freepages) Diagnostics for error 707 state 8: addr=0x0x1b63bb560 mem free failed retcode=-3 pgcount=0
00:01:00000:00127:2015/06/11 20:43:19.13 kernel pc: 0x000000000176740f pcstkwalk+0x30()
00:01:00000:00127:2015/06/11 20:43:19.13 kernel pc: 0x0000000001767205 ucstkgentrace+0x1a5()
00:01:00000:00127:2015/06/11 20:43:19.13 kernel pc: 0x0000000001706771 ucbacktrace+0x43()
00:01:00000:00127:2015/06/11 20:43:19.13 kernel pc: 0x00000000019c8715 mem_freepages+0x205()
00:01:00000:00127:2015/06/11 20:43:19.13 kernel pc: 0x0000000000b868a8 free_memchunk+0x98()
00:01:00000:00127:2015/06/11 20:43:19.13 kernel pc: 0x0000000000b7c390 fm_free_frame+0x60()
00:01:00000:00127:2015/06/11 20:43:19.13 kernel pc: 0x0000000000b7c3c8 fm_free_list+0x18()
00:01:00000:00127:2015/06/11 20:43:19.13 kernel [Handler pc: 0x0x000000000153acde hdl_backout_msg installed by the following function:-]
00:01:00000:00127:2015/06/11 20:43:19.13 kernel pc: 0x0000000000b68d56 db__create_sysusages_map+0x126()
00:01:00000:00127:2015/06/11 20:43:19.13 kernel pc: 0x0000000000b66cf9 db_fill+0x159()
00:01:00000:00127:2015/06/11 20:43:19.13 kernel [Handler pc: 0x0x000000000153acde hdl_backout_msg installed by the following function:-]
00:01:00000:00127:2015/06/11 20:43:19.13 kernel pc: 0x0000000000b6bdfb dbt(float, int, long, long)+0x16b()
00:01:00000:00127:2015/06/11 20:43:19.13 kernel pc: 0x00000000019b0a70 dbt_get+0x440()
00:01:00000:00127:2015/06/11 20:43:19.13 kernel pc: 0x0000000000b6dd4f dbt_get_with_statuscheck+0xf()
00:01:00000:00127:2015/06/11 20:43:19.13 kernel pc: 0x0000000000b62f4a usedb_cmd+0x7a()
00:01:00000:00127:2015/06/11 20:43:19.13 kernel [Handler pc: 0x0x00000000010b644f hdl__login installed by the following function:-]
00:01:00000:00127:2015/06/11 20:43:19.13 kernel pc: 0x000000000074c7d4 login+0x734()
00:01:00000:00127:2015/06/11 20:43:19.13 kernel [Handler pc: 0x0x0000000000d4e930 ut_handle installed by the following function:-]
00:01:00000:00127:2015/06/11 20:43:19.13 kernel pc: 0x000000000191c52f conn_hdlr+0x51f()
00:01:00000:00127:2015/06/11 20:43:19.13 kernel end of stack trace, spid 127, kpid 1491992657, suid 1853
00:01:00000:00127:2015/06/11 20:43:19.13 server Error: 707, Severity: 20, State: 8
00:01:00000:00127:2015/06/11 20:43:19.13 server System error detected during attempt to free memory at address 0x1b63bb560. Please consult the ASE error log for more details.
00:01:00000:00127:2015/06/11 20:43:20.03 server WARNING: memory usage in procedure headers (2)
does not match memory usage count in Pss (4294965921) for server process id 127.
00:01:00000:00127:2015/06/11 20:43:20.33 server Error: 715, Severity: 20, State: 2
We have tried tweaking the server settings in this post SAP Sybase Forums - ASE - Administration - Maintenance Token and high CPU but it has not helped. We have a support license for one of our machines, unfortunately it it not for this machine. Our other servers perform similar work and have similar hardware specifications but this server is the only one with a problem. The login lGbReportXXXXXXXX is one login of many and it changes each time - the issue does not appear to be related to a single database or login.
The server version is:
Adaptive Server Enterprise/15.5/EBF 18661 SMP ESD#4/P/x86_64/Enterprise Linux/a
sear155/2545/64-bit/FBO/Thu Jun 16 06:45:54 2011
Any idea what is causing these errors and how we can fix the issue?
Any help appreciated. Cheers