Rusty,

please send the WHOLE mysqld .err log to me. Please also send also the output of

SHOW CREATE TABLE Inbound;

I am starting to suspect there might be an InnoDB memory corruption bug associated with the blob_heap of the table handle. You are the second to report a crash in the same place.

Though it is hard to see what might corrupt exactly 4, or maybe 16, bytes at the start of an InnoDB additional mem pool block. It does not look like an ordinary memory buffer overrun. Anyway, it is easy to add diagnostic code to 4.0.2x, so that we can study this in detail.

Best regards,

Heikki Tuuri Innobase Oy Foreign keys, transactions, and row level locking for MySQL InnoDB Hot Backup - a hot backup tool for InnoDB which also backs up MyISAM tables http://www.innodb.com/order.php

Order MySQL technical support from https://order.mysql.com/

0x80705d0 handle_segfault + 420 0x8297878 pthread_sighandler + 184 0x822837c mem_area_free + 548 0x82271d1 mem_heap_block_free + 385 0x81322fd row_sel_store_mysql_rec + 65 0x8137047 row_search_for_mysql + 11491 0x80cbe07 general_fetch__11ha_innobasePcUiUi + 359 0x80cbeb3 index_next__11ha_innobasePc + 35 0x809dd75 join_read_next__FP14st_read_record + 33 0x809d109 sub_select__FP4JOINP13st_join_tableb + 337 0x809cdc0 do_select__FP4JOINPt4List1Z4ItemP8st_tableP9Procedure + 412 0x8095402 mysql_select__FP3THDP13st_table_listRt4List1Z4ItemP4ItemP8st_orderT4T3 T4UlP13select_result + 6994 0x8093876 handle_select__FP3THDP6st_lexP13select_result + 102 0x807afd0 mysql_execute_command__Fv + 1004 0x807e9ba mysql_parse__FP3THDPcUi + 146 0x807a0c3 dispatch_command__F19enum_server_commandP3THDPcUi + 1435 0x8079b1d do_command__FP3THD + 157 0x8079359 handle_one_connection + 641 0x829502c pthread_start_thread + 220 0x82be82a thread_start + 4

void row_sel_store_mysql_rec( /*====================*/ byte* mysql_rec, /* out: row in the MySQL format */ row_prebuilt_t* prebuilt, /* in: prebuilt struct */ rec_t* rec) /* in: Innobase record in the index which was described in prebuilt's template */ { mysql_row_templ_t* templ; mem_heap_t* extern_field_heap = NULL; byte* data; ulint len; byte* blob_buf; ulint i;

ut_ad(prebuilt->mysql_template);

if (prebuilt->blob_heap != NULL) { mem_heap_free(prebuilt->blob_heap); <--- crash here prebuilt->blob_heap = NULL; }

.................

List:MySQL Bugs« Previous Message From:Rusty W. ShanklinDate:May 17 2004 5:11pm Subject:mysqld crash due to innoDB problems?

DESCRIPTION: Mysqld appears to crash every few days or so. The .err file shows that it may be an innodb problem. (See below for detail)

How-To-Repeat I am not able to repeat the problem. The query that runs when the mysqld server dies, is automatically re-run when the server comes back up. The query always appears to run successfully after the restart.

Fix: The automatic restart of mysqld solves the problem, however the server going down is now my problem.

Submitter-ID: RustyShanklin Organization: Pikeville Medical Center MySQL support: NONE Synopsis: mysqld crash - Possible InnoDB corruption? Severity: serious Severity: medium Category: mysql Class: sw-bug Release: mysql-4.0.18-standard (Official MySQL-standard binary) C compiler: 2.95.3 C++ compiler: 2.95.3 Environment: System: Linux mish02.pikevillehospital.org 2.4.18-14smp #1 SMP Wed Sep 4 12:34:47 EDT 2002 i686 i686 i386 GNU/Linux (4 processor system) 2GB Ram. Architecture: i686 Some paths: /usr/bin/perl /usr/bin/make /usr/bin/gmake /usr/bin/gcc /usr/bin/cc GCC: Reading specs from /usr/lib/gcc-lib/i386-redhat-linux/3.2/specs Configured with: ../configure --prefix=/usr --mandir=/usr/share/man --infodir=/u sr/share/info --enable-shared --enable-threads=posix --disable-checking --host=i 386-redhat-linux --with-system-zlib --enable-__cxa_atexit Thread model: posix gcc version 3.2 20020903 (Red Hat Linux 8.0 3.2-7) Compilation info: CC='gcc' CFLAGS='-O2 -mcpu=pentiumpro' CXX='gcc' CXXFLAGS=' -O2 -mcpu=pentiumpro -felide-constructors' LDFLAGS='' ASFLAGS='' LIBC: lrwxrwxrwx 1 root root 14 Mar 4 2003 /lib/libc.so.6 -> libc-2 .2.93.so -rwxr-xr-x 1 root root 1235468 Sep 5 2002 /lib/libc-2.2.93.so -rw-r--r-- 1 root root 2233342 Mar 10 2003 /usr/lib/libc.a -rw-r--r-- 1 root root 178 Mar 10 2003 /usr/lib/libc.so Configure command: ./configure '--prefix=/usr/local/mysql' '--localstatedir=/usr /local/mysql/data' '--libexecdir=/usr/local/mysql/bin' '--with-comment=Official MySQL-standard binary' '--with-extra-charsets=complex' '--with-server-suffix=-st andard' '--enable-thread-safe-client' '--enable-local-infile' '--enable-assemble r' '--disable-shared' '--with-client-ldflags=-all-static' '--with-mysqld-ldflags =-all-static' '--with-embedded-server' '--with-innodb' 'CFLAGS=-O2 -mcpu=pentium pro' 'CXXFLAGS=-O2 -mcpu=pentiumpro -felide-constructors' 'CXX=gcc'

----------mysql.err file: InnoDB: Error: Mem area size is 0. Possibly a memory overrun of the InnoDB: previous allocated area! InnoDB: Apparent memory corruption: mem dump len 500; hex 4e43303b5350433a4c414 23b3133343a3a3a3a3a3a3a4e3a3a22223a3a3a31333439383b5354414d5045523b44414 e49454c3 b483b3b3b4d443a3a22223a22223a3a3a3230303430353135323230343a3b20203a3a4f7 26465726 5643a3a313b22223b22223b3230303430353136303230323b22223b333b3b3b3a3a3a222 23a3a3a3 a3a3a0d4447313a39393a49393a3a3a3a574f524b494e473a3a3a3a3a3a3a3a0d5a43413 a22223a2 2223a3b3b3b0d3506001e05353738303708535441525f57454213323030342d30352d313 52032323 a30343a3133016e03413038fc09064d53483a3b7e5c263a000000000000000000000000a 90100000 0000000b010a808a0010000f63d775f9268a520773073656c2e6300a2080000000000000 00000000 00000000000000000000000900100000000000000000000900100004000000000000000f 00e41084 d53483a3b7e5c263a48424f433a413a534f46544c41423a413a323030343035313530373 33830363 a3a4f524d3b4f30313b30313a373239303430343a503a322e313a373239303430343a3a0 d5049443 a3a3a3031303030353b3b3b413a3a5455524e45523b4b4154484c45454e3a3a313933303 03631333 a463a3a3a3a3a3a3a3a3a3a303431333130333336373b3b3b413a3430302d33382d38393 1380d4f5 2433a4e413a31353630; asc NC0;SPC:LAB;134:::::::N::"":::13498;STAMPER;DANIEL;H;;; MD::"":"":::200405152204:; ::Ordered::1;"";"";200405160202;"";3;;;:::""::::::.D G1:99:I9::::WORKING::::::::.ZCA:"":"":;;;.5....57807.STAR_WEB.2004-05-15 22:04:1 3.n.A08...MSH:;~\&:.............................=w_.h. w0sel.c.................. .......................@.........A.MSH:;~\&:HBOC:A:SOFTLAB:A:20040515073 806::ORM ;O01;01:7290404:P:2.1:7290404::.PID:::010005;;;A::TURNER;KATHLEEN::19300 613:F::: :::::::0413103367;;;A:400-38-8918.ORC:NA:1560; InnoDB: Scanning backward trying to find previous allocated mem blocks Freed mem block at - 52160, file w0sel.c, line 2210 Freed mem block at - 53016, file w0sel.c, line 2210 Freed mem block at - 54584, file w0sel.c, line 2210 Freed mem block at - 117720, file w0sel.c, line 2210 Mem block at - 117912, file 0pcur.c, line 28 Mem block at - 118128, file w0upd.c, line 289 Mem block at - 118232, file m0rec.c, line 443 Freed mem block at - 124928, file w0sel.c, line 2210 Mem block at - 125216, file w0sel.c, line 2273 Mem block at - 126704, file w0sel.c, line 2210 InnoDB: Scanning forward trying to find next allocated mem blocks Freed mem block at + 16, file w0sel.c, line 2210 Mem block at + 2448, file nodb.cc, line 1920 Freed mem block at + 2736, file w0sel.c, line 2210 Mem block at + 3832, file m0rec.c, line 443 Mem block at + 3936, file w0upd.c, line 289 Mem block at + 4152, file 0pcur.c, line 28 Freed mem block at + 4344, file w0sel.c, line 2210 Mem block at + 15560, file w0sel.c, line 2210 Mem block at + 17520, file w0sel.c, line 2210 Freed mem block at + 17912, file w0sel.c, line 2210 040517 9:32:12 InnoDB: Assertion failure in thread 110620 in file mem0pool.c l ine 493 InnoDB: Failing assertion: 0 InnoDB: We intentionally generate a memory trap. InnoDB: Send a detailed bug report to mys...@lists.mysql.com mysqld got signal 11; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. key_buffer_size=268435456 read_buffer_size=1044480 max_used_connections=70 max_connections=100 threads_connected=38 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 466543 K bytes of memory Hope that's ok; if not, decrease some variables in the equation.

thd=0x50e28b08 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... Cannot determine thread, fp=0xbfc3e728, backtrace may not be correct. Stack range sanity check OK, backtrace follows: 0x80705d0 0x8297878 0x822837c 0x82271d1 0x81322fd 0x8137047 0x80cbe07 0x80cbeb3 0x809dd75 0x809d109 0x809cdc0 0x8095402 0x8093876 0x807afd0 0x807e9ba 0x807a0c3 0x8079b1d 0x8079359 0x829502c 0x82be82a New value of fp=(nil) failed sanity check, terminating stack trace! Please read http://www.mysql.com/doc/en/Using_stack_trace.html and follow instru ctions on how to resolve the stack trace. Resolved stack trace is much more helpful in diagnosing the problem, so please do resolve it Trying to get some variables. Some pointers may be invalid and cause the dump to abort... thd->query at 0x8a7b610 = select * from Inbound where status = 'n' order by id thd->thread_id=18105 thd->thread_id=18105 The manual page at http://www.mysql.com/doc/en/Crashing.html contains information that should help you find out what is causing the crash.

Number of processes running now: 0 040517 09:32:12 mysqld restarted 040517 9:32:13 InnoDB: Database was not shut down normally. InnoDB: Starting recovery from log files... InnoDB: Starting log scan based on checkpoint at InnoDB: log sequence number 1 2535462740 InnoDB: Doing recovery: scanned up to log sequence number 1 2535485165 040517 9:32:13 InnoDB: Starting an apply batch of log records to the database. .. InnoDB: Progress in percents: 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 4 9 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 InnoDB: Apply batch completed InnoDB: Last MySQL binlog file position 0 460531001, file name ./mish02-bin.046 040517 9:32:14 InnoDB: Flushing modified pages from the buffer pool... 040517 9:32:14 InnoDB: Started /usr/local/mysql-4.0.18/bin/mysqld: ready for connections. Version: '4.0.18-standard-log' socket: '/var/lib/mysql/mysql.sock' port: 3306

Rusty W. Shanklin, Director-IT Pikeville Medical Center (606) 218 - 4940