atom feed8 messages in org.freebsd.freebsd-pfRe: Panic in packet filter
FromSent OnAttachments
Ali MdidechFeb 23, 2012 12:43 am 
Ermal LuçiFeb 23, 2012 11:20 pm 
Ali MdidechFeb 24, 2012 1:10 am 
Alexander VyrlanovichFeb 24, 2012 6:29 am 
Theodor-Iulian CiobanuApr 12, 2012 4:16 am 
Ermal LuçiApr 12, 2012 6:01 am 
Theodor-Iulian CiobanuApr 12, 2012 3:29 pm 
Ermal LuçiApr 13, 2012 1:36 am 
Subject:Re: Panic in packet filter
From:Ermal Luçi (er@freebsd.org)
Date:Apr 12, 2012 6:01:23 am
List:org.freebsd.freebsd-pf

Hello,

On Thu, Apr 12, 2012 at 1:16 PM, Theodor-Iulian Ciobanu <thci@nth.ro> wrote:

Hello,

I came across this same issue yesterday on a system I have just set up. I'm currently using the default kernel:

FreeBSD changeme 9.0-RELEASE FreeBSD 9.0-RELEASE #0: Tue Jan  3 07:46:30 UTC 2012 ro@farrell.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC amd64

with pf obviously loaded as a module. Even with kern.smp.disabled=1 pf will crash as soon as it matches a rule that contains tables with counters (I added such a table with just three addresses).

I'll have this machine around for testing for about a week or so and am willing to try out any available patches to help fix the issue.

Try this patch http://people.freebsd.org/~eri/pf_table_counter_fix.diff. It should fix the issue for you.

Seems there is a forgotten pool initialization for this, my fault!

Though looking at it the whole thing seems a microoptimization that is still present on latest OpenBSD code, that saves about 16bytes!

Anyway see if it fixes the issue to get this committed.

On Fri Feb 24 14:47:53 2012 iskander at apple-park.kiev.ua (Alexander Vyrlanovich) wrote:

On 24 Feb 2012, at 11:10, Ali Mdidech wrote:

Hi Ermal,

2012/2/24 Ermal Lu?i <er@freebsd.org>:

On Thu, Feb 23, 2012 at 8:44 AM, Ali Mdidech <al@moua7.com> wrote:

Hi List,

I've a box that panics multiple times randomly since a year whatever the release is (8 or 9) The crash dump shows that the problem is related to pf. Is this some sort of identified bug? Below some info and my pf.conf file.

Thank you very much for your help.

Can you try do disable SMP through sysctl and see if you still get this? What are you doing to get the panic?

Well, I'm able now to avoid or reproduce the panic. Disabling counters in <ssh_brute> table makes the server stable enough and no panic for 48 hours. Restoring the counters and adding a host in the table by hand (pfctl -t ssh_brute -T add someip) provokes the panic within few seconds. I've disabled smp (adding kern.smp.disabled=1 in loader.conf and rebooting) => kernel still panics.

FreeBSD somehost 9.0-RELEASE FreeBSD 9.0-RELEASE #1: Sat Jan 21 09:31:30 CET 2012     root@somehost:/usr/obj/usr/src/sys/DDX3KRNL i386

I can confirm that problem with counters in pf tables persist at last on i386 and amd64. My systems is:

FreeBSD gw 9.0-RELEASE FreeBSD 9.0-RELEASE #1: Tue Jan  3 15:55:41 EET 2012 root@gw:/usr/obj/usr/src/sys/GW3  amd64

FreeBSD gw2 9.0-RELEASE FreeBSD 9.0-RELEASE #0: Wed Jan 25 13:52:48 EET 2012 root@gw2:/usr/obj/usr/src/sys/GWS90  i386

pf + altq compiled in kernel

Same result: kernel panic. Without counters systems is rock solid.

Also its very helpful to know the `uname -a` command output.

panic: page fault

GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB.  Type "show warranty" for details. This GDB was configured as "i386-marcel-freebsd"...

Unread portion of the kernel message buffer:

Fatal trap 12: page fault while in kernel mode cpuid = 0; apic id = 00 fault virtual address   = 0x6c fault code              = supervisor read, page not present instruction pointer     = 0x20:0xc0a25dc0 stack pointer           = 0x28:0xc4df5910 frame pointer           = 0x28:0xc4df5954 code segment            = base 0x0, limit 0xfffff, type 0x1b                        = DPL 0, pres 1, def32 1, gran 1 processor eflags        = interrupt enabled, resume, IOPL = 0 current process         = 12 (irq256: em0:rx 0) trap number             = 12 panic: page fault cpuid = 0 KDB: stack backtrace: #0 0xc08380b7 at kdb_backtrace+0x47 #1 0xc0805617 at panic+0x117 #2 0xc0aebcc3 at trap_fatal+0x323 #3 0xc0aec802 at trap+0x182 #4 0xc0ad5f8c at calltrap+0x6 #5 0xc589f7cc at pfr_update_stats+0x1cc #6 0xc588de21 at pf_test+0x981 #7 0xc5895e79 at pf_check_in+0x39 #8 0xc08c3c68 at pfil_run_hooks+0x78 #9 0xc08e18ae at ip_input+0x24e #10 0xc08c2d9f at netisr_dispatch_src+0x8f #11 0xc08c3040 at netisr_dispatch+0x20 #12 0xc08b9721 at ether_demux+0x171 #13 0xc08b9b6f at ether_nh_input+0x37f #14 0xc08c2d9f at netisr_dispatch_src+0x8f #15 0xc08c3040 at netisr_dispatch+0x20 #16 0xc08b9269 at ether_input+0x19 #17 0xc05b383f at em_rxeof+0x30f Uptime: 1h45m44s Physical memory: 2002 MB Dumping 185 MB: 170 154 138 122 106 90 74 58 42 26 10

Reading symbols from /boot/kernel/pf.ko...Reading symbols from /boot/kernel/pf.ko.symbols... done. done. Loaded symbols for /boot/kernel/pf.ko #0  doadump (textdump=1) at pcpu.h:244 244     pcpu.h: No such file or directory.        in pcpu.h (kgdb) #0  doadump (textdump=1) at pcpu.h:244 #1  0xc08053ba in kern_reboot (howto=260)    at /usr/src/sys/kern/kern_shutdown.c:442 #2  0xc0805651 in panic (fmt=Variable "fmt" is not available. ) at /usr/src/sys/kern/kern_shutdown.c:607 #3  0xc0aebcc3 in trap_fatal (frame=0xc4df58d0, eva=108)    at /usr/src/sys/i386/i386/trap.c:975 #4  0xc0aec802 in trap (frame=0xc4df58d0) at /usr/src/sys/i386/ i386/trap.c:352 #5  0xc0ad5f8c in calltrap () at /usr/src/sys/i386/i386/ exception.s:168 #6  0xc0a25dc0 in uma_zalloc_arg (zone=0x0, udata=0x0, flags=257)    at pcpu.h:244 #7  0xc589f7cc in pfr_update_stats (kt=0xc58d44d8, a=0xc56aa01a, af=2 '\002',    len=52, dir_out=0, op_pass=0, notrule=0) at uma.h:305 #8  0xc588de21 in pf_test (dir=1, ifp=0xc5253c00, m0=0xc4df5acc, eh=0x0,    inp=0x0) at /usr/src/sys/modules/pf/../../contrib/pf/net/pf.c: 7057 #9  0xc5895e79 in pf_check_in (arg=0x0, m=0xc4df5acc, ifp=0xc5253c00, dir=1,    inp=0x0) at /usr/src/sys/modules/pf/../../contrib/pf/net/ pf_ioctl.c:4139 #10 0xc08c3c68 in pfil_run_hooks (ph=0xc0d685e0, mp=0xc4df5b24,    ifp=0xc5253c00, dir=1, inp=0x0) at /usr/src/sys/net/pfil.c:82 #11 0xc08e18ae in ip_input (m=0xc567db00)    at /usr/src/sys/netinet/ip_input.c:510 #12 0xc08c2d9f in netisr_dispatch_src (proto=1, source=0, m=0xc567db00)    at /usr/src/sys/net/netisr.c:1013 #13 0xc08c3040 in netisr_dispatch (proto=1, m=0xc567db00)    at /usr/src/sys/net/netisr.c:1104 #14 0xc08b9721 in ether_demux (ifp=0xc5253c00, m=0xc567db00)    at /usr/src/sys/net/if_ethersubr.c:937 #15 0xc08b9b6f in ether_nh_input (m=0xc567db00)    at /usr/src/sys/net/if_ethersubr.c:756 #16 0xc08c2d9f in netisr_dispatch_src (proto=9, source=0, m=0xc567db00)    at /usr/src/sys/net/netisr.c:1013 #17 0xc08c3040 in netisr_dispatch (proto=9, m=0xc567db00)    at /usr/src/sys/net/netisr.c:1104 #18 0xc08b9269 in ether_input (ifp=0xc5253c00, m=0xc567db00)    at /usr/src/sys/net/if_ethersubr.c:797 #19 0xc05b383f in em_rxeof (rxr=0xc520bc00, count=99, done=0x0)    at /usr/src/sys/dev/e1000/if_em.c:4340 #20 0xc05b3a06 in em_msix_rx (arg=0xc520bc00)    at /usr/src/sys/dev/e1000/if_em.c:1577 #21 0xc07da6eb in intr_event_execute_handlers (p=0xc5157588, ie=0xc5241680)    at /usr/src/sys/kern/kern_intr.c:1257 #22 0xc07dbeaa in ithread_loop (arg=0xc52506e0)    at /usr/src/sys/kern/kern_intr.c:1270 #23 0xc07d78f7 in fork_exit (callout=0xc07dbe30 <ithread_loop>,    arg=0xc52506e0, frame=0xc4df5d28) at /usr/src/sys/kern/ kern_fork.c:995 #24 0xc0ad6004 in fork_trampoline () at /usr/src/sys/i386/i386/ exception.s:275 (kgdb)

################## pf.conf ################## ext_if = "em0"

public_tcp_ports = "{21,25,53,80,143,443,873,993,50021:50121}" public_udp_ports = "53"

table <secure> {someip} table <ssh_brute> persist counters

### Redirection for SMTP rdr on $ext_if proto tcp from any to $ext_if port 225 -> $ext_if port 25

### Block everything in an pass everything out pass out on $ext_if all modulate state block in on $ext_if all

### secure users pass in quick on $ext_if proto tcp from <secure> to any flags S/SA \ modulate state

### public tcp/udp ports rules pass in on $ext_if proto udp to $ext_if port $public_udp_ports pass in on $ext_if proto tcp to $ext_if port $public_tcp_ports flags S/SA \ modulate state

### block ssh bruteforce block in quick from <ssh_brute> pass in quick on $ext_if proto tcp to $ext_if port 22 flags S/SA modulate state \ (max-src-conn 5, max-src-conn-rate 10/60, overload <ssh_brute> flush global)

### block icmp timestamp request/response block in quick on $ext_if inet proto icmp all icmp-type {13, 14} pass in quick on $ext_if proto icmp all

############ end pf.conf ##############