Date: Thu, 4 Apr 2002 23:17:29 +0200 From: Henning Brauer To: misc@openbsd.org Subject: pf: A success story Message-ID: <20020404231729.I24374@bsw005.bsws.de> Mail-Followup-To: misc@openbsd.org I finally did it. I replaced our main firewall, a 2.9/ipf box, with a 3.0/pf box. There was one thing that prevented me from doing this until recently, a big bummer: the box crashed after some hours of operation. Two weeks ago Daniel and I found the error (related to ICMP error messages and some not very sensefull rules for that, it's fixed in -current). I backported -current pf to my frankenstein'd tree. Old and new box are identical hardware-wise: Duron 700, 128 MB RAM, 3x 21143-based NICs using dc(4). With 2.9 and ipf, it ran at over 90% CPU usage at prime time and delays began to be noticeable. The rule file was already fairly high optimized. Short story: I haven't seen the 3.0/pf box less than 89% idle CPU-wise. We are having about 10000 packets per seconds each on the main external and the main internal interface; I have about 1000 rules. One should also note that the new box does even more than the old one, I added two more /24s which also leads to a lot of additional rules. When I tried to add just a _few_ rules for this new space to ipf the load instantly was at 100% even outside the prime time. There wasn't a _single_ block I couldn't explain, there is not the slightest evidence of a problem (opposed to out-of-window error occurring regularily with ipf). I never had such a fine-grained control on packet filtering, and colleagues here without a clue about pf (nor ipf) understand the rule file without further explanations (ok, small exceptions, though that's not due to pf's syntax but due to complicated filter rules based on tcp flags and stuff they neither have a clue about). Thanks to the rule label addition, I have a more detailed and nontheless easier to implement accounting than ever. nmap syn scans now take about 45 minutes per host and report zero open ports, nmap's OS detection fails. That's great. pfctl -si output as of now: Status: Enabled Time: 1017952733 Since: 1017671773 Debug: Urgent Bytes In IPv4: 0 Bytes Out: 0 IPv6: 0 Bytes Out: 0 Inbound Packets IPv4: Passed: 0 Dropped: 0 IPv6: Passed: 0 Dropped: 0 Outbound Packets IPv4: Passed: 0 Dropped: 0 IPv6: Passed: 0 Dropped: 0 States: 20241 pf Counters state searches 1459984318 state inserts 9641366 state removals 9621125 Counters match 706343270 bad-offset 0 fragment 287 short 20 normalize 9928 memory 0 I've seen 25000 concurrent states during normal operations. This is with aggressive timeouts, with normal timeouts I've seen over 40000 states. btw, that's (that beeing the counters) after 3 days 6h. I'm heavily impressed. I have to publically express a hughe "Thank you" to Daniel. The amount of help he gave, the incredible speed in implementing suggestions, the analysis of the crashes due to the icmp-error-messages-with-statefull-filtering-memory-leak and the pool_get issue we did together, the uncounted discussions about modifications, changes, new features, improvements; the analysis of some blocks we both did not understand initially (but that were sooooooooo logical afterwards), that's just incredible. Daniel, I owe you more beers than we can drink ;-). Not to forget that he is a really nice guy and all this conversations didn't just had these great results, it was (and is) also a pleasure and funny. And now that after roughly half a year of intensive conversation we noticed we both have german as native language... ;-)) For completeness, dmesg below. Now further testing 3.1-beta, the changes art did in the vm area are very promising... Greetz Henning OpenBSD 3.0-henning (cr2x) #0: Sun Mar 31 16:32:40 CEST 2002 root@bss004:/usr/src/sys/arch/i386/compile/cr2x cpu0: AMD Duron ("AuthenticAMD" 686-class) 702 MHz cpu0: FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,SYS,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR real mem = 133804032 (130668K) avail mem = 121856000 (119000K) using 1658 buffers containing 6791168 bytes (6632K) of memory mainbus0 (root) bios0 at mainbus0: AT/286+(35) BIOS, date 03/16/01, BIOS32 rev. 0 @ 0xfb3c0 apm0 at bios0: Power Management spec V1.2 apm0: AC on, battery charge unknown pcibios0 at bios0: rev. 2.1 @ 0xf0000/0xb848 pcibios0: PCI IRQ Routing Table rev. 1.0 @ 0xfdd80/160 (8 entries) pcibios0: PCI Exclusive IRQs: 10 11 15 pcibios0: PCI Interrupt Router at 000:07:0 ("VIA VT82C596A PCI-ISA" rev 0x00) pcibios0: PCI bus #1 is the last bus pci0 at mainbus0 bus 0: configuration mode 1 (no bios) pchb0 at pci0 dev 0 function 0 "VIA VT8363 Host" rev 0x03 ppb0 at pci0 dev 1 function 0 "VIA VT8363 PCI-AGP" rev 0x00 pci1 at ppb0 bus 1 pcib0 at pci0 dev 7 function 0 "VIA VT82C686 PCI-ISA" rev 0x40 pciide0 at pci0 dev 7 function 1 "VIA VT82C571 IDE" rev 0x06: ATA100, channel 0 configured to compatibility, channel 1 configured to compatibility wd0 at pciide0 channel 0 drive 0: wd0: 16-sector PIO, LBA, 1625MB, 3303 cyl, 16 head, 63 sec, 3329424 sectors wd0(pciide0:0:0): using PIO mode 4, DMA mode 2 pciide0: channel 1 ignored (disabled) "VIA VT82C686 SMBus" rev 0x40 at pci0 dev 7 function 4 not configured dc0 at pci0 dev 8 function 0 "DEC 21142/3" rev 0x41: irq 10 address 00:40:f4:0a:d1:f5 sqphy0 at dc0 phy 17: Seeq 84220 10/100 media interface, rev. 0 dc1 at pci0 dev 9 function 0 "DEC 21142/3" rev 0x41: irq 11 address 00:00:cb:53:5a:fc sqphy1 at dc1 phy 17: Seeq 84220 10/100 media interface, rev. 0 dc2 at pci0 dev 10 function 0 "DEC 21142/3" rev 0x41: irq 15 address 00:00:cb:53:5f:e2 sqphy2 at dc2 phy 17: Seeq 84220 10/100 media interface, rev. 0 isa0 at pcib0 isadma0 at isa0 pckbc0 at isa0 port 0x60/5 pckbd0 at pckbc0 (kbd slot) pckbc0: using irq 1 for kbd slot wskbd0 at pckbd0: console keyboard pcppi0 at isa0 port 0x61 sysbeep0 at pcppi0 npx0 at isa0 port 0xf0/16: using exception 16 pccom0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo pccom0: console biomask 4000 netmask cc00 ttymask cc02 pctr: user-level cycle counter enabled mtrr: Pentium Pro MTRR support dkcsum: wd0 matched BIOS disk 80 root on wd0a rootdev=0x0 rrootdev=0x300 rawdev=0x302 -- http://2suck.net/hhwl.html Unix is very simple, but it takes a genius to understand the simplicity. (Dennis Ritchie)