| 1 | ============================================================================ |
| 2 | |
| 3 | Prometheus QoS - steal fire from your ISP ! |
| 4 | "fair-per-IP" quality of service utility |
| 5 | requires Linux kernel with HTB qdisc enabled |
| 6 | |
| 7 | Copyright(C) 2005-2008 Michael Polak (xChaos) original source code |
| 8 | Copyright(C) 2007-2008 Martin Svoboda (Ludva) iptables-restore, CLASSIFY |
| 9 | |
| 10 | Credit: CZFree.Net+Netdave (idea), Aquarius (.rpm), Gandalf (.deb) |
| 11 | ...and: Martin Devera (.cz) for his HTB qdisc (of course) |
| 12 | ...and: Jakub Walczak (.pl) for providing feedback and patches |
| 13 | ...and: Ing. Jiri Engelthaler (.cz) for bugfixes and Asus WL500 port |
| 14 | ...and: Dial Telecom (our slightly expensive ISP) for chance to test it |
| 15 | |
| 16 | Feedback: xchaos(at)arachne.cz |
| 17 | Homepage: http://gpl.arachne.cz |
| 18 | SVN tree: https://dev.arachne.cz/svn/prometheus |
| 19 | ============================================================================ |
| 20 | |
| 21 | QoS (or Quality-of-service) is IPv4 traffic shaper replacement for Internet |
| 22 | Service Providers (ISP). Dump your vintage hard-wired routers/shapers |
| 23 | (C|sco, etc.) in favour of powerful open source and free solution ! |
| 24 | |
| 25 | Prometheus QoS generates multiple nested HTB tc classes with various rate |
| 26 | and ceil values, and implements optional daily traffic quotas and data |
| 27 | transfer statistics (as HTML). It is compatible with NAT, both asymetrical |
| 28 | and symetrical, yet still provides good two-way shaping and prioritizing, |
| 29 | both upload and download. Prometheus QoS allows both "hard shaping" |
| 30 | (reducing HTB ceil value for aggressive downloaders) and "soft shaping" |
| 31 | (keeping HTB ceil, but reducing HTB prio, probably optimal solution for |
| 32 | normal users). |
| 33 | |
| 34 | Prometheus iGW was written in C<<1, which means it compiles simply with |
| 35 | GNU C Compiler, and doesn't require any external liberaries (except libc) |
| 36 | and huge interpreter packages (like Perl or Java) to run. However, it |
| 37 | depends on HTB algorithm hardcoded in Linux kernel. It is currently being |
| 38 | tested in real-world enviroment to provide QoS services on 30 Mbps internet |
| 39 | gateway and proxy being used by 2000+ PCs connected to gateway using |
| 40 | CZFree.Net broadband community network. |
| 41 | |
| 42 | Advantages over more straightforward traffic control scripts include |
| 43 | HTB fine tuning features (rate and ceil magic), data transfer statistics, |
| 44 | optional data transfer quotas, full NAT (both symetric and one way) |
| 45 | compatibility and optinal sharing of bandwith by IPs in completely |
| 46 | different subnets. |
| 47 | |
| 48 | Performance and scaling - current release: |
| 49 | |
| 50 | we run Prometheus QoS on Celeron 2.8 Ghz serving around 600 individual |
| 51 | traffic classes (fine tuning is using five user-defined prometheus.conf |
| 52 | keyword) and another 2000 IPs sharing bandwith with certain other IPs |
| 53 | ("sharing-" keyword). Prometheus QoS is especially strong tool if you want |
| 54 | IP's from different subnets to share the same traffic class. |
| 55 | |
| 56 | With 30 Mbps (each way) total capacity of line, Cisco Catalyst 2950 on |
| 57 | ISP side and up to cca 6000 packets per seconds, we were running into some |
| 58 | problems with overall system load. We moved QoS from Athlon 1700 XP to |
| 59 | Celeron 2.8 Ghz, and kept all SNAT related stuff (see optinal-tools directory) |
| 60 | on Athlon 1.7 Ghz, which alowed for peak throughput up to 10000 pps. |
| 61 | |
| 62 | Performance fine tuning - history: |
| 63 | |
| 64 | With kernel version 2.4.20 and release 0.2 we started to experience problems |
| 65 | at cca 1500 packets/sec. However, with new iptables indexing feature |
| 66 | implemented in 0.3 release, system load seems to be approximately |
| 67 | 10 times lower. Same HW was later shaping 2000 packets/sec without problems, |
| 68 | and it looked like comparable relatively low-end system should be able to do |
| 69 | traffic shaping for at least 10000 packets/sec (well, if HotSaNIC was turned |
| 70 | off, of course <g>). With 0.6 release and dynamicaly calculated iptables |
| 71 | indexing scheme we made it up to 6000 packets/sec, and then we ran into some |
| 72 | performance-related problems, which may be related to the fact we are doing |
| 73 | SNAT of 1000+ individual IP addresses on the same machine which is doing |
| 74 | also the QoS: something on the way seems to be limited to 34 Mbps HD |
| 75 | (half-duplex, sum of upload and download) no matter what we try. Our ISP |
| 76 | claims the fault is not on his side, so our next step will be to separate |
| 77 | traffic shaping and massive SNAT (IP masquerading) and assign separate |
| 78 | PC-based router to do each task. |
| 79 | |
| 80 | Maximum performance observed with prometheus 0.6 and hashtable optimization |
| 81 | of tables with individual SNAT targets was up to 9000 packets/sec at cca 40 |
| 82 | Mbps half-duplex (more then 20 Mbps fyull-duplex). However, this required |
| 83 | massive optimization, including |
| 84 | echo -n 65000 > /proc/sys/net/ipv4/ip_conntrack_max |
| 85 | and |
| 86 | echo -n 21600 > /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_timeout_established |
| 87 | and disabling of most userspace applications (like eg. hotsanic). At the |
| 88 | same time, router machine and system was accumulating wide set of various |
| 89 | performance related problems, which required us to reboot it at least |
| 90 | mohtly. |
| 91 | |
| 92 | Note: Some time ago it seemed that maximum index of tc classes was restricted |
| 93 | to 10000 - but I haven't checked this again for quite a while. |
| 94 | Another note: All the echo stuff in previous paragraph can be also achieved by |
| 95 | adding following lines to /etc/sysctl.conf which is much cleaner way to do it: |
| 96 | sys.net.ipv4.ip_conntrack_max=65000 |
| 97 | sys.net.ipv4.netfilter.ip_conntrack_tcp_timeout_established=21600 |
| 98 | |
| 99 | Future plans include also setting of individual daily limits on maximum |
| 100 | pps (packets per second) rates allocated to individual IP addresses (this |
| 101 | may be needed partly because of problems mentioned above). |
| 102 | |
| 103 | ============================================================================ |
| 104 | |
| 105 | Prometheus QoS is free software; you can redistribute it and/or |
| 106 | modify it under the terms of the GNU General Public License as |
| 107 | published by the Free Software Foundation; either version 2.1 of |
| 108 | the License, or (at your option) any later version. |
| 109 | |
| 110 | Prometheus QoS is distributed in the hope that it will be useful, |
| 111 | but WITHOUT ANY WARRANTY; without even the implied warranty of |
| 112 | MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU |
| 113 | General Public License for more details. |
| 114 | |
| 115 | You should have received a copy of the GNU General Public License |
| 116 | along with Prometheus QoS source code; if not, write to |
| 117 | Michael Polak, Svojsikova 7, 169 00 Praha 6 Czech Republic |