a4f661fc |
1 | ============================================================================ |
2 | |
3 | Prometheus QoS - steal fire from your ISP ! |
4 | "fair-per-IP" quality of service utility |
5 | requires Linux kernel with HTB qdisc enabled |
6 | GNU+ Copyright(G)2007, Michael Polak (xChaos) |
7 | Credits: Credits: CZFree.Net, Netdave, aquarius |
8 | ...and Martin Devera (.cz) for his HTB qdisc (of course) |
9 | ...and Jakub Walczak (.pl) for providing feedback and patches |
10 | ...and Ing. Jiri Engelthaler (.cz) for bugfixes and Asus WL500 port |
11 | ...and Dial Telecom (slightly expensive ISP) for chance to test it |
12 | |
13 | Feedback: xchaos(at)arachne.cz |
14 | Homepage: http://gpl.arachne.cz |
15 | |
16 | ============================================================================ |
17 | |
18 | QoS (or Quality-of-service) is IPv4 traffic shaper replacement for Internet |
19 | Service Providers (ISP). Dump your vintage hard-wired routers/shapers |
20 | (C|sco, etc.) in favour of powerful open source and free solution ! |
21 | |
22 | Prometheus QoS generates multiple nested HTB tc classes with various rate |
23 | and ceil values, and implements optional daily traffic quotas and data |
24 | transfer statistics (as HTML). It is compatible with NAT, both asymetrical |
25 | and symetrical, yet still provides good two-way shaping and prioritizing, |
26 | both upload and download. Prometheus QoS allows both "hard shaping" |
27 | (reducing HTB ceil value for aggressive downloaders) and "soft shaping" |
28 | (keeping HTB ceil, but reducing HTB prio, probably optimal solution for |
29 | normal users). |
30 | |
31 | Prometheus iGW was written in C<<1, which means it compiles simply with |
32 | GNU C Compiler, and doesn't require any external liberaries (except libc) |
33 | and huge interpreter packages (like Perl or Java) to run. However, it |
34 | depends on HTB algorithm hardcoded in Linux kernel. It is currently being |
35 | tested in real-world enviroment to provide QoS services on 30 Mbps internet |
36 | gateway and proxy being used by 2000+ PCs connected to gateway using |
37 | CZFree.Net broadband community network. |
38 | |
39 | Advantages over more straightforward traffic control scripts include |
40 | HTB fine tuning features (rate and ceil magic), data transfer statistics, |
41 | optional data transfer quotas, full NAT (both symetric and one way) |
42 | compatibility and optinal sharing of bandwith by IPs in completely |
43 | different subnets. |
44 | |
45 | Performance and scaling - current release: |
46 | |
47 | we run Prometheus QoS on Celeron 2.8 Ghz serving around 600 individual |
48 | traffic classes (fine tuning is using five user-defined prometheus.conf |
49 | keyword) and another 2000 IPs sharing bandwith with certain other IPs |
50 | ("sharing-" keyword). Prometheus QoS is especially strong tool if you want |
51 | IP's from different subnets to share the same traffic class. |
52 | |
53 | With 30 Mbps (each way) total capacity of line, Cisco Catalyst 2950 on |
54 | ISP side and up to cca 6000 packets per seconds, we were running into some |
55 | problems with overall system load. We moved QoS from Athlon 1700 XP to |
56 | Celeron 2.8 Ghz, and kept all SNAT related stuff (see optinal-tools directory) |
57 | on Athlon 1.7 Ghz, which alowed for peak throughput up to 10000 pps. |
58 | |
59 | Performance fine tuning - history: |
60 | |
61 | With kernel version 2.4.20 and release 0.2 we started to experience problems |
62 | at cca 1500 packets/sec. However, with new iptables indexing feature |
63 | implemented in 0.3 release, system load seems to be approximately |
64 | 10 times lower. Same HW was later shaping 2000 packets/sec without problems, |
65 | and it looked like comparable relatively low-end system should be able to do |
66 | traffic shaping for at least 10000 packets/sec (well, if HotSaNIC was turned |
67 | off, of course <g>). With 0.6 release and dynamicaly calculated iptables |
68 | indexing scheme we made it up to 6000 packets/sec, and then we ran into some |
69 | performance-related problems, which may be related to the fact we are doing |
70 | SNAT of 1000+ individual IP addresses on the same machine which is doing |
71 | also the QoS: something on the way seems to be limited to 34 Mbps HD |
72 | (half-duplex, sum of upload and download) no matter what we try. Our ISP |
73 | claims the fault is not on his side, so our next step will be to separate |
74 | traffic shaping and massive SNAT (IP masquerading) and assign separate |
75 | PC-based router to do each task. |
76 | |
77 | Maximum performance observed with prometheus 0.6 and hashtable optimization |
78 | of tables with individual SNAT targets was up to 9000 packets/sec at cca 40 |
79 | Mbps half-duplex (more then 20 Mbps fyull-duplex). However, this required |
80 | massive optimization, including |
81 | echo -n 65000 > /proc/sys/net/ipv4/ip_conntrack_max |
82 | and |
83 | echo -n 21600 > /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_timeout_established |
84 | and disabling of most userspace applications (like eg. hotsanic). At the |
85 | same time, router machine and system was accumulating wide set of various |
86 | performance related problems, which required us to reboot it at least |
87 | mohtly. |
88 | |
89 | Note: Some time ago it seemed that maximum index of tc classes was restricted |
90 | to 10000 - but I haven't checked this again for quite a while. |
91 | Another note: All the echo stuff in previous paragraph can be also achieved by |
92 | adding following lines to /etc/sysctl.conf which is much cleaner way to do it: |
93 | sys.net.ipv4.ip_conntrack_max=65000 |
94 | sys.net.ipv4.netfilter.ip_conntrack_tcp_timeout_established=21600 |
95 | |
96 | Future plans include also setting of individual daily limits on maximum |
97 | pps (packets per second) rates allocated to individual IP addresses (this |
98 | may be needed partly because of problems mentioned above). |
99 | |
100 | ============================================================================ |
101 | |
102 | Prometheus QoS is free software; you can redistribute it and/or |
103 | modify it under the terms of the GNU General Public License as |
104 | published by the Free Software Foundation; either version 2.1 of |
105 | the License, or (at your option) any later version. |
106 | |
107 | Prometheus QoS is distributed in the hope that it will be useful, |
108 | but WITHOUT ANY WARRANTY; without even the implied warranty of |
109 | MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU |
110 | General Public License for more details. |
111 | |
112 | You should have received a copy of the GNU General Public License |
113 | along with Prometheus QoS source code; if not, write to |
114 | Michael Polak, Svojsikova 7, 169 00 Praha 6 Czech Republic |