a4f661fc |
1 | ============================================================================ |
2 | |
3 | Prometheus QoS - steal fire from your ISP ! |
4 | "fair-per-IP" quality of service utility |
5 | requires Linux kernel with HTB qdisc enabled |
ae776b10 |
6 | |
7 | Copyright(C) 2005-2008 Michael Polak (xChaos) original source code |
8 | Copyright(C) 2007-2008 Martin Svoboda (Ludva) iptables-restore, CLASSIFY |
9 | |
10 | Credit: CZFree.Net+Netdave (idea), Aquarius (.rpm), Gandalf (.deb) |
e3e0ce38 |
11 | ...and: Martin Devera (.cz) for his HTB qdisc (of course) |
ae776b10 |
12 | ...and: Jakub Walczak (.pl) for providing feedback and patches |
13 | ...and: Ing. Jiri Engelthaler (.cz) for bugfixes and Asus WL500 port |
14 | ...and: Dial Telecom (our slightly expensive ISP) for chance to test it |
a4f661fc |
15 | |
16 | Feedback: xchaos(at)arachne.cz |
17 | Homepage: http://gpl.arachne.cz |
ae776b10 |
18 | SVN tree: https://dev.arachne.cz/svn/prometheus |
a4f661fc |
19 | ============================================================================ |
20 | |
21 | QoS (or Quality-of-service) is IPv4 traffic shaper replacement for Internet |
22 | Service Providers (ISP). Dump your vintage hard-wired routers/shapers |
23 | (C|sco, etc.) in favour of powerful open source and free solution ! |
24 | |
25 | Prometheus QoS generates multiple nested HTB tc classes with various rate |
26 | and ceil values, and implements optional daily traffic quotas and data |
27 | transfer statistics (as HTML). It is compatible with NAT, both asymetrical |
28 | and symetrical, yet still provides good two-way shaping and prioritizing, |
29 | both upload and download. Prometheus QoS allows both "hard shaping" |
30 | (reducing HTB ceil value for aggressive downloaders) and "soft shaping" |
31 | (keeping HTB ceil, but reducing HTB prio, probably optimal solution for |
32 | normal users). |
33 | |
34 | Prometheus iGW was written in C<<1, which means it compiles simply with |
35 | GNU C Compiler, and doesn't require any external liberaries (except libc) |
36 | and huge interpreter packages (like Perl or Java) to run. However, it |
37 | depends on HTB algorithm hardcoded in Linux kernel. It is currently being |
38 | tested in real-world enviroment to provide QoS services on 30 Mbps internet |
39 | gateway and proxy being used by 2000+ PCs connected to gateway using |
40 | CZFree.Net broadband community network. |
41 | |
42 | Advantages over more straightforward traffic control scripts include |
43 | HTB fine tuning features (rate and ceil magic), data transfer statistics, |
44 | optional data transfer quotas, full NAT (both symetric and one way) |
45 | compatibility and optinal sharing of bandwith by IPs in completely |
46 | different subnets. |
47 | |
48 | Performance and scaling - current release: |
49 | |
50 | we run Prometheus QoS on Celeron 2.8 Ghz serving around 600 individual |
51 | traffic classes (fine tuning is using five user-defined prometheus.conf |
52 | keyword) and another 2000 IPs sharing bandwith with certain other IPs |
53 | ("sharing-" keyword). Prometheus QoS is especially strong tool if you want |
54 | IP's from different subnets to share the same traffic class. |
55 | |
56 | With 30 Mbps (each way) total capacity of line, Cisco Catalyst 2950 on |
57 | ISP side and up to cca 6000 packets per seconds, we were running into some |
58 | problems with overall system load. We moved QoS from Athlon 1700 XP to |
59 | Celeron 2.8 Ghz, and kept all SNAT related stuff (see optinal-tools directory) |
60 | on Athlon 1.7 Ghz, which alowed for peak throughput up to 10000 pps. |
61 | |
62 | Performance fine tuning - history: |
63 | |
64 | With kernel version 2.4.20 and release 0.2 we started to experience problems |
65 | at cca 1500 packets/sec. However, with new iptables indexing feature |
66 | implemented in 0.3 release, system load seems to be approximately |
67 | 10 times lower. Same HW was later shaping 2000 packets/sec without problems, |
68 | and it looked like comparable relatively low-end system should be able to do |
69 | traffic shaping for at least 10000 packets/sec (well, if HotSaNIC was turned |
70 | off, of course <g>). With 0.6 release and dynamicaly calculated iptables |
71 | indexing scheme we made it up to 6000 packets/sec, and then we ran into some |
72 | performance-related problems, which may be related to the fact we are doing |
73 | SNAT of 1000+ individual IP addresses on the same machine which is doing |
74 | also the QoS: something on the way seems to be limited to 34 Mbps HD |
75 | (half-duplex, sum of upload and download) no matter what we try. Our ISP |
76 | claims the fault is not on his side, so our next step will be to separate |
77 | traffic shaping and massive SNAT (IP masquerading) and assign separate |
78 | PC-based router to do each task. |
79 | |
80 | Maximum performance observed with prometheus 0.6 and hashtable optimization |
81 | of tables with individual SNAT targets was up to 9000 packets/sec at cca 40 |
82 | Mbps half-duplex (more then 20 Mbps fyull-duplex). However, this required |
83 | massive optimization, including |
84 | echo -n 65000 > /proc/sys/net/ipv4/ip_conntrack_max |
85 | and |
86 | echo -n 21600 > /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_timeout_established |
87 | and disabling of most userspace applications (like eg. hotsanic). At the |
88 | same time, router machine and system was accumulating wide set of various |
89 | performance related problems, which required us to reboot it at least |
90 | mohtly. |
91 | |
92 | Note: Some time ago it seemed that maximum index of tc classes was restricted |
93 | to 10000 - but I haven't checked this again for quite a while. |
94 | Another note: All the echo stuff in previous paragraph can be also achieved by |
95 | adding following lines to /etc/sysctl.conf which is much cleaner way to do it: |
96 | sys.net.ipv4.ip_conntrack_max=65000 |
97 | sys.net.ipv4.netfilter.ip_conntrack_tcp_timeout_established=21600 |
98 | |
99 | Future plans include also setting of individual daily limits on maximum |
100 | pps (packets per second) rates allocated to individual IP addresses (this |
101 | may be needed partly because of problems mentioned above). |
102 | |
103 | ============================================================================ |
104 | |
105 | Prometheus QoS is free software; you can redistribute it and/or |
106 | modify it under the terms of the GNU General Public License as |
107 | published by the Free Software Foundation; either version 2.1 of |
108 | the License, or (at your option) any later version. |
109 | |
110 | Prometheus QoS is distributed in the hope that it will be useful, |
111 | but WITHOUT ANY WARRANTY; without even the implied warranty of |
112 | MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU |
113 | General Public License for more details. |
114 | |
115 | You should have received a copy of the GNU General Public License |
116 | along with Prometheus QoS source code; if not, write to |
117 | Michael Polak, Svojsikova 7, 169 00 Praha 6 Czech Republic |