Repository navigation
ci: sync upstream 0.9.87 and enforce clean server sources - #21
Merged
Merged
Conversation
…er, search by file hash, hole punch fixes (andrey23127#35, andrey23127#37, andrey23127#38), 0xB2 frame types (andrey23127#36) Bots * A source is judged only once its queries span bots.min_span_secs (10 s). A burst of 30 packets in 15 ms read as 120 000 qpm and got a 24 h ban. * "Uniform timing" sources (~567 qpm, one hash per packet: one client build with broken UDP pacing, ordinary users) are throttled for 24 h to bots.throttle_qpm (60) and still answered, not banned. bots.uniform_action = "ban" restores the old behaviour. Floods are banned. * A banned source that keeps sending through its ban (bots.ipfilter_min_dropped, 10 000 datagrams; x10 if throttled) is appended to storage.ipfilter_path as "a.b.c.d - a.b.c.d , 000 , bot", only when no line covers it, and the filter is reloaded at once. Recorded in bot-ipfilter.txt next to the config and put back when the filter file is replaced (update service). Ledger event ipfilter_added. * The IP filter applies to UDP as well as TCP. * The IP filter file is watched (30 s mtime poll) and reloaded on change. * A new IP filter (bot added, file changed, SIGHUP, /api/reload) closes the TCP sessions from addresses it now blocks; before, the filter was read only at accept and a scanner stayed logged in until it reconnected. Counted as ipfilter_kick. * Bots tab: THROTTLED and IP FILTER badges; filter by state (IP filter, banned, throttled, over) and rule, search, sort by last sanction (default), first seen, dropped, files, rate, IP. * Detections are dropped a day after their ban or throttle ended (they stayed until restart); hashes kept per bot capped at 2 000 (was 20 000); /api/memsize counts them. Scanners and hammers (from a day of /api/activity, 100 000 sources) * Per source and hour, TCP OP_GETSOURCES and UDP source requests counted by file. Scanner: >= bots.scan_min_files (2000) files, >= scan_min_distinct (1000) different, each <= scan_max_repeats (3) times, judged after scan_min_minutes (10) -> scan_action "ban". Hammer: >= hammer_min_files (3000), each file >= hammer_min_rate (5) times a minute -> hammer_action "throttle". jdonkey users (each download once a minute) are neither. "log" records only. Ordinary eMule: ~180 files/h; busiest ordinary source < 1000/h. * bots.exempt (default 178.250.170.0/28, ed2k.shortypower.org): never judged, banned or written to the IP filter. * A bot ban covers TCP: sessions closed, login refused for 24 h (bot_kick, bot_tcp_refused). Throttled sources get throttle_qpm TCP source answers a minute. Search activity (to tune the bot rules from data) * Per source (IPv4, IPv6 /64): TCP/UDP searches, more-results pages, results; TCP/UDP source requests, files per request, files asked, distinct files (64-register HyperLogLog, no hash kept); gaps between requests, busiest minute, requests per hour. * GET /api/activity (totals, by UTC hour, spread over sources), GET /api/activity/sources (rows, sortable, csv), POST /api/activity/reset. * bots.activity_stats (off by default: ~25 MB and a record update per request while on), bots.activity_max_sources (100 000); both apply from the Settings tab, and switching off clears the table. * Distinct-files estimate mixes each hash (structured hashes were overcounted ~9x) and never exceeds the files asked. LowID hole punch (issue andrey23127#35) * OP_IDCHANGE TCP flags carry 0x20000 when the server accepts OP_LOWID_HOLEPUNCH_REQUEST, so a client can tell it from Lugdunum's NAT callback (0x1000) and send 0x60 only where it is understood. * README: CT_SERVER_FLAGS 0x1000 removed from the IPv6 contract (it is Lugdunum's SRVCAP_NATTRAVERSAL); the server keys on CT_MOD_IP_V6 only. Search by file hash * ed2k::HASH, ed2k:SIZE:HASH, ed2k:// links and ed2k magnets in a search string return that file from the hash index (size checked when given), over TCP and UDP, as Lugdunum's Is_ed2k_search. ed2k.shortypower.org sends ed2k::HASH in UDP 0x90 searches and got nothing from this server. * Search result integers in the narrowest type (UINT8/UINT16/UINT32), as Lugdunum; the service showed no source counts from 0x83 tags. Hole punch: per-client UDP port (issue andrey23127#37) * The port handed to a peer in 0x61 came from a per-IP map written by every 0x9F and global query; two LowID clients behind one IP got each other's (or their own) port. Each client's last 0x9F source port is now kept by user hash and preferred; the per-IP port only for a client alone on its IP; then the announced port. * 0x9F counts only from the session's own IP. The 0x60 port no longer replaces the keepalive port. Dropped with the session. Hole punch: peer ID in 0x61 (issue andrey23127#38) * OP_LOWID_HOLEPUNCH_INFO ends with peer_id(4): the requester gets its 0x60 target id, the target the requester's id. 29 bytes; clients read the id only from 29 bytes or more. NAT-T documentation (issue andrey23127#36) * README: the retired eMule 0.72a NAT-T edition is no longer the reference client; its 0xB2 frame type 0x01 collides with eMuleAI's QUIC. The peer-to-peer part of the contract keeps only what the server relies on. * README: table of 0xB2 frame types in use (eMuleAI, emule-qt). * No server change: 0x60-0x62, 0x9F and the 0x20000 flag stay. New optional section [bots]. Compatible with every eMule since obfuscation appeared (0.48a), 0.49c included. Tests: 590 unit + 27 integration pass. Checked against a running server: a 30-packet burst is not flagged; a sustained flood is banned, then appended to the filter file once, reloaded and recorded; a replaced filter file gets the line back; OP_IDCHANGE flags 0x205dd; a logged-in session is closed when its address is added to the filter file; /api/activity counts TCP and UDP searches and source requests and resets; two LowID clients on one IP each get the other's keepalive port and id in a 29-byte 0x61; a TCP client asking 2 000 different files is banned, its session closed and its reconnect refused; a UDP 0x90 search for ed2k::HASH of an indexed file gets its 0x99, a wrong size or unknown hash nothing.
…23127#39) Lugdunum's NAT callback (issue andrey23127#39), read from Lugdunum's code and checked against aMule's implementation, so a client written for it works here unchanged. * Login: 0x1000 in the OP_IDCHANGE TCP flags for every client, as Lugdunum. A client that set 0x1000 in CT_SERVER_FLAGS gets an odd LowID and a 24-byte OP_IDCHANGE whose last field is its TCP port as seen (connection source port for a LowID, its own port for a HighID). Other clients get even LowIDs. * UDP 0xA6 <target_id><requester_id>, server-key obfuscated, from a LowID at its TCP address, is forwarded to the target as <ip><udp_port><requester_id><crypt>[<hash 16>]: by UDP 0xA7 to the target's endpoint when it sent 0xA6 itself within 20 s, else TCP 0x37. Crypt byte and hash as Lugdunum; nothing sent when one side requires the crypt layer and the other can't. No answer to the requester. * Stricter than Lugdunum: the target must have set 0x1000, and requests are limited per requester (burst 10, then 1/s). * network.lugdunum_natt (default on, live). Status tab: forwards, refusals, capable clients; Blocks tab: natcb_* reasons. * No conflict with 0x60-0x62 / 0x9F / 0x20000 (unchanged, separate state), with eNode-go's 0xA6/0xA7 (other directions), or with stock clients (unknown flag bit ignored, OP_IDCHANGE read up to byte 20). Hole-punch extension, flag 0x100000 (agreed in andrey23127#39), only between two clients that both set 0x100000 with 0x1000 in CT_SERVER_FLAGS: * 0x60 wakes the target with Lugdunum's 0x37; the server waits up to 3 s for the target's 0x9F and sends 0x61 (30 bytes, peer_crypt appended) with that fresh port; without it, the same 0x61 with today's ports. * A repeated 0x60 while one waits adds nothing. A session that sent an obfuscated 0x9F has its plain ones ignored. * Every other pair: today's 29-byte 0x61 at once, no 0x37. * network.holepunch_extension (default on, live; needs lugdunum_natt). Bridge: a 0x60 whose target set 0x1000, lacks the extension and has sent no 0x9F in its session also sends that target 0x37, so Lugdunum-only clients (NeoMule's CanLow2Low) can be reached by 0x60. * README: tables of OP_IDCHANGE flag bits (eMule, Lugdunum, NeoLoader / ed2kNET, IPv6 extension, eNode-go, this server; first free 0x200000) and of client CT_SERVER_FLAGS bits; the extension and bridge documented. * Config template: highid_verify_observe = false (the verdict mode feeds the same counters). Tests: 603 unit + 27 integration pass. Checked against a running server: two 0x1000 clients get odd LowIDs and a 24-byte OP_IDCHANGE with the seen port, a stock client an even LowID and 20 bytes; an obfuscated 0xA6 reaches the target as a 27-byte 0x37 with the requester's UDP endpoint and hash; a target that asked within 20 s gets 0xA7 by UDP from the obfuscation port; a target without 0x1000 and a plain 0xA6 get nothing. Extension, scripted clients against a running server: an extension pair gets 0x37 to the target, then 30-byte 0x61 right after the target's 0x9F with its fresh port and crypt byte; with the target silent, 0x61 after 3 s with the announced port; mixed and legacy pairs get 29-byte 0x61 at once and no 0x37; a 0x1000 target that never sent 0x9F gets 0x61 plus 0x37; a plain 0x9F from another port after an obfuscated one does not move the client's port.
… takeover (andrey23127#41), FOUNDSOURCES_OBFU (andrey23127#42), IPv6 dial-back (andrey23127#43), fixes andrey23127#44-andrey23127#49, filter audit Two LowIDs on different servers reach each other through the target's own server, with no registry keyed by user hash. Agreed with aMule and emule-qt in andrey23127#40. * Lugdunum's UDP 0xA6 <target_id><0>[<crypt>] is accepted from a client not logged in here (requester id 0), server-key obfuscated only: the key is bound to the sender's IP, so a datagram that decodes proves it. * The target (logged-in LowID, set 0x1000) gets the usual 0x37 / 0xA7 with requester id 0 and no hash; nothing when one side requires the crypt layer. * A target with the extension (0x100000) answers with 0x9F; the requester then gets UDP 0xA7 <target_ip><target_udp_port><target_id><crypt><hash>, obfuscated, from the port it wrote to; after 3 s without 0x9F, the same with the port the server has. The hash is always there: the requester dials (no 0x61 follows a 0x37 with id 0). Resent at +1.2 s and +3 s. No other answer is ever sent. Requester ID 0 marks a foreign request on every path that wakes a target; same-server forwards never carry it. * Limits per source IP (burst 10 then 1/s; at most 60 target ids in 10 min) and per target (6 foreign wake-ups a minute); one answer in flight per requester and target, inside the hole-punch retry pool. * Announced with 0x100000 in the UDP flags of the status reply (plain and obfuscated, to clients and servers) and in ST_UDPFLAGS at login. * network.holepunch_foreign (default on, live; needs holepunch_extension). Status tab: foreign forwards and answers; Blocks tab: natcb_foreign_*. * README: wire contract and limits; UDP flag note; eNode-go's 0x200000 (NAT wake-up) in both flag tables, first free bit 0x400000. Same-hash logins (issue andrey23127#41): a login whose user hash is already online replaces that session only from the same IPv4 / IPv6 prefix, from the IPv6 the IPv4 session advertised with the same listen port, or when the old session is gone. Otherwise the login is held while the old session is probed: 10 s TCP_USER_TIMEOUT and an OP_SERVERSTATUS on its socket; no ACK (a user who moved) aborts the old connection and the held login goes ahead in ~11 s, inside the client's 25 s login timeout; an ACK refuses the login with a server message, counted (login_hash_in_use). One probe per session at a time, then a 10 s pause; a contender refused after a probe is refused without one for 10 minutes. A shared hash (degenerate, or in the built-in list: DLP antiLeech's 30 community / corrupt / invalid hashes and mldonkey's 3649E68B070E85991648EF223A7F6FFD; or in limits.shared_user_hashes, added to it) keeps the old rule; contender counts are only reported. /api/takeover lists refused hashes with contenders and owner. checked before the HighID probe and again before the insert. limits.hash_takeover_from_any_address (default false, live) restores the old rule. NAT-T counters and caches: * /api/natt: all NAT-T counters and live gauges. New counters for 0x60 (by client family, repeats per pair in 10 min, first answer / 0x62 reason), 0x9F outcomes, extension timing / timeouts / late 0x9F, bridges, Lugdunum 0xA6 (non-capable targets by parity and requester family) and the cross-server path. Status "NAT-T clients" counts 0x9F senders instead of the 0xF9 port. * Obfuscated UDP decode cache aged (30 min, cap 262k) instead of cleared at 8192 every minute; a cold sender is tried as a client (one attempt) before the peer-server tables are scanned; hot / cold / crypt-ping / failed counters on Status. * GETSOURCES cache: TTL 120 s by default (limits.getsources_cache_secs, live; at 15 s half the misses were 30 s - 2 min old), expired entries swept every minute (key and insert time remembered 20 more minutes, outside the cap, so late requests still count), cap 200k, a full cache skips the new entry instead of clearing; expired misses by how long past the TTL they came, and skipped inserts, on Status. * Undecodable obfuscated UDP from an address with a session is counted; from a session with 0x1000 also listed in /api/natt "recent" (once a minute per address): a 0x9F under a stale server key otherwise looks like none was sent (andrey23127#40). * 0x62 reason 5 when either session of a 0x60 is not over IPv4, instead of a 0x61 carrying 0.0.0.0; no extension wake-up for such a pair. * Counter client families include the mod name; /api/natt "recent" keeps the last 60 NAT-T requests in detail. README: aMule's OFFERFILES v1 (PR #1715) merged and on by default (#19). Issues andrey23127#44-andrey23127#49 (itlezy): * andrey23127#44: the obfuscated identity probe sends the 12-byte opening eMule reads (methods supported + preferred + padding length), writes the HELLO only after the peer's answer is checked, and reads that answer's padding as a full byte (up to 254 to anyone but the server being logged into). * andrey23127#45: a publisher with another size than the stored one is not merged (offer_size_mismatch); a sized GETSOURCES / UDP 0x94 record for a hash stored with another size gets no sources (getsources_size_mismatch_ {tcp,udp}_{larger,smaller}). The stored size gives way once the file has no sources. 0 matches anything, and so does a size cut to 32 bits against the full one above 4 GiB (the record takes the full size). A 64-bit FT_FILESIZE tag in OFFERFILES is read whole. /api/sizemismatch: the last 1000 disagreements (hash, sizes, publisher or requester address, publisher software, complete or not; no names), grouped by file. * andrey23127#46: a replaced session's next frame closes it (replaced_session_frame_refused); silent ones still go after 10 min. Not for shared hashes: fencing those made their installs displace each other in a loop (11k shared-hash takeovers in 8 h instead of ~1k). * andrey23127#49: up to 800 sources scanned (UDP 200), HighID first; stock LowID requesters get no LowID sources, NAT-T LowID requesters the NAT-T LowIDs, HighID requesters all; decided when served, the cache stays shared. * andrey23127#48: lockfile back on Rust 1.75 (base64ct =1.6.0, zeroize <1.9, idna_adapter =1.2.0 -> ICU 1.5), resolver.incompatible-rust-versions = "fallback" in .cargo/config.toml. Filter audit: * /api/review?format=audit|auditcsv: false-positive audit that withholds every name with explicit words, a child's age, a word for a child or a second listed term, and all names of rules that are working markers. Per rule: hits, signals, file kinds, publishers, verdict; visible names; hashes withheld for explicit words only; L3 decoy and alias leads with clean names only. CSV with a suggestion per hash. * /api/indexcheck: the whole index against the current filter, hashes only, whitelisted marked; ?purge=1 removes the blocked files from the index. Issue andrey23127#42: OP_GETSOURCES_OBFU (0x23) gets OP_FOUNDSOURCES_OBFU (0x44) with each source's crypt options and, when it supports the crypt layer, 0x80 and its user hash, as Lugdunum; 0x19 still gets 0x42; cached apart. Every 0x61 now carries the peer's crypt setting in byte 29 (30 bytes), not only between extension clients. Issue andrey23127#43: a client's IPv6 (login tag, OP_IPV6_UPDATE or IPv6 session) is published only after the server dialled [address]:login port with the HighID identity probe and got the logged-in user hash back. Until then ST_IPV6_STATUS is 0x01; the verdict comes in a fresh OP_SERVERIDENT, 0x07 (published) or 0x05 (IPV6ST_PROBED = 0x04, not published). A new address in OP_IPV6_UPDATE is dialled again, the same keeps its verdict. Shared probe ceiling (retried a minute later when full), at most 4 dials per session. network.ipv6_probe (default on, live), in force with ipv6_publish_sources and an IPv6 route on the host, announced with 0x400000 in OP_IDCHANGE; otherwise addresses are published on trust. Status "IPv6 dial-back" with failures by stage (timeout / refused / unreachable / after connect), Blocks ipv6_probe_*. A connect error other than a reset is "connect unreachable" (HighID statistics too). Tests: 625 unit + 29 integration pass. Checked against a running server with an unlogged UDP requester: the status reply carries 0x100000; an extension target gets 0x37 (requester id 0, its UDP port) and, after its 0x9F, the requester gets 0xA7 from the obfuscation port with the target's fresh port and id; a Lugdunum-only target gets 0x37 and the requester nothing; a silent extension target is answered after 3 s; a plain request and an unknown target get nothing. Same-hash login, against a running server: a second login with the hash from 127.0.0.2 gets the server message and is closed while the first session stays up; a new login from the first session's address replaces it. Probe: with the old session's path dropped (iptables, no ACK and no RST), a login with its hash from another address was held and accepted after 10.7 s; a third login while that one lived was refused after the probe. Audit and index check against a running server with a test term: three clean names shown under a "likely false positive" rule, the explicit one withheld and listed by hash; a term added later finds the indexed file, purge removes it. 0x23 against a running server: 0x44 with crypt 0x83 and the publisher's hash for a source with 0x600; 0x19 still 0x42.
…ad UDP 0x94 sizes From the first 14 hours of 0.9.86 on ed2k-rust (/api/sizemismatch): * Cross-server callback (andrey23127#40): for a target with the extension, a required crypt layer on either side only needs support on the other (the requester dials with the target's hash from 0xA7); the old refusal stays for Lugdunum-only targets, which dial without the requester's hash. * A size equal to (hashes - 1) x 9728000 agrees with the real size of the file (issue andrey23127#45): it is the lower bound a client uses for a download started from a bare hash, and most disagreements were exactly that. Such publishers are merged as sources, such requests answered (getsources_size_part_estimate), and a record holding only the guess takes the real size when it arrives. * UDP 0x94 records with the large-file sentinel followed by a size above 256 GiB are answered by hash alone and counted (udp_getsources2_bad_size) instead of being refused as a size mismatch. * NAT-T token (issue andrey23127#40): sessions that may send 0x9F (0x100000, 0x1000 or 0xF9) get 16 random bytes in OP_SERVERIDENT, tag ST_NATT_TOKEN 0xAC; 0x100000 clients get OP_SERVERIDENT at login for it. A 0x9F of <user hash 16><token 16> is believed from any address, and that address and port go to peers in 0x61 / 0xA7 (an IPv6 session can then punch over IPv4). Wrong token ignored. natt_9f_token, natt_9f_token_other_ip, natt_9f_bad_token. * Filter audit: format=hashlists (ban-list additions: films, pictures, archives a term layer blocked; filter-list additions: indexed files under 3+ unrelated names; audio/doc/software hits to look at), format=terms (words in 5+ blocked files, 90%+ blocked, not blocked alone), format=audio (every blocked audio file with its name, unless it carries a child signal). A ban-list candidate is never also listed as a decoy. cbr/cbz count as archives, disc images as other. The second- marker signal strips the rule's term across separators (it had made every L1/L4 rule a "marker"). Names shown only for audio, documents, software. The age rule on a film or picture is never a whitelist suggestion and reads "age alone (by design)". * login_takeover_same_address counts live sessions replaced from their own address, beside replaced_session_frame_refused. Tests: 626 unit + 29 integration pass.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Tracks emulebb/emulebb-rust#164
Summary
Validation