Security In Practice | CyOps Consulting Team

Callum McMahon was not running anything unusual. A Cursor MCP plugin, normal work. His machine exhausted its RAM and crashed. The cause: a transitive litellm dependency he had not knowingly installed was executing a credential sweep in the background. He had not installed litellm directly. He did not know it was there.

The malicious versions were live on PyPI for approximately three hours before quarantine (Snyk 2026). LiteLLM downloads roughly 3.4 million times per day (Snyk 2026). A three-hour window against that volume is not a narrow escape; it is a structural exposure event. The telnyx PyPI package had accumulated more than 3.75 million total downloads before its own compromise (ReversingLabs 2026).

The incident advisories tell you what to rotate. This piece explains what structural conditions allowed the chain to happen, and what remains unaddressed after the rotation is complete.

The entry point was not a vulnerability in LiteLLM's code. It was a configuration decision in LiteLLM's CI/CD pipeline. Trivy, the container scanner, was running unpinned with access to all pipeline environment secrets, including the PyPI publishing token it had no functional reason to hold. TeamPCP modified Trivy's GitHub Actions tags to deliver credential-harvesting malware that swept keys, tokens, and configuration data from the pipeline environment (Hartman 2026; Snyk 2026). The PyPI publishing token came out of that sweep, and that single credential unlocked a package serving 95 million monthly downloads (Snyk 2026).

The LiteLLM compromise is Phase 09 of a campaign active since at least December 2025 (Hartman 2026; Datadog Security Labs 2026). Infrastructure consistency across all phases is unambiguous: the same RSA key pair, the same archive naming conventions, and tpcp-docs-prefixed GitHub repositories used as dead-drop staging throughout. This is a disciplined, persistent operator with repeatable infrastructure, not an opportunistic one reusing tools between unrelated operations.

The credential triage logic matters for scoping your response. TeamPCP is not spraying every harvested credential automatically. It appears to be selecting targets with PyPI publishing privileges from a curated trove (Hartman 2026; Snyk 2026). If your credentials were in scope of any affected system, you are a candidate for active exploitation, not a peripheral casualty of a mass campaign.

As of March 31, 2026, three operational tracks are running simultaneously: direct credential exploitation against high-value targets, proprietary ransomware operations via CipherForce, and mass affiliate distribution via Vect through BreachForums (Hartman 2026; HelpNetSecurity 2026). The supply chain phase has paused. The monetisation phase has not.

As Jacob Krell of Suzu Labs observed in analysis cited by ReversingLabs: "One unmanaged dependency. One chain reaction. Five ecosystems compromised in under a month." (Krell 2026). Three structural conditions made this chain possible, and all three remain unaddressed in the majority of affected pipelines.

Gap 1: Security Tools Run With Unrestricted Secret Access, Making Them the Highest-Value Target

TeamPCP's target selection was not opportunistic. Trivy, KICS, and LiteLLM share a structural characteristic: each requires broad read access to credentials, environment variables, and configuration data to perform its function (Datadog Security Labs 2026; Hartman 2026). In most CI/CD implementations, this access is inherited from the pipeline job, not granted per step. Security scanners, build steps, and deployment steps run in the same environment with the same secret scope. LiteLLM's pipeline gave Trivy access to its PyPI publishing token and GitHub PATs, credentials Trivy had no operational reason to hold. It held them because the pipeline was configured the way most pipelines are configured.

This pattern persists because per-step secret scoping is rarely implemented. Pipeline configurations are written once and seldom revisited from a least-privilege perspective. Compounding this, automated update behaviour removes the only manual checkpoint between a tag hijack and credential access: when a pipeline pulls the newest patch the moment it drops, there is no human review before that code runs with full secret scope (Hartman 2026).

The practical consequence is a single-pivot exposure model. One compromised scanner, whether through a tag hijack, a Docker image swap, or a GitHub Action hash drift, reaches every secret in the pipeline's scope within minutes (Datadog Security Labs 2026; Hartman 2026).

The immediate action requires no new tooling: audit which pipeline steps currently have access to PyPI tokens, NPM tokens, container registry credentials, and GitHub PATs with write scope. Any step that does not actively use a credential should not have it in scope.

The structural shift is distinct: implement least-privilege secret scoping at the CI job level, so publishing credentials are scoped exclusively to publishing steps. This is a multi-sprint engineering project, not a configuration patch. Treat the configuration audit and the architectural remediation as separate work items with separate timelines.

The secret scoping failure did not operate in isolation. It compounded a second structural condition that gave many teams false confidence their existing defences had already addressed the exposure.

Gap 2: Hash Pinning Confirms File Integrity, But Cannot Detect Whether What the Registry Sent Is Malicious

The prevailing assumption: pinning to a hash or a specific digest protects against supply chain poisoning. Against credential-based compromise of a legitimate maintainer account, it does not (Snyk 2026).

The malicious litellm versions were published using legitimate maintainer credentials. The hash PyPI computed was correct for the content it received. There was no mismatch to detect, no suspicious domain, and no misspelled package name. pip install --require-hashes would have passed without error (Snyk 2026). The RECORD file was regenerated with correct hashes for the malicious content. No anomalous registry behaviour occurred. The attack surface is the maintainer account, not the package name or the registry hash.

The telnyx package compounds this further. Static analysis of the published package yielded nothing because the credential-stealing payload did not exist inside it. The payload resided on the C2 server, embedded in the audio frame data of a structurally valid WAV file, XOR-encrypted with a key stored in the first 8 bytes of the audio buffer (Endor Labs 2026). There was nothing in the package for a hash to detect.

The malicious litellm versions abused Python's .pth file mechanism, which executes arbitrary code during interpreter initialisation, before application code runs (Snyk 2026; Datadog Security Labs 2026). The injected file, litellm_init.pth, was written to Python's site-packages directory. The .pth mechanism fires when any Python process starts, including pip itself, meaning the payload executed during build steps, not only at application runtime. Detection query, validated by Datadog: @file.path:(*litellm_init.pth) (Datadog Security Labs 2026).

Hash pinning does protect against typosquatting and CDN-level tampering, the attack vectors that motivated most teams to implement it after earlier supply chain incidents. The mental model is correct for the threat it was designed against. It is wrong for credential-based compromise of a legitimate maintainer account, and most teams have not updated their controls to reflect that distinction.

The only install-time detection path that would have flagged either compromise is source-to-artifact comparison: verifying the published wheel against the upstream GitHub source commit, looking for files present in the wheel but absent from the repository. For litellm, litellm_init.pth was present in the malicious wheel and absent from the upstream source. For telnyx, unexpected entries in pyproject.toml and new files in package subdirectories were the detectable differentiators. Virtually no organisation performs this comparison automatically in CI.

For high-risk package categories (security tooling, LLM proxies, CI/CD-adjacent libraries), run a source-to-artifact diff before each update. Check for litellm_init.pth in Python's site-packages on every system that ran Python between March 24 to 27, 2026. Source-to-artifact comparison is a manual process. It adds pipeline latency. Apply it to the highest-risk packages in your pipeline, not uniformly across all dependencies; uniform application requires tooling investment that most teams do not currently have budgeted. For telnyx specifically, no validated post-execution detection query has been published at the time of writing; check pip show telnyx for installed versions and treat any confirmed installation during the exposure window as requiring credential rotation and network log review.

Even practitioners who run this check correctly may be auditing a narrower scope than their actual exposure warrants.

Gap 3: Post-Incident Audits Focus on Direct Installs, But the True Exposure Is Wider

LiteLLM is present in 36% of cloud environments (Wiz 2026). The majority of that 36% did not explicitly install it. It arrives as a transitive dependency inside AI agent frameworks, MCP server plugins, LLM orchestration tools, and more than 600 public GitHub projects with unpinned LiteLLM dependencies (Snyk 2026). The audit question "did you install litellm?" returns false negatives at scale. The correct question is: does any tool, plugin, or framework in your environment have litellm in its dependency graph? These are materially different questions with materially different answers.

Callum McMahon's case establishes the mechanism, and what it establishes for audit scope is this: his workstation, running AI-adjacent tooling in the normal course of his role, would have held Kubernetes configuration files, cloud credentials across AWS, GCP, and Azure, CI/CD secrets, SSH keys, and Docker configs. Not through carelessness, but because those credentials were operationally necessary. The malware targeted all of them: environment variables, SSH keys, cloud credentials, Kubernetes configs, CI/CD secrets, Docker configs, database credentials, and cryptocurrency wallets (Hartman 2026; Snyk 2026). A transitive install on a developer workstation with Kubernetes config access is not a low-severity event. The framing of "this was just a developer machine" is a scoping error that produces incomplete remediation.

This gap persists because dependency graph visibility for direct installs is reasonably mature, but transitive dependency mapping across developer workstations, CI runners, and containerised workloads pulling dependencies at runtime is rarely complete. Most software composition analysis (SCA) tooling reads declared dependency files: requirements.txt, pyproject.toml, package-lock.json. It does not enumerate what is actually installed across developer environments at runtime. The gap between what was declared and what is installed is where this exposure lived.

Organisations limiting their audit to production services are missing three environment categories with confirmed exposure paths: developer workstations running AI-adjacent tooling, CI runners executing LLM-adjacent build steps, and containerised workloads pulling dependencies at runtime. The litellm_init.pth detection check (@file.path:(*litellm_init.pth), per Datadog Security Labs 2026) must run across all three, not only production containers.

The immediate checks require no new tooling: run pip show litellm and pip show telnyx across developer workstations as well as CI runners and production containers. Check for litellm_init.pth in Python's site-packages on every system that ran any Python process between March 24 to 27, 2026, including pip itself. Check network logs for outbound HTTP connections to 83.142.209.203 (Datadog Security Labs 2026) during that same window. The driving audit question is not "which package names are installed" but "which systems in your environment ran any Python process during that window?"

The structural shift requires either tooling investment or a significant manual audit effort. The target state is the ability to answer "is litellm installed anywhere in our environment?" across workstation, runner, and container contexts at runtime, not just against declared dependency files. That is a programme-level investment. The pip show and .pth checks are executable this week. The runtime visibility capability is not.

Rotating secrets addresses what the campaign extracted. It does not address what the campaign does next with a credential trove it has already collected.

You Rotated Your Secrets. What It Does Not Cover.

TeamPCP is now operating two concurrent ransomware tracks from the same credential trove (Hartman 2026; HelpNetSecurity 2026). CipherForce is their proprietary operation, targeting high-value victims directly through centralised TeamPCP management. Vect is a mass affiliate programme distributed through BreachForums, designed for high-volume distributed operations that are materially harder to track and attribute than centralised deployments. The 300GB trove feeds both simultaneously. Use on one track does not deplete availability for the other.

The scale of the affiliate distribution compounds the attribution problem. Approximately 300,000 BreachForums users (claimed forum membership, not independently verified) received Vect affiliate keys (Hartman 2026). The practical consequence: organisations cannot wait for a TeamPCP-attributed ransomware deployment before activating readiness protocols. The affiliate distribution has already occurred. Deployments that follow will not carry a consistent attribution signature that enables advance warning.

The trove itself originated from an estimated 500,000 infected machines, as reported by vx-underground and cited by SecurityWeek (SecurityWeek 2026). That figure is an aggregator estimate, not a confirmed primary research finding, and should be treated as directional rather than precise.

CVE-2026-33634 carries a CVSS score of 9.4 and appears on the CISA Known Exploited Vulnerabilities catalog, with a federal remediation deadline of April 8, 2026 (CISA 2026). Organisations not subject to federal compliance should still treat April 8 as a forcing function for ransomware readiness reviews. The deadline functions as an external benchmark to ensure the review process is formally assigned and resourced.

One active investigation warrants monitoring without overstating its current status. If confirmed, the Databricks investigation would represent the first major cloud platform identified as a downstream victim of credential exploitation from the trove, distinct from the tool vendors directly compromised in the supply chain phase. That distinction matters: downstream victims are being actioned from harvested credentials, not from a new compromise. As of March 31, the investigation is ongoing and unconfirmed.

If any credentials were in scope of an affected system during the March 19 to 27 window, rotation is the floor of the response, not the ceiling. Backup integrity verification, network segmentation validation, and incident response retainer activation should be reviewed now, against this threat timeline, not at the next scheduled programme review. These are not new programme elements; they are existing ones whose current state needs to be confirmed before a deployment event, not after.

Rotating your secrets addresses this incident. Re-architecting your pipeline secret scoping addresses the next one.

Both obligations are now active, and determining which applies to your environment requires answering two diagnostic questions your team can work through without new tooling.

Two Questions That Determine Your Actual Posture

First: did any system that ran Python between March 19 to 27, 2026, have access to credentials beyond its functional requirements? If yes, treat those credentials as compromised, regardless of whether that system installed a named affected package directly. The exposure criterion is credential scope and Python execution window, not package name.

Second: if a scanner in your current CI/CD pipeline were compromised tomorrow, through a tag hijack, a Docker image swap, or a GitHub Action hash drift, which secrets would it be able to reach? If the answer is "everything in the pipeline," the structural problem identified in Gap 1 is unresolved. Credential rotation from this incident does not change that answer. Per-step secret scoping does.

The supply chain attack surface is not "what packages do we use." It is "what can any tool in our build process reach?" Until secret scoping is per-step, the answer is: more than it should be.

What to Watch Over the Next 30 Days

Three developments will clarify the campaign's trajectory and, depending on how they resolve, may require you to accelerate your response.

First, monitor for public confirmation or denial of the Databricks investigation. If confirmed, it establishes the first major cloud platform as a downstream credential victim, distinct from the tool vendors directly compromised in the supply chain phase (HelpNetSecurity 2026). That would shift the scope of the credential trove's impact from a developer tooling problem to a cloud platform exposure problem, with different blast radius implications for any organisation using Databricks services.

Second, monitor for the first confirmed CipherForce ransomware deployment using TeamPCP-sourced credentials. None has been publicly reported as of March 31 (Hartman 2026). When the first deployment is confirmed, it will establish what TeamPCP considers a high-value target and give incident responders a more concrete profile for prioritising their own exposure risk.

Third, monitor for independent verification of the LAPSUS$/AstraZeneca data breach. While claimed by LAPSUS$ and attributed to TeamPCP operations via credential sharing, these reports remain publicly stated but unverified. Confirmation would establish a second major non-tooling victim and materially widen the picture of who is at risk from this trove.

Structurally, watch for new GitHub Actions tag hijacks across security-adjacent tooling. That is how this campaign started, and if PyPI's operational cost continues rising, the vector will shift before the campaign ends. The pause in new supply chain compromises is operational. The credential trove does not expire.

Special thanks to Coursera for supporting today’s content.

Brought to by the CyOps Consulting Team.

Key Takeaways

• This incident is less about a single poisoned package and more about a structural CI/CD failure. Security tools were given broad access to secrets they did not need, which let a compromised scanner reach publishing credentials and turn one pipeline into a single-pivot breach path.

• Hash pinning alone is not a reliable control against legitimate maintainer account compromise. The malicious artefacts carried correct registry hashes. The more reliable control is source-to-artifact comparison, particularly for high-risk packages and security tooling.

• Exposure is wider than direct installs. Audit transitive dependencies on developer workstations, CI runners, and container workloads. "Did we install this package?" is the wrong question when the package arrives through another tool's dependency graph.

• Rotating secrets is the floor, not the fix. The campaign has entered its monetisation phase. Organisations should treat this as both an incident response problem and a readiness problem, with immediate checks for Python execution during the exposure window, secret scope review, and ransomware preparedness validation.