Web Archive File Exposure Scanner

Web applications continuously generate and expose file resources as part of their operation, including documents, configuration files, database exports, build artefacts, and backup files. Over time these resources are indexed by public web archive services such as the Wayback Machine, Common Crawl, AlienVault OTX, and URLScan, which preserve snapshots of URLs that were once reachable from the internet. Security teams and researchers use aggregation tools like GetAllURLs (gau) to query these archives and reconstruct the historical URL surface of a target domain. Organizations may be unaware that sensitive files uploaded or temporarily exposed during development or incident response have been captured by these archives. The indexed URLs remain searchable long after the original content has been removed from the server, giving attackers a reliable roadmap to probe for residual exposure. Understanding the historical file surface of a web application is an essential part of attack surface management and pre-engagement reconnaissance.

Historical file URL detection refers to the automated identification of file-bearing URLs that have at some point been publicly reachable on a target domain and captured by internet archive services. These URLs may point to documents, spreadsheets, archives, database dumps, configuration files, source code, log files, credential stores, or cryptographic key material. Even if the files have since been removed from the server, probing the original URLs can reveal whether the content is still accessible due to incomplete cleanup, caching, or infrastructure changes. The range of file types tracked extends from benign documents to highly sensitive artefacts such as .env files, .pem certificates, SQL dumps, and .htpasswd files. The presence of such URLs in public archives means that any party can discover and probe them without prior knowledge of the application's internal structure. Organizations that have undergone migrations, decommissioning events, or emergency deployments are particularly susceptible to residual file exposure detectable through this method.

The scanner invokes the gau binary with five worker threads and two retries, querying all major public archive providers for historically known URLs associated with the target domain. A blacklist of common non-file extensions is passed to gau at invocation time to reduce noise from media, font, stylesheet, and JavaScript URLs before they reach the Python layer. Each returned URL is checked against an extension registry covering over eighty file types grouped into categories including Document, Spreadsheet, Archive, Database, Backup, Config, Credential, Source, Log, Dev-Artefact, and Executable. Matched URLs are deduplicated and sorted by sensitivity priority, with Credential, Database, Backup, and Config categories ranked highest. Up to three hundred matched URLs are retained, and the first fifty are probed via HTTP HEAD requests with a five-second timeout to determine current liveness, recording the HTTP response status for each. Results are emitted as category, extension, HTTP status, and URL tuples, ordered so that the highest-risk file types appear first.

Discovery of live credential files such as .env, .pem, .htpasswd, netrc, or private key files gives attackers direct access to service credentials, API keys, database passwords, and cryptographic material that can be used to compromise cloud accounts, production databases, and third-party integrations. Accessible database dump files containing user records, session tokens, or payment data can result in large-scale data breaches with regulatory and reputational consequences. Exposed backup archives may contain a full copy of the application's source code, configuration, and data, enabling an attacker to conduct thorough offline analysis and identify further vulnerabilities before launching targeted attacks. Source code files reveal internal logic, hardcoded secrets, endpoint paths, and authentication flows that are not visible from the application's public interface. Configuration and infrastructure files such as YAML, TOML, or Dockerfile artefacts disclose deployment architecture, internal service addresses, and environment-specific settings that assist lateral movement. Log files may contain authentication tokens, session identifiers, stack traces with internal paths, and personally identifiable information that should never be publicly accessible.