SRM-Dataset

A cross-language dataset of security-relevant methods (SRMs) which are library and framework functions labelled as taint-analysis sources, sinks, sanitizers, or propagators, with CWE identifiers and data-flow properties.

The dataset serves two purposes: it can be loaded directly to configure taint-analysis tools, and it provides labelled training and evaluation data for machine-learning approaches that detect SRMs (ML4SRM). SRM-Dataset records per-method provenance (discovery), includes negative examples (methods reviewed and judged not security-relevant), and ships the metadata needed to re-extract code-level features (method body, documentation, enclosing artefact, usages).

The Java dataset contains 10,180 methods aggregated from find-sec-bugs, SSCM, SWAN, TheCodeMaster, TaintBench, CoDoC, SecuCheck and the OWASP Benchmark. Eight further datasets cover Android, Python, C, C#, JavaScript, PHP and Ruby. The datasets by languages are summarized below:

Language / scope	Entries	Dataset
Java (main dataset)	10,180	datasets/java/srm-dataset-java.json
Android Java	13,726	datasets/java/srm-dataset-android.json
C# / VB.NET	595	datasets/c#/srm-dataset-c#.json
JavaScript	347	datasets/js/srm-dataset-js.json
PHP	232	datasets/php/srm-dataset-php.json
C	212	datasets/c/srm-dataset-c.json
Python (Python-level APIs)	103	datasets/python/srm-dataset-python.json
CPython audit events	192	datasets/python/srm-dataset-Cpython.json
Ruby	55	datasets/ruby/srm-dataset-ruby.json

Repository layout

datasets/<language>/     final + intermediate datasets, and a README documenting
                         exactly which sources and scripts produced them
datasets/scripts/        the shared Python extraction/merge/dedup/conversion pipeline
parser/                  Maven module: read-only Java API to load and filter the
                         dataset by SRM role / CWE (+ the schema reference)

Schema

Top level is { version, methods: [...] }. Every field is documented in parser/README.md. datasets/scripts/srm-dataset-maker.py holds the canonical template for a new entry.

{
  "name": "java.io.FileWriter.<init>",
  "signature": "void java.io.FileWriter.<init>()",
  "parameters": [],
  "discovery": "find-sec-bugs",
  "dataIn":  { "parameters": [], "return": false },
  "dataOut": { "parameters": [], "return": false },
  "srm": ["sink"],
  "cwe": ["CWE35"],
  "known": true,
  "artifacts": { "identifier": "", "sources": "", "compiled": "java-rt-jar-stubs-1.5.0.jar" }
}

Conventions worth knowing before writing a consumer:

• Constructors use <init> in name and signature.
• dataIn/dataOut parameter references may be zero-based indices or parameter names.
• Metadata that was never recovered is an empty string, empty array, or null. An empty srm array means "not labelled security-relevant", which is a negative example, not a missing label.
• known: false marks an entry whose label is unvalidated or partially blank.

Using the dataset

From Java (parser/, Java 17, Jackson + Soot):

cd parser && mvn package     # then use Parser.filterSrm / filterSrmWithAndOp

From anything else: the JSON is self-contained. The files are large (up to ~17 MB), so prefer streaming or grep over loading a whole dataset when you only need a slice.

Reproducing the datasets

Each datasets/<language>/README.md is the authoritative record of the sources used and the scripts run for that language. The general pipeline (datasets/scripts/, Python 3 + pandas):

1. Extract candidates from a source: semgrep_extractor.py (Semgrep rules), extract_from_banned.py (C banned.h), extract_ruby_srms_from_codeql.py, enrich_c#_catalog.py, gxa_labelling_to_agreed_csv.py + gxa_csv_to_srm_dataset.py.
2. Merge sources: merge_sources.py.
3. Dedup by fully qualified name and signature: check_for_dupes.py, check_for_dubs_and_delete_from_txt_files.py (also checks against the existing dataset).
4. Convert TXT → short JSON → final schema: make_short_json.py, then srm-dataset-maker.py.
5. Report: dataset-count.py (role/CWE summary), owasp_benchmark_labelling_stats.py and owasp_benchmark_kappa.py (labelling statistics and inter-rater agreement; the two compute deliberately different κ, see the OWASP-Benchmark README).

Rule/config-derived sources (SAST rules in particular) give identifiers but rarely signatures, return types, or which parameter is attacker-controlled. Those fields were completed by manual review with LLM assistance; the Java dataset was additionally enriched from real artefacts with method-extractor and JimpleProvider.

The Java Maven modules are independent projects (no parent POM) and several mains carry hardcoded local paths and toggle flags. They are scripts to edit for the task at hand, not as configurable CLIs. Build each from inside its own directory.

Limitations

• Label reliability. Two authors independently labelled the 160 methods extracted from the OWASP Benchmark; chance-corrected inter-rater agreement is Cohen's κ = 0.19 (slight). Disagreements were resolved by consensus. See datasets/java/sources/owasp-benchmark/README.md for the full breakdown, per-role κ, and the open discrepancies between the consensus sheet and the shipped datasets. The GXA labelling is reported the same way with per-label κ 0.60/0.73/0.92 with directional rather than noisy disagreement in datasets/java/sources/gxa/README.md.
• Multi-CWE methods. A method may legitimately map to several CWEs. The multi-label representation records all of them, while single-label per-CWE representations use the most specific applicable CWE. This is a construction decision, not ground truth.
• Uneven metadata depth. Only Java entries are enriched with bodies, documentation and usages. Other languages carry signature-level information only.