Measured Performance

Real numbers from real AWS

Every metric here comes from the Glue JobRun API and the benchmark summary JSON. Zero projections, zero marketing math.

3.28M

Rows per second

on 10 × G.1X Glue workers

79.5M

Rows scored

24 months NYC TLC yellow taxi

$0.17

Cost per run

1,394 DPU-seconds @ $0.44/DPU-hr

24.25s

Scoring wallclock

apply_with_counts single pass

Built for Scale

Engineered for billion-row workloads

Strategy A compiles your DRL rules to Spark SQL and runs them as JVM Catalyst bytecode via WholeStageCodegen - the same codepath Spark uses for native joins.

Linear Horizontal Scale

Add workers, get proportional throughput. Measured 3.28M rows/sec on 10 workers scales to ~32M rows/sec on 100 workers.

Drools DRL Syntax

Drop-in DRL parser. Same syntax your auditors and regulators already know. Works with existing Drools rule packs.

Auditable & GxP Ready

Every rule fire has a reason code, salience, and deterministic output. Strategy classification proves which rules run as SQL vs Python.

Cloud-Native

AWS Glue, Databricks, EMR, Azure Synapse, GCP Dataproc. Any Spark 3.5+. Terraform module included for Glue.

Streaming + Batch

Same rule pack for Kafka streaming and daily batch. Structured Streaming compatible. Iceberg sink for lakehouse integration.

Experimental Rust Scorer

Starter PyO3 crate at sparkrules_native. Not yet on PyPI. Current Tier-1 JSON-FFI interpreter measures ~1.1-1.3x vs Python local scorer. Active area of contribution.

1B

rows in ~1 minute

On 25 × G.1X workers at $0.28. Projected from measured 10-worker baseline.

30

rules fused to one Project

Spark's Catalyst optimizer fuses all 30 rule predicates into one WholeStageCodegen stage per partition.

$0

license fee

Apache 2.0 open source. No per-DPU, per-rule, per-fact, or per-seat pricing.

Architecture

DRL goes in, Spark SQL comes out

Strategy A classifier identifies rules that translate cleanly to Spark SQL and compiles them to Catalyst expressions. These rules run as native JVM bytecode on Spark executors.

INPUT

DRL rules

Drools syntax

→

COMPILE

Strategy Classifier

SQL_PUSHDOWN | ALPHA_SHARED | FALLBACK

→

OUTPUT

Catalyst

WholeStageCodegen

Catalyst plan 30 rules, measured on Glue 5.0

Project [t, r_r01_fare_zero_or_negative, r_r06_tip_negative, ..., r_r30_group_ride]
  +- Project [
       ((t.fare_amount <= 0.0) <=> true) AS r_r01_fare_zero_or_negative,
       ((t.tip_amount < 0.0) <=> true)   AS r_r06_tip_negative,
       ... 28 more rule predicates ...
     ]
    +- Repartition 80
      +- Relation [VendorID, tpep_pickup_datetime, ...] parquet

Cluster Sizing

From 1 worker to 400 DPUs

Linear scaling with Spark. Per-DPU pricing is identical across G.1X through G.8X - pick a worker type based on memory needs, not price.

Throughput vs Cluster Size

Cost per 1 Billion Rows

View full benchmark report →

Production Workloads

Built for demanding use cases

FINANCE

Fraud detection

Score every transaction against hundreds of rules. 500k TPS sustained on 2 G.1X workers. Real-time or batch.

RETAIL

Ad bid scoring

Evaluate bid requests against targeting rules. 10M+ QPS on 50 G.2X workers. Same rule pack runs in Kafka streaming.

HEALTHCARE

Claims adjudication

Every fire has a reason code and rule ID. Deterministic output across runs - a foundation for GxP audit trails, not a shipped GxP attestation.

TELCO

CDR enrichment

Verizon-scale CDR streams at 250k events/sec. One G.1X handles it with 13x headroom.

INSURANCE

Policy underwriting

Complex multi-fact rule packs with salience and action merging. Hot-swap rules without restart.

DATA QUALITY

ETL validation

Rule-based data quality at lakehouse scale. Billions of rows per day in daily batch windows.

Examples and Tooling

25+ runnable examples · built-in Workbench UI

sparkrules ships working code for every feature - DRL rule files, Python scripts, Spark jobs, Jupyter notebooks. Plus a browser-based Workbench at /workbench/ that lets you author, validate, simulate, and promote rules without writing glue code.

📝 5 files

DRL rule files

Minimal, discount tier, credit underwriting, fraud detection, insurance claims. Sample facts included.

🐍 20+ scripts

Python demos

Adverse action, DMN, DQ profiling, OPA export, AI authoring, chaos harness, two-pass orchestrator.

⚡ 3 jobs

Spark jobs

V2 SparkRuleExecutor, lakehouse sinks, pure-Python row iterator. With --explain Catalyst printout.

📓 7 notebooks

Jupyter notebooks

Getting Started, Decision Tables, API & Simulation, Credit Underwriting, Fraud, Architecture, Databricks.

Recommended learning path

01

Getting Started

DRL rules, evaluation, explainable results

02

Decision Tables

Hit policies, JSON export

04

Credit Underwriting

Activation groups, ECOA/FCRA notices

05

Fraud Detection

Velocity checks, OPA Rego export

03

API & Simulation

REST validation, counterfactuals, LSP

07

Databricks production

Pinning, Delta/Iceberg scoring

SHIPPED · Not a roadmap item

Browser Workbench, already in the package

Start the API server and open /workbench/ - you get a full rule authoring IDE in the browser. No separate install, no third-party UI, no license key.

✓ Monaco DRL editor with syntax highlighting

✓ Namespace picker (multi-tenant)

✓ Inline test with fact JSON

✓ LSP diagnostics + completions

✓ Rule pack export / drag-drop import

✓ Simulate endpoint with counterfactuals

✓ Cost estimator tab

✓ Lineage table + Copy-as-curl

✓ Light/dark theme

✓ Toasts for 4xx/5xx errors

Workbench feature list Login + auth setup

Launch in 2 commands

# 1. Install with API extras
pip install sparkrules[api]

# 2. Start the server
python -m uvicorn \
  sparkrules.api.app:create_app \
  --factory --port 8042

# 3. Open in browser
http://localhost:8042/workbench/

Default dev creds: admin / admin - override via env vars for prod.

Get Started

One pip install away

Available on PyPI today. Works on any Spark 3.5+ cluster. Terraform module for Glue included.

Python engine

pip install sparkrules

Default Python engine. Works everywhere. 4.2k rows/sec local, millions/sec on Spark.

Experimental Rust scorer

# Not on PyPI yet. Build locally: cd sparkrules_native && maturin develop --release

Tier-1 PyO3 scorer with JSON FFI. Measured ~1.1-1.3x over Python local scorer. Not wired to Spark executors. See docs/NATIVE_TIER1.md.

See full benchmark

The distributed rule engine for Spark