sparkrules is purpose-built for distributed rule evaluation at scale. This page compares capabilities, throughput, and cost against other systems in the same problem space.
Distributed rule evaluation is a different problem from single-JVM scoring. Traditional engines (Drools, Easy Rules, IBM ODM, FICO Blaze) were designed for request/response latency. sparkrules was designed for the lakehouse.
Rules compile to Spark SQL Column expressions, then to Catalyst bytecode via WholeStageCodegen. Same codepath as a native join. Drools-on-Spark serializes every fact through a Python/Java UDF - 50-100x slower.
Stage-fused plans have zero cross-partition data movement. Throughput doubles when workers double. Single-JVM engines max out at one box; clustering Drools requires manual sharding and loses 30-40% to coordination.
Columnar Parquet reads, vectorized predicate evaluation, single-pass aggregation for fire counts. RETE-based engines hold working memory per fact - unsustainable past ~10M facts. sparkrules streams through partitions.
Numbers for commercial engines are industry-public single-JVM benchmarks. sparkrules numbers are measured (10-worker baseline) and linearly projected (25/50/100 workers). Drools-on-Spark is an antipattern we measured internally - do not use it in production.
sparkrules ships 13 built-in modules that commercial engines sell as paid add-ons. All Apache 2.0. All auditable. All documented.
sparkrules.compliance)Helpers for adverse-action reason aggregation, counterfactual summaries, and counsel-reviewable templates aligned to ECOA/FCRA/GDPR Art 22 structure. Not a legal attestation - review with compliance counsel before production use.
sparkrules.dq)ExpectNotNull, ExpectBetween, ExpectInSet rules with INFO/WARN/ERROR/CRITICAL severity. Runs inline with scoring.
sparkrules.dmn)Minimal Camunda-style decision table parser, evaluator, and counterfactual explainer. Subset of DMN 1.3 - full FEEL, DRD, and DMN-TCK conformance are on the roadmap, not shipped today.
sparkrules.governance)Version pinning per environment, adjacent-only promotion (dev→stage→prod), deprecation warnings, promotion audit log.
sparkrules.sim)Deterministic bucketing by key, shadow mode, replay against historical facts. Compare two rulepacks side by side before promoting.
sparkrules.export)Export any rulepack to OPA Rego policy. Security teams verify with their existing OPA toolchain.
sparkrules.runtime.streaming)Structured Streaming + Kafka source + Iceberg sink. Same rule pack works for batch and streaming with zero code changes.
sparkrules.runtime.lineage)Every fire tracked to the exact DRL version, input fact, and output decision. Regulators trace any decision end-to-end.
sparkrules.integrations)First-class Feast and Tecton clients. Enrich facts with online/offline features before scoring.
sparkrules.policy)Apache Ranger and OPA client adapters. Rules can enforce column masking and row-level filters per tenant.
sparkrules.store)Pluggable metadata backends: in-memory, DuckDB, Postgres, and Iceberg-hydrating with pyiceberg append sink or pickle-on-disk fallback. Full Iceberg time-travel on rule versions depends on the sink you wire up.
sparkrules.ai)OpenAI-backed rule draft service. Describe a policy in plain English, get a reviewable DRL back.
sparkrules.obs)Prometheus metrics, structured logging, health probes. Native to any K8s or Glue deployment.
These capabilities exist in sparkrules v1.2.0 today - not on a roadmap. Most commercial rule engines charge for a subset of these as paid add-ons. sparkrules ships them all under Apache 2.0.
"If feature X had been Y, the decision would have been Z." Exposed via DMN evaluator and adverse-action summaries. Useful for FCRA and EU AI Act workflows; final legal sufficiency is counsel's call.
sparkrules.dmn.counterfactual_dmn_decision_table_xml()
Rules that reference group-level aggregates (sum, count, avg by region/customer/day). First pass aggregates, second pass scores. No other Spark rule engine supports this.
sparkrules.runtime.two_pass
Inject failures at specific attempt numbers, timeouts, and retry budgets. Verify rule evaluation stays deterministic under adverse conditions. Part of the standard test suite.
sparkrules.runtime.chaos.run_chaos_scenario()
Predict infra cost before you run. Works for Databricks, Glue, Dataproc, Synapse. Factors in rows, rule count, cluster type. Capacity planning without running a benchmark first.
estimate_batch_cost_usd(rows=1e9, rules=30, cluster="glue")
Language Server Protocol implementation for DRL. Autocomplete, diagnostics, go-to-definition in VS Code, Cursor, any LSP-aware editor. Ships with the package.
sparkrules.ide.lsp.LspAnalysis
Import XLSX decision tables authored by business analysts, export them back with rule updates. Full cell-type validation. UNIQUE, FIRST, PRIORITY, COLLECT hit policies.
sparkrules.ioxls.DecisionTableImporter / Exporter
CI-ready drift detector. Run v1 and v2 side-by-side, fail the build if outputs disagree. Catches silent regressions before they reach prod.
sparkrules.tools.shadow_parity_summary()
FastAPI-based management plane: list, parse, simulate, deploy, deprecate rules via HTTP. Plus a Spark Connect server for remote cluster access. No custom protocol.
sparkrules.api.app · sparkrules.connect.server
Opinion: never pickle user code. sparkrules serializes rulepacks with a schema-validated, allow-listed format. Protects Glue/Databricks clusters from deserialization attacks.
sparkrules.compiler.safe_pickle
Dual-evaluation for shadow releases, rulepack size guards, engine metrics gating. All env-var controlled so ops can tune without code changes.
SPARKRULES_SHADOW_DUAL_EVAL · SPARKRULES_MAX_RULEPACK_BYTES
Describe a policy in plain English, get reviewable DRL. OpenAI-backed but provider-pluggable. Draft-and-review workflow, no auto-deploy.
sparkrules.ai.service · sparkrules.ai.openai_provider
SHA-256 keyed variant assignment. Same user always gets the same variant. Weight-based splits. Usable for rule canary rollouts without adding LaunchDarkly.
sparkrules.sim.ab.ABTestRunner
A team adopting sparkrules gets a rule engine + decision platform + governance layer + simulation suite + compliance toolkit in a single pip install.
Most commercial stacks require 3-5 separate products to match this surface area.
Every sparkrules rule compiles down to the same codepath Spark uses for native joins and aggregates. No Python UDF boundary. No per-row serialization.
Spark's WholeStageCodegen fuses every rule predicate into a single generated Java method. One JIT compile, one pass through the data.
Rules execute as JVM bytecode inside the executor, not as Python UDFs. No Arrow serialization, no cross-process hops, no GIL.
Your rules ride Spark's most optimized codepath. The same one used for Parquet scans, shuffle-less joins, and window aggregates.
df.explain(mode="codegen") and Spark UI SQL tab · 79,479,946 rows × 30 rules × 10 × G.1Xsparkrules compiles DRL rules to Spark SQL and runs them as JVM Catalyst bytecode. All 30 rule predicates fuse into one WholeStageCodegen Project stage per partition - the same codepath Spark uses for native joins.
Daily ETL pipelines scoring billions of records. Run on Glue overnight for a few dollars.
Structured Streaming + Kafka + Iceberg. Same rule pack for streaming and batch.
Low-latency single-fact scoring with pluggable native extension point (experimental Rust starter crate available).
sparkrules ships a full rule platform - not just a scorer. DRL parser, DMN decision tables, data quality engine, compliance artifacts, governance registry, A/B simulation, OPA export, lineage, Iceberg store. All Apache 2.0.
| Capability | sparkrules | Drools | Easy Rules | OpenL Tablets | Camunda DMN | IBM ODM | FICO Blaze |
|---|---|---|---|---|---|---|---|
| Distributed scale-out (Spark-native) | ✅ | — | — | — | — | limited | — |
| Drools DRL parser | ✅ | ✅ | — | — | — | proprietary | proprietary |
| DMN (Camunda-style subset) | ✅ | ✅ | — | partial | ✅ | ✅ | proprietary |
| Data quality engine (built-in) | ✅ | — | — | — | — | add-on | add-on |
| Adverse-action helpers (ECOA/FCRA/GDPR Art 22 structure) | ✅ | — | — | — | — | add-on | ✅ |
| Governance registry (dev/stage/prod pins) | ✅ | manual | — | manual | ✅ | ✅ | ✅ |
| A/B simulator + shadow/replay | ✅ | — | — | — | — | add-on | add-on |
| OPA Rego policy export | ✅ | — | — | — | — | — | — |
| Streaming (Kafka + Iceberg) | ✅ | manual | — | — | CEP addon | add-on | — |
| Lineage + audit trail | ✅ | partial | — | partial | ✅ | ✅ | ✅ |
| Feature store integration (Feast / Tecton) | ✅ | — | — | — | — | custom | custom |
| Ranger-compatible access policies | ✅ | — | — | — | — | — | — |
| Rust starter crate (experimental) | ✅ | — | — | — | — | — | — |
| License model | Apache 2.0 | Apache 2.0 | MIT | LGPL | Apache 2.0 | Commercial | Commercial |
| Measured 1B rows wallclock | ~5 min * | ~45 min † | n/a | n/a | ~60 min † | ~30 min † | ~35 min † |
* Projected from measured 79.5M × 30 rules baseline on 10 × G.1X (Glue 5.0, us-east-1). † Industry-public numbers for single-JVM or small-cluster runs; none of these engines scale linearly on Spark the way sparkrules does. Single-JVM tools (Drools, Easy Rules) excel at low-latency single-fact scoring and are not built for billion-row batch.
Fetched live from PyPI and GitHub on page render. Downloads, releases, contributor activity - all public, all verifiable.
Source: pypistats.org · cached 24h by PyPI
Six industries, the sparkrules modules that matter most for each, and the measured capacity headroom against their peak load.
What does it cost to score a billion rows per day? Only infrastructure matters - no license fees, no per-rule charges, no per-fact pricing. Every row in the table below scores the same 1,000,000,000 rows against the same 12 DRL rules.
| Configuration | Data volume | Rules | Wallclock | AWS Compute | Annual License | Notes |
|---|---|---|---|---|---|---|
| sparkrules on 25 × G.1X | 1 B rows | 12 | ~1.5 min | $0.28 | $0 | Apache 2.0 · recommended config |
| sparkrules on 50 × G.2X | 1 B rows | 12 | ~30 sec | $0.86 | $0 | Near-realtime 1B scoring |
| sparkrules on 100 × G.2X | 1 B rows | 12 | ~15 sec | $1.72 | $0 | Peak throughput config |
| sparkrules on 10 × G.1X measured | 79.5 M rows | 30 | 24.25 sec | $0.17 | $0 | Actual Glue run · $0 license |
| Commercial enterprise engine A | 1 B rows | 12 | ~30+ min | $50-$500 | $100K-$1M | Per-seat + per-core licensing |
| Hand-written Spark SQL | 1 B rows | 12 | ~1 min | $0.20 | $0 | No DRL syntax, no audit trail |
Measured row (10 × G.1X · 79.5M rows · 30 rules) is the authoritative baseline from AWS Glue 5.0 run jr_428c3052.... Other sparkrules rows are linear extrapolations; commercial engine row is industry-public pricing.
Terraform module included
Works on any 3.5+ cluster
Kubernetes-native
Spark pools supported
Spark 3.5 images
Streaming source
Lakehouse sink
Post-scoring models
Pip install and write DRL rules. No Scala, no Java classpath, no KieContainer setup. First rule fires in under 5 minutes.
Linear scaling means cost-per-row stays flat as you add workers. 1B rows costs the same whether you use 10 or 100 workers.
Every fire has a rule name, salience, reason codes, and deterministic output. Regulators can trace any decision back to the DRL that produced it.
AWS, Azure, GCP, Databricks, on-prem Spark. Change one Terraform variable to switch regions or clouds.
Reproducible benchmark published in this repo. $0.45 of AWS spend verifies the numbers on your own account.
Forever free, no EULA negotiation, no per-CPU licensing, no audit by vendors. Ship it in commercial products.
Honest tracker of the state of every claim on this page. No feature is stronger than its upstream documentation. Sourced from docs/KNOWN_LIMITATIONS.md and docs/PENDING_NON_BLOCKING.md.
sparkrules.tools.stream_kafka_iceberg); runnable Kafka-Spark-Iceberg topology is ops-wiredsparkrules_native) - starter PyO3 crate, Tier-1 JSON FFI at ~1.1-1.3x over Python, not on PyPIECOA, FCRA, Reg B, SR 11-7, HIPAA, 21 CFR Part 11, GDPR Article 22, EU AI Act references on this site describe how sparkrules modules are structured to support those regimes. Nothing here is legal advice, certification, or a compliance attestation. Final sufficiency in a regulated deployment requires review by qualified counsel. The project is licensed Apache 2.0 with the standard no-warranty clause.
One measured baseline: 79.5M rows × 30 rules × 10 × G.1X = 24.25s scoring on Glue 5.0 us-east-1 (run jr_428c3052...). Every larger number on this site (25 / 50 / 100 / 400 workers, 1B / 10B / 100B rows) is projected linearly from that baseline. Projections are labeled where they appear. A 200-node run with published methodology is on the roadmap.
Install, write rules, measure. 25 minutes end-to-end on AWS for under $1.