apache/pinot — documented rules

user-supplied SQL, source-table reference, source partition expressions, rewriteEnabled, stalenessThresholdMs.

user-supplied SQL, source-table reference, source partition expressions, rewriteEnabled, stalenessThresholdMs.

watermarkMs plus a Map<Long, PartitionInfo> keyed by bucketStartMs.

The public entry point onBaseTableFullInvalidation(rawTableName) exists today (PR 1 shipped it for the time-MV "unknown range" case);

Interface MaterializedViewRuntimeMetadata, State today _partitions Map<Long, PartitionInfo>, Action needed when fixed-MV lands Generalize to Map<PartitionKey, PartitionInfo> (or Map<String, …> with a separate PartitionKind )

State today _watermarkMs long, Action needed when fixed-MV lands Make optional populated only for TIME

Interface MaterializedViewAnalyzer.validateSourceTable, State today Time-MV only, Action needed when fixed-MV lands Sibling validator for CATEGORICAL require partition column existence and (optional) frozen key list

Interface MaterializedViewAnalyzer.validateTaskConfigs, State today Hard-requires bucketTimePeriod, Action needed when fixed-MV lands Make required-on- TIME optional / absent on CATEGORICAL

Interface MaterializedViewTaskScheduler.generateTasks, State today Watermark + APPEND, Action needed when fixed-MV lands Branch on partitionKind CATEGORICAL goes through FIFO-STALE + refresh-interval logic

Interface MaterializedViewTaskExecutor.executeTask, State today Window-based, Action needed when fixed-MV lands Branch on task-config partitionKey shape segment building is unchanged

JSONMessageDecoder — a StreamMessageDecoder for real-time (streaming) ingestion.

JSONRecordReader — a RecordReader for batch ingestion of newline-delimited JSON files.

JSONRecordExtractor — the shared extractor that turns a parsed JSON object into a Pinot GenericRow.

DdlCompiler.compile(String sql) — single static entry point, returns a CompiledDdl discriminated by DdlOperation.

The IR is what populates the Schema (via toFieldSpec) and the TableConfig (via PropertyMapping.apply).

DataTypeMapper.resolve(String) is case-insensitive (Locale.ROOT) and accepts both ANSI-SQL and Pinot-native names:

The compiler runs dataType.convert(literalString) and surfaces a DdlCompilationException mapped to HTTP 400 if it fails.

validateTableConfig is then called against the **stored** schema (when one exists), not the compiled schema, so upsert/dedup PK validation sees the canonical PK list rather than the synthesized empty list from the DDL column-list-only projection.

TableConfigTunerUtils.applyTunerConfigs runs **before** validation, mirroring POST /tables.

CanonicalDdlEmitter.emit(schema, tableConfig, databaseName?) produces a deterministic DDL string.

PropertyExtractor calls PropertyMapping.isReservedRoundTripKey(lowerKey) on every custom-config entry it encounters.

The SPI invariant TableConfigUtils.validateMaterializedViewInvariants keeps the flag and the task.MaterializedViewTask.definedSQL body consistent, so the dispatch never has to inspect the task block to decide identity.

MaterializedViewPropertyRouter.periodToCron rewrites every accepted REFRESH EVERY '<N><unit>' value into a Quartz cron expression that the scheduler ingests.

The reverse direction (cronToPeriod) must be the **exact inverse**:

Adding a knob to MaterializedViewPropertyRouter.TASK_CONFIG_KEYS (the forward-router map; the reverse extractor consults it via canonicalKnobName(...)) is the **only** edit needed to promote a future knob to bare-DDL form.

The compiler stores the user's definedSQL as the **verbatim substring** of the original DDL covered by the AS <query> parser node (see DdlCompiler.extractDefinedSql).

The check lives in the controller (PinotDdlRestletResource.executeShowCreate / executeShowCreateMaterializedView) and delegates to the canonical TableConfig#isMaterializedView flag — the same predicate the emitter dispatches on — so a config that the emitter would have routed through the MV branch is caught before any rendering happens.

The check lives in the controller (PinotDdlRestletResource.executeShowCreate / executeShowCreateMaterializedView) and delegates to the canonical TableConfig#isMaterializedView flag — the same predicate the emitter dispatches on — so a config that the emitter would have routed through the MV branch is caught before any rendering happens.

**Compile.** DdlCompiler.compile(sql) → CompiledDdl.

**MV-specific compile route.** DdlCompiler.compileCreateMaterializedView routes REFRESH EVERY '<period>', AS <query>, and MV-specific knobs through MaterializedViewPropertyRouter before invoking the shared validateTableConfig (see §4 / §5.2).

**MV-specific compile route.** DdlCompiler.compileCreateMaterializedView routes REFRESH EVERY '<period>', AS <query>, and MV-specific knobs through MaterializedViewPropertyRouter before invoking the shared validateTableConfig (see §4 / §5.2).

Both §6.4 and §6.7 share the same canonical TableConfig#isMaterializedView flag the emitter dispatches on, so the controller never asks the emitter to render a form that disagrees with the underlying TableConfig shape.

Cleanup is delegated entirely to PinotHelixResourceManager#deleteTable for the same reason PR4's SHOW CREATE was deliberately small: that path is already the single source of truth for "drop a Pinot table, including any MV metadata it carries" — adding a parallel cleanup in the controller would risk drift between the two doors.

**No ZK schema change.** DdlCompiler returns a stock (Schema, TableConfig) pair and persists them through the existing PinotHelixResourceManager paths.

**No ZK schema change.** DdlCompiler returns a stock (Schema, TableConfig) pair and persists them through the existing PinotHelixResourceManager paths.

ZK writes (addSchema, addTable, deleteTable) go through the existing PinotHelixResourceManager paths — no new ZK write surface introduced.

ZK writes (addSchema, addTable, deleteTable) go through the existing PinotHelixResourceManager paths — no new ZK write surface introduced.

Sub-package compile/, Responsibility Forward path AST → (Schema, TableConfig) Entry point DdlCompiler

Sub-package reverse/, Responsibility Reverse path stored (Schema, TableConfig) → canonical DDL string Entry point CanonicalDdlEmitter