Docs & rules

apache/stormGitHub

This is a public, read-only report. Striff reads this repository's docs, turns each sentence that makes a claim about the code into a rule, and checks the rule against the code on the default branch. How this works

master3e933c5Last read Oct 20

Is this yours? Install to manage it

Once installed, Striff checks every pull request.

Rules summary
Documents
Rules
160
139
5
1
1
13
10
4
7
21
3
5
6
7
15
1
17
17
3
3
17
17
17
4
26 docs
apache/stormOpen repository

apache/storm — documented rules

160 of 160 rules, printed November 17, 2026.

The sentence in your docs
workers > write them to local disk, supervisors relay them to Nimbus over Thrift, and Nimbus > keeps them in an in-memory heartbeat cache.
Nimbus depends on HeartbeatCache
Holds
All cluster state writes go through Utils.serialize(...) / Utils.deserialize(...), which in turn delegate to a pluggable SerializationDelegate selected by the storm.meta.serialization.delegate config.
Utils depends on SerializationDelegate
Holdsutils serialization
Earlier versions used GzipThriftSerializationDelegate, wrapped by GzipBridgeThriftSerializationDelegate to allow rolling upgrades from clusters that had previously stored raw Thrift bytes;
GzipBridgeThriftSerializationDelegate depends on GzipThriftSerializationDelegate
Holds
Time.currentTimeSecs() and Time.deltaSecs(int) are deprecated but retained for such relative-duration callers.
2 rules from this sentence
Time has a currentTimeSecs method
checks “Time.currentTimeSecs()”
Holds
Time has a deltaSecs method
checks “Time.deltaSecs(int)”
Holds
The concrete implementation of this interface is the “NimbusBlobStore”.
NimbusBlobStore is a concrete implementation
Holds
The config section is simply a map of Storm topology configuration parameters that will be passed to the org.apache.storm.StormSubmitter as an instance of the org.apache.storm.Config class:
StormSubmitter depends on Config
Holds
To allow users to define custom metrics, the following methods have been added to the TopologyContext class, an instance of which is passed to spout's open() method and bolt's prepare() method: public Timer registerTimer(String name)
TopologyContext uses com.codahale.metrics.Timer
Holds
To allow users to define custom metrics, the following methods have been added to the TopologyContext class, an instance of which is passed to spout's open() method and bolt's prepare() method:
4 rules from this sentence
TopologyContext uses com.codahale.metrics.Histogram
Holds
TopologyContext uses com.codahale.metrics.Meter
Holds
TopologyContext uses com.codahale.metrics.Counter
Holds
TopologyContext uses com.codahale.metrics.Gauge
Holds
public Timer registerTimer(String name)
TopologyContext has a registerTimer method
Holds
public Histogram registerHistogram(String name)
TopologyContext has a registerHistogram method
Holds
public Meter registerMeter(String name)
TopologyContext has a registerMeter method
Holds
public Counter registerCounter(String name)
TopologyContext has a registerCounter method
Holds
public Gauge registerGauge(String name, Gauge gauge)
TopologyContext has a registerGauge method
Holds
Prometheus Reporter (org.apache.storm.metrics.prometheus.PrometheusStormReporter, in the storm-metrics-prometheus module): Serves the metrics of each worker over HTTP for Prometheus to scrape, with the task dimensions as labels.
PrometheusStormReporter is in external/storm-metrics-prometheus
Holds
Custom metrics reporters can be created by implementing org.apache.storm.metrics2.reporters.StormReporter interface or extending org.apache.storm.metrics2.reporters.ScheduledStormReporter class.
StormReporter is a contract
Holds
Custom filters can be created by implementing the org.apache.storm.metrics2.filters.StormMetricsFilter interface:
StormMetricsFilter is a contract
Holds
Metrics have to implement IMetric which contains just one method, getValueAndReset -- do any remaining work to find the summary value, and reset back to an initial state.
IMetric has a getValueAndReset method
HoldsPR #9093 · Sep 30
MeanReducer -- track a running average of values given to its reduce() method. (It accepts Double, Integer or Long values, and maintains the internal average as a Double.) Despite his reputation, the MeanReducer is actually a pretty nice guy in person.
MeanReducer has a reduce method
Holds
You can refer Config#registerMetricsConsumer and overloaded methods from javadoc.
Config has a registerMetricsConsumer method
Holds
Also, Storm exposes the interface IMetricsConsumer for implementing Metrics Consumer so you can create custom metrics consumers and attach to their topologies, or use other great implementation of Metrics Consumers provided by Storm community.
2 rules from this sentence
IMetricsConsumer is a contract
checks “Storm exposes the interface IMetricsConsumer”
HoldsPR #9093 · Sep 30
IMetricsConsumer is exported
Holds
The meaning of first and second parameters are straightforward, metric name and instance of IMetric. Third parameter of TopologyContext#registerMetric is the period (seconds) to publish and reset the metric.
TopologyContext depends on IMetric
Holdstask metric
Third parameter of TopologyContext#registerMetric is the period (seconds) to publish and reset the metric.
TopologyContext has registerMetric
Holds
You can register your own worker level metrics by adding them to Config.WORKER_METRICS for all workers in cluster, or Config.TOPOLOGY_WORKER_METRICS for all workers in specific topology.
2 rules from this sentence
Config has WORKER_METRICS
checks “adding them to Config.WORKER_METRICS”
Holds
Config has TOPOLOGY_WORKER_METRICS
checks “Config.TOPOLOGY_WORKER_METRICS”
Holds
C) Bucket size (seconds) for metrics is fixed to Config.TOPOLOGY_BUILTIN_METRICS_BUCKET_SIZE_SECS.
Config has TOPOLOGY_BUILTIN_METRICS_BUCKET_SIZE_SECS
Holds
Users can implement the IStrategy interface and define new strategies to schedule specific topologies.
IStrategy is a contract
Holds
For the owner to define his or her own prioritization strategy, she or he needs to implement the ISchedulingPriorityStrategy interface.
ISchedulingPriorityStrategy is a contract
Holds
A user can set the scheduling priority strategy by setting the DaemonConfig.RESOURCE_AWARE_SCHEDULER_PRIORITY_STRATEGY to point to the class that implements the strategy.
DaemonConfig has RESOURCE_AWARE_SCHEDULER_PRIORITY_STRATEGY
Holds
To try and be as fair as possible to users running short lived topologies the FIFOSchedulingPriorityStrategy extends the DefaultSchedulingPriorityStrategy so that any negative score (a.k.a. a topology that fits within a user's guarantees) would remain unchanged, but positive scores are replaced with the up-time of the topology.
FIFOSchedulingPriorityStrategy extends DefaultSchedulingPriorityStrategy
HoldsPR #9093 · Sep 30
If your filter authenticates in a different way you can look at setting ui.http.creds.plugin to point to an instance of IHttpCredentialsPlugin that can take the HttpServletRequest and return a user name and populate the needed fields in the current ReqContext.
IHttpCredentialsPlugin depends on ReqContext
Holdsstorm-server storm-client
If your filter authenticates in a different way you can look at setting ui.http.creds.plugin to point to an instance of IHttpCredentialsPlugin that can take the HttpServletRequest and return a user name and populate the needed fields in the current ReqContext. These are advanced features and you may want to look at the DefaultHttpCredentialsPlugin as an example of how to do this.
DefaultHttpCredentialsPlugin implements IHttpCredentialsPlugin
Holds
For example, if a userX submits an oozie workflow and as part of workflow execution if user oozie wants to submit a topology on behalf of userX it can do so by leveraging the impersonation feature.In order to submit topology as some other user , you can use StormSubmitter.submitTopologyAs API.
StormSubmitter has submitTopologyAs
Holds
Alternatively you can use NimbusClient.getConfiguredClientAs to get a nimbus client as some other user and perform any nimbus action(i.e. kill/rebalance/activate/deactivate) using this client.
NimbusClient has getConfiguredClientAs
Holds
To ensure only authorized users can perform impersonation you should start nimbus with nimbus.impersonation.authorizer set to org.apache.storm.security.auth.authorizer.ImpersonationAuthorizer.
ImpersonationAuthorizer is in org.apache.storm.security.auth.authorizer
Holds
topology.auto-credentials is a list of java plugins, all of which must implement the IAutoCredentials interface, that populate the credentials on gateway and unpack them on the worker side. On a kerberos secure cluster they should be set by default to point to org.apache.storm.security.auth.kerberos.AutoTGT
AutoTGT implements IAutoCredentials
Holdskerberos IAutoCredentials.java
All autocredential classes that desire to implement the IMetricsRegistrant interface can register metrics automatically for each topology. The AutoTGT class currently implements this interface and adds a metric named TGT-TimeToExpiryMsecs showing the remaining time until the TGT needs to be renewed.
AutoTGT implements IMetricsRegistrant
Holdssecurity metric
The new provided org.apache.storm.hdfs.format.SimpleFileNameFormat and org.apache.storm.hdfs.trident.format.SimpleFileNameFormat are more flexible, and the withName method support parameters as following:
SimpleFileNameFormat has a withName method
Holds
Both the HDFS bolt and Trident State implementation allow you to register any number of RotationActions.
2 rules from this sentence
HdfsState depends on RotationAction
Holdstrident common
HdfsBolt depends on RotationAction
Holdsbolt common
To partition your your data, write a class that implements the Partitioner interface and pass it to the withPartitioner() method of your bolt.
Partitioner is a contract
Holds
To partition your your data, write a class that implements the Partitioner interface and pass it to the withPartitioner() method of your bolt. The getPartitionPath() method returns a partition path for a given tuple.
Partitioner has a getPartitionPath method
Holds
The org.apache.storm.hdfs.bolt.SequenceFileBolt class allows you to write storm data to HDFS sequence files:
SequenceFileBolt is in org.apache.storm.hdfs.bolt
Holds
The SequenceFileBolt requires that you provide a org.apache.storm.hdfs.bolt.format.SequenceFormat that maps tuples to key/value pairs:
2 rules from this sentence
SequenceFileBolt depends on SequenceFormat
checks “The SequenceFileBolt requires that you provide a org.apache.storm.hdfs.bolt.format.SequenceFormat”
HoldsSequenceFileBolt.java format
SequenceFormat is in org.apache.storm.hdfs.bolt.format
checks “a org.apache.storm.hdfs.bolt.format.SequenceFormat”
Holds
AvroUtils.addAvroKryoSerializations(conf);
AvroUtils has an addAvroKryoSerializations method
Holds
Method .setReaderType(), Alternative config name (deprecated) ~~hdfsspout.reader.type~~, Description Determines which file reader to use Set to 'seq' for reading sequence files or 'text' for text files Set to a fully qualified class name if using a custom file reader class (that implements interface org.apache.storm.hdfs.spout.FileReader)
2 rules from this sentence
FileReader is a contract
checks “implements interface org.apache.storm.hdfs.spout.FileReader”
Holds
HdfsSpout has a setReaderType method
Holds
Method .withOutputFields(), Description Sets the names for the output fields for the spout The number of fields depends upon the reader being used For convenience, built-in reader types expose a static member called defaultFields that can be used for setting this
3 rules from this sentence
HdfsSpout has a withOutputFields method
checks “.withOutputFields()”
Holds
SequenceFileReader has defaultFields
checks “built-in reader types expose a static member called defaultFields”
Holds
TextFileReader has defaultFields
checks “built-in reader types expose a static member called defaultFields”
Holds
Method .setHdfsUri(), Alternative config name (deprecated) ~~hdfsspout.hdfs~~, Description HDFS URI for the hdfs Name node Example hdfs://namenodehost:8020
HdfsSpout has a setHdfsUri method
Holds
Method .setSourceDir(), Alternative config name (deprecated) ~~hdfsspout.source.dir~~, Description HDFS directory from where to read files E.g /data/inputdir
HdfsSpout has a setSourceDir method
Holds
Method .setArchiveDir(), Alternative config name (deprecated) ~~hdfsspout.archive.dir~~, Description After a file is processed completely it will be moved to this HDFS directory If this directory does not exist it will be created E.g /data/done
HdfsSpout has a setArchiveDir method
Holds
Method .setBadFilesDir(), Alternative config name (deprecated) ~~hdfsspout.badfiles.dir~~, Description if there is an error parsing a file's contents, the file is moved to this location If this directory does not exist it will be created E.g /data/badfiles
HdfsSpout has a setBadFilesDir method
Holds
Method .setCommitFrequencyCount(), Alternative config name (deprecated) ~~hdfsspout.commit.count~~, Default 20000, Description Record progress in the lock file after these many records are processed If set to 0, this criterion will not be used
HdfsSpout has a setCommitFrequencyCount method
Holds
Method .setCommitFrequencySec(), Alternative config name (deprecated) ~~hdfsspout.commit.sec~~, Default 10, Description Record progress in the lock file after these many seconds have elapsed Must be greater than 0
HdfsSpout has a setCommitFrequencySec method
Holds
Method .setMaxOutstanding(), Alternative config name (deprecated) ~~hdfsspout.max.outstanding~~, Default 10000, Description Limits the number of unACKed tuples by pausing tuple generation (if ACKers are used in the topology)
HdfsSpout has a setMaxOutstanding method
Holds
Option withRecordMapper(RecordMapper), Default FieldNameRecordMapper, Description Tuple → Record conversion
FieldNameRecordMapper implements RecordMapper
Holds
For anything custom (structs, lists, renames), implement RecordMapper:
RecordMapper is a contract
Holds
IcebergWriterBolt writes no WAL entries, because it makes nothing visible.
IcebergWriterBolt may not depend on WalEntry
Holdsbolt common
This class uses a Builder pattern and can be started either by calling one of the Builders constructors or by calling the static method builder in the KafkaSpoutConfig class.
KafkaSpoutConfig has a builder method
Holds
This provides a method routedTo that will say which specific stream the tuple should go to.
KafkaTuple has a routedTo method
Holds
Please take care when supplying a custom implementation, since an incorrect ManualPartitioner implementation could leave some partitions unread, or concurrently read by multiple spout instances. See the RoundRobinManualPartitioner for an example of how to implement this functionality.
RoundRobinManualPartitioner implements ManualPartitioner
Holds
If you need to customize partition assignment, you must implement the ManualPartitioner interface.
ManualPartitioner is a contract
Holds
You can customize how the spout discovers existing partitions, by implementing the TopicFilter interface.
TopicFilter is a contract
Holds
Storm-redis provides basic Bolt implementations, RedisLookupBolt and RedisStoreBolt, and RedisFilterBolt.
3 rules from this sentence
RedisLookupBolt is in external/storm-redis
checks “Storm-redis provides basic Bolt implementations, RedisLookupBolt”
Holds
RedisStoreBolt is in external/storm-redis
checks “Storm-redis provides basic Bolt implementations, RedisLookupBolt and RedisStoreBolt”
Holds
RedisFilterBolt is in external/storm-redis
Holds
These interfaces are combined with RedisLookupMapper and RedisStoreMapper and RedisFilterMapper which fit RedisLookupBolt and RedisStoreBolt, and RedisFilterBolt respectively.
3 rules from this sentence
RedisLookupBolt depends on RedisLookupMapper
checks “RedisLookupMapper”
Holdsbolt common
RedisStoreBolt depends on RedisStoreMapper
checks “RedisStoreMapper”
Holdsbolt common
RedisFilterBolt depends on RedisFilterMapper
checks “RedisFilterMapper which fit”
Holdsbolt common
The StreamBuilder tracks the overall pipeline of operations expressed via the Stream.
StreamBuilder depends on Stream
Holds
Value mappers are passed as arguments to the StreamBuilder.newStream.
StreamBuilder has newStream
Holds
Stream<T> windowedStream = stream.window(Window<?, ?> windowConfig);
Stream depends on Window
HoldsStream.java windowing
These operations transform a Stream of values into a stream of key-value pairs.
2 rules from this sentence
Stream has mapToPair
Holds
Stream has flatMapToPair
Holds
A CombinerAggregator interface is used as the argument of aggregate to enable this.
CombinerAggregator is a contract
Holds
The Storm config topology.state.provider can be used to choose the state provider implementation. For example set this to org.apache.storm.redis.state.RedisKeyValueStateProvider for redis based state store.
RedisKeyValueStateProvider implements StateProvider
Holdsexternal storm-client
A serialized java object (that implements IBolt)
IBolt is a contract
Holds
Specifying a bolt this way will cause Storm to instantiate a ShellBolt object to handle the communication between the JVM-based worker process and the non-JVM-based implementation of the component.
ShellBolt is a concrete implementation
Holds
The code that transforms the user's topology into the runtime topology is located here.
StormCommon is in storm-client/src, and StormCommon is in org.apache.storm.daemon
Holds
The main difference between them is the addition of the declareOutputFields method in the "Rich" versions of the interfaces.
2 rules from this sentence
IRichBolt has a declareOutputFields method
Holds
IRichSpout has a declareOutputFields method
Holds
CoordinatedBolt is the most important class here.
CoordinatedBolt is in org.apache.storm.coordination, and CoordinatedBolt is a concrete implementation
Holds
Definition of spout and associated interfaces (like the SpoutOutputCollector).
SpoutOutputCollector is in org.apache.storm.spout
Holds
Also contains ShellSpout which implements the protocol for defining spouts in non-JVM languages.
ShellSpout is in org.apache.storm.spout
Holds
Definition of bolt and associated interfaces (like OutputCollector).
OutputCollector is in org.apache.storm.task
HoldsPR #9093 · Sep 30
Also contains ShellBolt which implements the protocol for defining bolts in non-JVM languages.
ShellBolt is in org.apache.storm.task
HoldsPR #9093 · Sep 30
Finally, TopologyContext is defined here as well, which is provided to spouts and bolts so they can get data about the topology and its execution at runtime.
TopologyContext is in org.apache.storm.task
HoldsPR #9093 · Sep 30
TopologyBuilder is here as well as the helpful base classes for the different spouts and bolts.
TopologyBuilder is in org.apache.storm.topology
Holds
The slightly-higher level IBasicBolt interface is here, which is a simpler way to write certain kinds of bolts.
IBasicBolt is in org.apache.storm.topology, and IBasicBolt is a contract
Holds
Used in various places in Nimbus and Supervisor to make functions execute in serial to avoid any race conditions.
Supervisor depends on something in org.apache.storm.event
Holdsstorm-server org
Various utilities for working with local clusters during tests, e.g. completeTopology for running a fixed set of tuples through a topology for capturing the output, tracker topologies for having fine grained control over detecting when a cluster is "idle", and other utilities.
Testing has completeTopology
Holds
When using regular bolts, you can call the fail method on OutputCollector to fail the tuple trees of which that tuple is a member.
OutputCollector has a fail method
Holds
You can look at other min and minBy operations on Stream
2 rules from this sentence
Stream has a min method
Holds
Stream has a minBy method
Holds
You can look at other max and maxBy functions on Stream
2 rules from this sentence
Stream has a max method
Holds
Stream has a maxBy method
Holds
SlidingCountWindow.of(int windowCount, int slidingCount)
SlidingCountWindow has an of method
Holds
SlidingDurationWindow.of(BaseWindowedBolt.Duration windowDuration, BaseWindowedBolt.Duration slidingDuration)
SlidingDurationWindow has an of method
Holds
TumblingCountWindow.of(int windowLength)
TumblingCountWindow has an of method
Holds
TumblingDurationWindow.of(BaseWindowedBolt.Duration windowLength)
TumblingDurationWindow has an of method
Holds
There are three different interfaces for defining aggregators: CombinerAggregator, ReducerAggregator, and Aggregator.
3 rules from this sentence
CombinerAggregator is a contract
Holds
ReducerAggregator is a contract
Holds
Aggregator is a contract
Holds
The init method is called before processing the batch.
Aggregator has an init method
Holds
The aggregate method is called for each input tuple in the batch partition.
Aggregator has an aggregate method
Holds
The complete method is called when all tuples for the batch partition have been processed by aggregate.
Aggregator has a complete method
Holds
partition: This method takes in a custom partitioning function that implements org.apache.storm.grouping.CustomStreamGrouping
2 rules from this sentence
CustomStreamGrouping is in org.apache.storm.grouping
checks “org.apache.storm.grouping.CustomStreamGrouping”
Holds
CustomStreamGrouping is a contract
checks “implements org.apache.storm.grouping.CustomStreamGrouping”
Holds
You can do that with the TridentTopology#merge method, like so:
TridentTopology has a merge method
Holds
The base State interface just has two methods:
State is a contract
Holds
Trident provides the QueryFunction interface for writing Trident operations that query a source of state, and the StateUpdater interface for writing Trident operations that update a source of state.
2 rules from this sentence
QueryFunction is a contract
checks “Trident provides the QueryFunction interface”
Holds
StateUpdater is a contract
checks “the StateUpdater interface”
Holds
QueryFunction's execute in two steps.
QueryFunction has execute
Holds
The StateUpdater receives the State and a batch of tuples with updates to that State.
StateUpdater depends on State
Holds
You can then get access to the new values stream for further processing via the TridentState#newValuesStream method.
TridentState has a newValuesStream method
Holds
In this case, since this is a grouped stream, Trident expects the state you provide to implement the "MapState" interface.
MapState is a contract
Holds
When you do aggregations on non-grouped streams (a global aggregation), Trident expects your State object to implement the "Snapshottable" interface:
Snapshottable is a contract
Holds
The OpaqueMap, TransactionalMap, and NonTransactionalMap classes implement all the logic for doing the respective fault-tolerance logic. You simply provide these classes with an IBackingMap implementation that knows how to do multiGets and multiPuts of the respective key/values.
3 rules from this sentence
OpaqueMap depends on IBackingMap
Holds
TransactionalMap depends on IBackingMap
Holds
NonTransactionalMap depends on IBackingMap
checks “and NonTransactionalMap classes implement all the logic for doing the respective fault-tolerance logic. You simply provide these classes with an IBackingMap implementation that knows how to do multiGets and multiPuts of the respective key/values.”
Holds
OpaqueMap's will call multiPut with OpaqueValue's for the vals, TransactionalMap's will give TransactionalValue's for the vals,
2 rules from this sentence
OpaqueMap depends on OpaqueValue
checks “OpaqueMap's will call multiPut with OpaqueValue's for the vals”
Holdsmap OpaqueValue.java
TransactionalMap depends on TransactionalValue
checks “TransactionalMap's will give TransactionalValue's for the vals”
Holdsmap TransactionalValue.java
Trident also provides the CachedMap class to do automatic LRU caching of map key/vals.
CachedMap is in org.apache.storm.trident
Holds
Finally, Trident provides the SnapshottableMap class that turns a MapState into a Snapshottable object, by storing global aggregations into a fixed key.
3 rules from this sentence
SnapshottableMap depends on Snapshottable
Holdsmap snapshot
SnapshottableMap depends on MapState
Holds
SnapshottableMap is in org.apache.storm.trident
checks “Finally, Trident provides the SnapshottableMap class”
Holds
The object containing the processing logic implements the IRichSpout interface for spouts and the IRichBolt interface for bolts.
2 rules from this sentence
IRichSpout is a contract
Holds
IRichBolt is a contract
Holds
If you implement a bolt that subscribes to multiple input sources, you can find out which component the Tuple came from by using the Tuple#getSourceComponent method.
Tuple has a getSourceComponent method
Holds
The bolt interface IWindowedBolt is implemented by bolts that needs windowing support.
IWindowedBolt is a contract
Holds
In this case late tuples are going to be emitted on the specified stream and accessible via the field WindowedBoltExecutor.LATE_TUPLE_FIELD. The value of this field is a org.apache.storm.tuple.DetachedTuple, a serializable copy of the original tuple (source component, task, stream, fields and values) detached from the topology context, so that the late tuple stream can also be consumed by bolts running in different workers.
WindowedBoltExecutor depends on DetachedTuple
Holdstopology tuple
In this case late tuples are going to be emitted on the specified stream and accessible via the field WindowedBoltExecutor.LATE_TUPLE_FIELD.
WindowedBoltExecutor has a LATE_TUPLE_FIELD field
Holds
The new provided org.apache.storm.hdfs.format.SimpleFileNameFormat and org.apache.storm.hdfs.trident.format.SimpleFileNameFormat are more flexible, and the withName method support parameters as following:
SimpleFileNameFormat has a withName method
Holds
Both the HDFS bolt and Trident State implementation allow you to register any number of RotationActions.
HdfsBolt depends on RotationAction
Holdsbolt common
To partition your your data, write a class that implements the Partitioner interface and pass it to the withPartitioner() method of your bolt.
Partitioner is a contract
Holds
The SequenceFileBolt requires that you provide a org.apache.storm.hdfs.bolt.format.SequenceFormat that maps tuples to key/value pairs:
SequenceFileBolt depends on SequenceFormat
HoldsSequenceFileBolt.java format
A convenience method is provided for this: AvroUtils.addAvroKryoSerializations(conf);
AvroUtils has an addAvroKryoSerializations method
Holds
Method .setReaderType(), Alternative config name (deprecated) ~~hdfsspout.reader.type~~, Description Determines which file reader to use Set to 'seq' for reading sequence files or 'text' for text files Set to a fully qualified class name if using a custom file reader class (that implements interface org.apache.storm.hdfs.spout.FileReader)
2 rules from this sentence
HdfsSpout has a setReaderType method
Holds
FileReader is a contract
checks “interface org.apache.storm.hdfs.spout.FileReader”
Holds
Method .withOutputFields(), Description Sets the names for the output fields for the spout The number of fields depends upon the reader being used For convenience, built-in reader types expose a static member called defaultFields that can be used for setting this
3 rules from this sentence
HdfsSpout has a withOutputFields method
checks “Method .withOutputFields()”
Holds
SequenceFileReader has defaultFields
checks “built-in reader types expose a static member called defaultFields”
Holds
TextFileReader has defaultFields
checks “built-in reader types expose a static member called defaultFields”
Holds
Method .setHdfsUri(), Alternative config name (deprecated) ~~hdfsspout.hdfs~~, Description HDFS URI for the hdfs Name node Example hdfs://namenodehost:8020
HdfsSpout has a setHdfsUri method
Holds
Method .setSourceDir(), Alternative config name (deprecated) ~~hdfsspout.source.dir~~, Description HDFS directory from where to read files E.g /data/inputdir
HdfsSpout has a setSourceDir method
Holds
Method .setArchiveDir(), Alternative config name (deprecated) ~~hdfsspout.archive.dir~~, Description After a file is processed completely it will be moved to this HDFS directory If this directory does not exist it will be created E.g /data/done
HdfsSpout has a setArchiveDir method
Holds
Method .setBadFilesDir(), Alternative config name (deprecated) ~~hdfsspout.badfiles.dir~~, Description if there is an error parsing a file's contents, the file is moved to this location If this directory does not exist it will be created E.g /data/badfiles
HdfsSpout has a setBadFilesDir method
Holds
Method .setCommitFrequencyCount(), Alternative config name (deprecated) ~~hdfsspout.commit.count~~, Default 20000, Description Record progress in the lock file after these many records are processed If set to 0, this criterion will not be used
HdfsSpout has a setCommitFrequencyCount method
Holds
Method .setCommitFrequencySec(), Alternative config name (deprecated) ~~hdfsspout.commit.sec~~, Default 10, Description Record progress in the lock file after these many seconds have elapsed Must be greater than 0
HdfsSpout has a setCommitFrequencySec method
Holds
Method .setMaxOutstanding(), Alternative config name (deprecated) ~~hdfsspout.max.outstanding~~, Default 10000, Description Limits the number of unACKed tuples by pausing tuple generation (if ACKers are used in the topology)
HdfsSpout has a setMaxOutstanding method
Holds
In order to submit a topology as some other user, you can use the StormSubmitter.submitTopologyAs API.
StormSubmitter has submitTopologyAs
Holds
Alternatively you can use NimbusClient.getConfiguredClientAs to get a nimbus client as some other user and perform any nimbus action (i.e., kill/rebalance/activate/deactivate) using this client.
NimbusClient has getConfiguredClientAs
Holds
To ensure only authorized users can perform impersonation, you should start nimbus with nimbus.impersonation.authorizer set to org.apache.storm.security.auth.authorizer.ImpersonationAuthorizer.
ImpersonationAuthorizer is in org.apache.storm.security.auth.authorizer
Holds
a list of java plugins, all of which must implement IAutoCredentials interface, that populate the credentials on gateway and unpack them on the worker side. On a kerberos secure cluster they should be set by default to point to org.apache.storm.security.auth.kerberos.AutoTGT.
AutoTGT implements IAutoCredentials
Holdskerberos IAutoCredentials.java
160 rules from 20 docs

More open-source reports

Every report →

Other public repositories whose docs Striff read and checked against their code.