Apache NiFi Processors List — Descriptions of Each Processor

GetHDFS

Extracts files from the Hadoop Distributed File System (HDFS) into FlowFiles. This processor deletes the file from HDFS after fetching.

GetHDFSEvents

This processor polls notification events provided by the HdfsAdmin API. Because it uses the HdfsAdmin API, it must run as the HDFS superuser. There are currently six event types (append, close, create, metadata, rename, and unlink). See org.apache.hadoop.hdfs.inotify.Event for full descriptions. This processor polls for new events based on a defined duration. For each event received, a new flow file is created with expected attributes, and the event itself is serialized to JSON and written to the flow file content. For example, if event.type is APPEND, the flow file content contains a JSON file with info about the append event. On success the flow file is sent to the “success” relationship. Note where the generated flow file is stored. If the flow file is stored in one of the processor’s watched directories, there will be an endless stream of events. It’s also important to know this processor must consume all events; filtering must happen within the processor, because the HDFS admin event notification API has no filtering.

GetHDFSFileInfo

Retrieves a list of files and directories from HDFS. This processor creates a FlowFile representing the HDFS file/directory with related information. Its main purpose is to provide functionality similar to HDFS client commands like count, du, ls, test. Unlike ListHDFS, this processor is stateless, supports incoming connections, and provides directory-level info.

GetHDFSSequenceFile

Extracts sequence files from the Hadoop Distributed File System (HDFS) into FlowFiles.

GetHTMLElement

Extracts HTML element values from the incoming flow file content using a CSS selector. The incoming HTML is first converted to an HTML Document Object Model so HTML elements can be selected the way CSS selectors apply styles to HTML. Then the generated HTML DOM is “queried” using a user-defined CSS selector string. The query may yield 0–N results. If no result is found, the flow file is transferred to the “element not found” relationship. If N results are found, a new flow file is created and emitted for each. The query result is placed in the new flow file’s content or as an attribute. By default the result is written to an attribute, controllable via the “Destination” property. By setting “Prepend Element Value” or “Append Element Value”, the generated query value can also have data prepended or appended. Prepend and append values are treated as strings and concatenated to the result retrieved from the HTML DOM query. A more comprehensive reference on CSS selector syntax is at http://jsoup.org/apidocs/org/jsoup/select/Selector.html.

GetHTTP

This processor is deprecated and may be removed in a future release.

GetIgniteCache

Fetches a byte array from the Ignite Cache and adds it as FlowFile content. The processor uses the value of a FlowFile attribute (the Ignite cache entry key) as the cache lookup key. If no entry corresponding to the key is found in the cache, an error message is associated with the FlowFile. Note — the Ignite Kernel periodically outputs node performance stats to the log. This can be turned off by setting the log level of logger ‘org.apache.ignite’ to WARN in the logback.xml config file.

GetJMSQueue

This processor is deprecated and may be removed in a future release.

GetJMSTopic

This processor is deprecated and may be removed in a future release.

GetKafka

Fetches messages from Apache Kafka, specifically version 0.8.x. The complementary NiFi processor for sending messages is PutKafka.

GetMongo

Creates FlowFiles from documents in MongoDB loaded by a user-specified query.

GetMongoRecord

A record-based version of GetMongo that uses a Record writer to write the MongoDB result set.

GetRethinkDB

The processor fetches a JSON document from RethinkDB (https://www.rethinkdb.com/) using a document ID. The FlowFile will contain the retrieved document.

GetSFTP

Fetches files from an SFTP server and creates FlowFiles from them.

GetSNMP

Retrieves information from an SNMP agent and outputs a FlowFile containing attribute information with no content.

GetSolr

Queries Solr and outputs results as a FlowFile in XML format or using a Record Writer.

GetSplunk

Retrieves data from Splunk Enterprise.

GetSQS

Fetches messages from the Amazon Simple Queuing Service Queue.

GetTCP

Connects to the provided endpoint via TCP. Received data is written to a FlowFile as content.

GetTwitter

Extracts status changes from Twitter’s streaming API. In versions starting with 1.9.0, the Consumer Key and Access Token are marked sensitive per https://developer.twitter.com/en/docs/basics/authentication/guides/securing-keys-and-tokens.

HandleHttpRequest

Starts an HTTP server and listens for HTTP requests. For each request, a FlowFile is created and transferred to “success”. Designed to be used with the HandleHttpResponse processor to create web services.

HandleHttpResponse

Sends an HTTP response to the requester that generated the FlowFile. Designed to be used with HandleHttpRequest to create web services.

HashAttribute

Hashes the key/value pairs of multiple flow file attributes together and adds the hash as a new attribute. Optional attributes can be added whose name is the flow file attribute to consider and whose value is a regular expression; if it matches the attribute value, that attribute is used as part of the hash. If the regex contains a capture group, only the capture group’s value is used. For a processor that accepts various attributes and produces a cryptographic hash of each, see “CryptographicHashAttribute”.

HashContent

This processor is deprecated and may be removed in a future release.

IdentifyMimeType

Attempts to identify the MIME type for the FlowFile. If a MIME type can be identified, an attribute named “mime.type” is added with that value. If it cannot be determined, the value is set to “application/octet-stream”. Additionally, if a common file extension for the MIME type is known, the attribute mime.extension is set.

InferAvroSchema

Inspects the incoming FlowFile’s content to infer an Avro schema. The processor uses the Kite SDK to try to auto-generate an Avro schema from the incoming content. When inferring a schema from JSON data, key names are used in the generated Avro schema definition. When inferring from CSV data, a “header definition” must be present as the first line of the input data, or explicitly set via the “CSV Header Definition” property. The “header definition” is just a comma-separated line defining each column’s name. A header definition is required to determine the name to assign to each field in the resulting Avro definition. When inferring data types, the higher-order data type is always used if ambiguous. For example, when inspecting numeric values, the type can be set to “long” instead of “integer”, since long can safely hold any “integer” value. Currently only CSV and JSON content are supported for auto-inferring Avro schemas. The content type present in the incoming FlowFile is set via the “Input Content Type” property, which can be explicitly set to CSV, JSON, or “Use mime.type value” (which checks the value of the mime.type attribute on the incoming FlowFile).

InvokeAWSGatewayApi

A client for AWS Gateway API endpoints.

InvokeHTTP

An HTTP client processor that can interact with a configurable HTTP endpoint. The target URL and HTTP method are configurable. FlowFile attributes are converted to HTTP headers, and FlowFile content is included as the request body (if the HTTP method is PUT, POST, or PATCH).

InvokeScriptedProcessor

Experimental — invokes a script engine for a processor defined in a given script. The script must define a valid class implementing the Processor interface, and must set a variable “processor” to an instance of that class. Processor methods (such as onTrigger()) are delegated to the scripted Processor instance. In addition, any Relationships or PropertyDescriptors defined by the scripted processor are added to the configuration dialog. Experimental: effects of sustained use not yet verified.

ISPEnrichIP

Looks up ISP information for an IP address and adds the information to FlowFile attributes. ISP data is provided as a MaxMind ISP database (note this differs from the GeoLite database used by some geo-enrichment tools). The attribute containing the IP to look up is provided by the “IP Address Attribute” property. If the provided attribute name is “X”, the attribute added via enrichment takes the form X.isp.

JoltTransformJSON

Applies a list of Jolt specifications to the FlowFile JSON payload. Creates a new FlowFile with the transformed content and routes it to the “success” relationship. If JSON transformation fails, the original FlowFile is routed to the “failure” relationship.

JoltTransformRecord

Applies a list of Jolt specifications to the FlowFile payload. Creates a new FlowFile with the transformed content and routes it to the “success” relationship. If transformation fails, the original FlowFile is routed to the “failure” relationship.

JsonQueryElasticsearch

A processor that lets the user run queries written in ElasticSearch’s JSON DSL (with aggregations). It does not automatically paginate queries for the user. If an incoming relationship is added to this processor, it uses the flow file’s content as the query. Care should be taken with query size, since the entire ElasticSearch response is loaded into memory at once and converted to the resulting flow file.

ListAzureBlobStorage

Lists blobs in an Azure storage container. List details are appended to an empty FlowFile for use with FetchAzureBlobStorage. This processor is intended to run only on the primary node of a cluster. If the primary node changes, the new primary picks up where the previous one stopped, without duplicating all data.

ListDatabaseTables

Generates a set of flow files, each containing attributes corresponding to metadata about a table in the database connection. Once metadata about a table is fetched, it is not fetched again until the refresh interval (if set) has passed or state is cleared manually.

ListenBeats

Uses Libbeat’s ‘output.logstash’ to listen for messages sent by libbeat-compatible clients (e.g., filebeat, metricbeat), writing their JSON payloads to FlowFile content. This processor replaces the now-deprecated ListenLumberjack.

ListenHTTP

Starts an HTTP Server and listens on a given base path to convert incoming requests into FlowFiles. The default service URI is http://{hostname}:{port}/contentListener. Only HEAD and POST requests are supported. GET, PUT, and DELETE result in an error and HTTP response status 405.

ListenLumberjack

This processor is deprecated and may be removed in the near future. Listens for Lumberjack messages sent via TCP to a given port. Each message is acknowledged after it is successfully written to a FlowFile. Each FlowFile contains the data portion of one or more Lumberjack frames. Where a Lumberjack frame contains a syslog message, the processor’s output can be sent to the ParseSyslog processor for further processing.

ListenRELP

Listens for RELP messages sent via TCP to a given port. Each message is acknowledged after successfully written to a FlowFile. Each FlowFile contains the data portion of one or more RELP frames. Where a RELP frame contains a syslog message, the output can be sent to ParseSyslog for further processing.

ListenSMTP

Implements a lightweight SMTP server on an arbitrary port, allowing NiFi to listen for incoming email. Note this server performs no email validation. If direct exposure to the internet is desired, combining NiFi with an industrial-scale MTA (such as Postfix) may be a better idea. This processor’s threads are managed by the underlying smtp server, so the processor itself needs no multi-thread support.

ListenSyslog

Listens for Syslog messages sent via TCP or UDP to a given port. Incoming messages are checked against a regex for RFC5424 and RFC3164 formats. Each message is formatted as: (PRI)(VERSION)(TIMESTAMP)(HOSTNAME)(BODY) where VERSION is optional. The timestamp can be an RFC5424 timestamp “yyyy-MM-dd’T’HH:mm:ss.SZ” or “yyyy-MM-dd’T’HH:mm:ss.S+hh:mm”, or an RFC3164 timestamp “MMM d HH:mm:ss”. If an incoming message matches one of these patterns, the message is parsed and the pieces are placed in FlowFile attributes, with the original message in the FlowFile content. If it doesn’t match, it is not parsed, the syslog.valid attribute is set to false, and the original message is in the FlowFile content. Valid messages are transferred on the success relationship, invalid on the invalid relationship.

ListenTCP

Listens for incoming TCP connections and reads data from each connection using a line delimiter as the message separator. The default behavior is one FlowFile per message, but higher throughput is controlled by increasing the batch size. The receive buffer size must be set as large as the largest expected message, meaning if there’s a line delimiter every 100kb, the receive buffer must be larger than 100kb.

ListenTCPRecord

Listens for incoming TCP connections and uses the configured record reader to read data from each connection, writing records to a flow file with the configured record writer. The chosen record reader type determines how the client sends data. For example, when reading logs with a Grok reader, the client can keep the connection open and stream data continuously, but when using a JSON reader the client cannot send an array of JSON documents and then another array on the same connection, because the reader would be in a bad state. Records are read from the connection in blocking mode and time out per the read timeout specified in the processor. If the read times out, or any other error is encountered while reading, the connection is closed and any records read up to that point are processed per the configured read-error strategy (discard or transfer). If the client keeps the connection open, the processor’s concurrent tasks should be adjusted to match the maximum allowed TCP connections, so there is one task per connection.

ListenUDP

Listens for Datagram Packets on a given port. The default behavior is one FlowFile per datagram, but for higher throughput the Max Batch Size property can be increased to specify the number of datagrams batched into a single FlowFile. By specifying “Sending Host” and “Sending Host Port” attributes, you can limit this processor to listening for datagrams from a specific remote host and port; otherwise it listens for datagrams from all hosts and ports.

ListenUDPRecord

Listens for Datagram Packets on a given port and uses the configured Record Reader to read each datagram’s content. Then each record is written to a flow file using the configured Record Writer. By specifying “Sending Host” and “Sending Host Port”, you can limit listening to datagrams from a specific remote host and port; otherwise all hosts and ports.

ListenWebSocket

Acts as a WebSocket server endpoint to accept client connections. FlowFiles are transferred to downstream relationships based on the received message type, as the WebSocket server configured with this processor receives client requests.

ListFile

Retrieves a list of files from the local file system. For each listed file, a FlowFile representing that file is created, so it can be fetched with FetchFile. This processor is intended to run only on the primary node of a cluster. If the primary changes, the new primary picks up where the previous stopped, without duplicating data. Unlike GetFile, this processor does not delete any data from the local file system.

ListFTP

Performs a listing of files residing on an FTP server. For each file found on the remote server, a new FlowFile is created with the filename attribute set to the remote file’s name. It can then be combined with FetchFTP to fetch those files.

ListGCSBucket

Retrieves a list of objects from a GCS bucket. For each listed object, a FlowFile representing the object is created, so it can be fetched with FetchGCSObject. Intended to run only on the primary node. If the primary changes, the new primary picks up where the previous stopped without duplicating data.

ListHDFS

Retrieves a list of files from HDFS. On each list execution, the file with the latest timestamp is excluded and picked up on the next execution, to ensure we don’t miss or duplicate files written with the same timestamp immediately before and after a single execution. For each file listed in HDFS, this processor creates a FlowFile representing the HDFS file to fetch with FetchHDFS. Intended to run only on the primary node. If the primary changes, the new primary picks up where the previous stopped without duplicating data. Unlike GetHDFS, this processor does not delete any data from HDFS.

ListS3

Retrieves a list of objects from an S3 bucket. For each listed object, a FlowFile representing the object is created, so it can be fetched with FetchS3Object. Intended to run only on the primary node. If the primary changes, the new primary picks up where the previous stopped without duplicating data.

ListSFTP

Performs a listing of files residing on an SFTP server. For each file found on the remote server, a new FlowFile is created with the filename attribute set to the remote file’s name. It can then be combined with FetchSFTP to fetch those files.

LogAttribute

Emits the FlowFile’s attributes at a specified log level.

LogMessage

Emits a log message at a specified log level.

LookupAttribute

Looks up lookup attributes in a lookup service.

LookupRecord

Extracts one or more fields from a Record and looks up their values in a LookupService. If the LookupService returns a result, it can optionally be added to the Record, in which case the processor acts as an enrichment processor. Either way, the record is then routed to the “match” relationship or the “unmatched” relationship (if the “Routing Strategy” property is configured to do so), indicating whether the LookupService returned a result, allowing the processor to also act as a routing processor. The “coordinates” used to look up values in the lookup service are defined by adding user-defined attributes. Each added attribute puts an entry into a Map, where the attribute name becomes the Map Key and the value returned by the RecordPath becomes that key’s value. If the RecordPath returns multiple values, the Record is routed to the “unmatched” relationship (or “success”, depending on the “Routing Strategy” config). If one or more fields match the Result RecordPath, all matched fields are updated. If no match is found in the configured LookupService, no fields are updated — i.e., it won’t overwrite existing values in the Record with null. But note that if your schema doesn’t account for the LookupService’s returned results (specifically the schema configured for the Record Writer), those fields won’t be written to the FlowFile.

MergeContent

Merges a set of FlowFiles into one FlowFile based on a user-defined strategy. Recommended to configure for a single incoming connection, since FlowFiles from different connections won’t be grouped. Updates the mime.type attribute as needed.

MergeRecord

Merges multiple record-oriented FlowFiles into one FlowFile containing all records of the input FlowFiles. Works by creating “bins” and adding FlowFiles to them until full. Once a bin is full, all FlowFiles are merged into one output FlowFile routed to the ‘merged’ relationship. A bin contains many ‘similar FlowFiles’. For two FlowFiles to be considered “similar FlowFiles”, they must have the same Schema (identified by the Record Reader), and if a property is set, the same value for the specified property. See the processor usage doc for more info.

ModifyBytes

Discards byte ranges from the beginning and end, or all content of a binary file.

ModifyHTMLElement

Modifies the value of an existing HTML element. Targets the desired element to modify using CSS selector syntax. The incoming HTML is first converted to an HTML DOM so elements can be selected as with CSS selectors. Then the generated HTML DOM is “queried” with a user-defined CSS selector string to find the element to modify. If found, the element’s value in the DOM is updated with the specified “Modified Value” property. All DOM elements matching the CSS selector are updated. Once all are updated, the DOM is rendered to HTML and the result replaces the flow file content with the updated HTML. See http://jsoup for a more comprehensive reference on CSS selector syntax.

MonitorActivity

Monitors the activity stream and emits an indication when the flow has no data for a specified time, and again when activity resumes.

MoveHDFS

Renames an existing file or file directory (non-recursive) on the Hadoop Distributed File System (HDFS).

Notify

Caches a release signal identifier in the distributed cache, optionally with the FlowFile’s attributes. Once that signal is found in the cache, any flow files held at the corresponding Wait processor are released.

ParseCEF

Parses the content of a CEF-format message and adds attributes to the FlowFile for the header and extensions of the CEF message parts. Note: this processor expects a CEF message with no syslog header (i.e., starting from “CEF:0”).

ParseEvtx

Parses the content of Windows Event Log files (evtx) and writes the resulting XML to the FlowFile.

ParseNetflowv5

Parses netflowv5 byte ingestion and adds it to the NiFi flow file as attributes or JSON content.

ParseSyslog

Attempts to parse the content of a Syslog message according to RFC5424 and RFC3164 formats and adds attributes to the FlowFile for each part of the Syslog message. Note: RFC3164 is informational and has many varied implementations in the wild. If a message can’t be parsed, consider using RFC5424 or a generic parsing processor like ExtractGrok.

ParseSyslog5424

Attempts to parse well-formed Syslog messages in RFC5424 format and add attributes for each part of the Syslog message, including structured data. Structured data is written as one attribute per item id + parameter; see https://tools.ietf.org/html/rfc5424. Note: ParseSyslog5424 is closer to the spec than ParseSyslog. If your Syslog generator doesn’t strictly follow the spec (e.g., uses ’-’ for missing header entries), those logs will fail with this parser but not with ParseSyslog.

PartitionRecord

Receives record-oriented data (data readable by the configured record reader) and evaluates one or more RecordPaths against each record in the incoming FlowFile. Each record is then grouped with other “similar records”, and a FlowFile is created for each group of “similar records”. Two records being “similar” is decided by user-defined properties. The user must input at least one user-defined property whose value is a RecordPath. If two records have the same value for all configured RecordPaths, they are considered similar. Because we know all records in a given output FlowFile have the same value for the RecordPath-specified fields, an attribute is added per field. See the “usage” page for more info and examples.

PostHTTP

This processor is deprecated and may be removed in a future release.

PublishAMQP

Creates an AMQP message from the FlowFile’s content and sends it to an AMQP Exchange. In the typical AMQP exchange model, messages sent to an AMQP Exchange are routed to their final destination in a queue (binding) based on the “routing key”. If, due to some misconfiguration, the binding between the exchange, routing key, and queue isn’t set, the message has no final destination and is returned (i.e., data doesn’t enter the queue). If this happens, you’ll see a log in app-log and bulletin recording the effect, and the FlowFile is routed to the “failure” relationship.

PublishGCPubSub

Publishes the incoming flow file’s content to a configured Google Cloud PubSub topic. The processor supports dynamic attributes. Any dynamic attributes are sent with the message as “attributes”.

PublishJMS

Creates a JMS message from the FlowFile’s content and sends it as a JMS BytesMessage or TextMessage to a JMS destination (queue or topic). FlowFile attributes are added to the outgoing JMS message as JMS headers and/or properties.

PublishKafka

Sends the FlowFile’s content as a message to Apache Kafka using the Kafka 0.9.x Producer. The message to send can be individual FlowFiles or separated by a user-specified delimiter (e.g., newline). Note that in some cases the publisher may stall indefinitely; we are watching the Kafka community and will adopt fixes. Meanwhile a state may be reached resolved only by restarting the JVM NiFi runs on. The complementary processor for receiving messages is ConsumeKafka.

PublishKafka_0_10

Sends the FlowFile’s content as a message to Apache Kafka using the Kafka 0.10.x Producer API. The message to send can be individual FlowFiles or separated by a user-specified delimiter (e.g., newline). The complementary processor for receiving messages is ConsumeKafka_0_10.

PublishKafka_0_11

Sends the FlowFile’s content as a message to Apache Kafka using the Kafka 0.11.x Producer API. The message to send can be individual FlowFiles or separated by a user-specified delimiter (e.g., newline). The complementary processor for receiving messages is ConsumeKafka_0_11.

PublishKafka_1_0

Sends the FlowFile’s content as a message to Apache Kafka using the Kafka 1.0 Producer API. The message to send can be individual FlowFiles or separated by a user-specified delimiter (e.g., newline). The complementary processor for receiving messages is ConsumeKafka_1_0.

PublishKafka_2_0

Sends the FlowFile’s content as a message to Apache Kafka using the Kafka 2.0 Producer API. The message to send can be individual FlowFiles or separated by a user-specified delimiter (e.g., newline). The complementary processor for receiving messages is ConsumeKafka_2_0.

PublishKafkaRecord_0_10

Sends the FlowFile’s content as individual records to Apache Kafka using the Kafka 0.10.x Producer API. The FlowFile content should be record-oriented data readable by the configured Record Reader. The complementary processor for receiving messages is ConsumeKafka_0_10_Record.

PublishKafkaRecord_0_11

Sends the FlowFile’s content as individual records to Apache Kafka using the Kafka 0.11.x Producer API. The FlowFile content should be record-oriented data readable by the configured Record Reader. The complementary processor for receiving messages is ConsumeKafka_0_11_Record.

PublishKafkaRecord_1_0

Sends the FlowFile’s content as individual records to Apache Kafka using the Kafka 1.0 Producer API. The FlowFile content should be record-oriented data readable by the configured Record Reader. The complementary processor for receiving messages is ConsumeKafkaRecord_1_0.

PublishKafkaRecord_2_0

Sends the FlowFile’s content as individual records to Apache Kafka using the Kafka 2.0 Producer API. The FlowFile content should be record-oriented data readable by the configured Record Reader. The complementary processor for receiving messages is ConsumeKafkaRecord_2_0.

PublishMQTT

Publishes a message to an MQTT topic.

PutAzureBlobStorage

Puts content into an Azure storage Blob.

PutAzureEventHub

Sends the FlowFile’s content to the Windows Azure Event Hub. Note: FlowFile content is buffered into memory before sending, so care should be taken to avoid sending FlowFiles exceeding available Java heap space.

PutAzureQueueStorage

Writes the incoming FlowFiles’ content to the configured Azure Queue Storage.

PutBigQueryBatch

Batch-loads streaming file content into a Google BigQuery table.

PutCassandraQL

Executes the provided Cassandra Query Language (CQL) statement on a Cassandra 1.x, 2.x, or 3.0.x cluster. The incoming FlowFile content should be the CQL command to execute. CQL commands can use ? escaped parameters. In that case the parameters must exist as FlowFile attributes with naming convention cql.args.N.type and cql.args.N.value, where N is a positive integer. cql.args.N.type should be a lowercase string representing the Cassandra type.

PutCassandraRecord

A record-aware processor that uses the configured “Record Reader” to read the incoming FlowFile’s content as individual records and writes them to Apache Cassandra using native protocol version 3 or higher.

PutCloudWatchMetric

Publishes a metric to Amazon CloudWatch. The metric can be a single value or a statistics set composed of minimum, maximum, sum, and sample count.

PutCouchbaseKey

Puts a document into Couchbase Server via key/value access.

PutDatabaseRecord

Uses the specified RecordReader to input (possibly multiple) records from the incoming flow file. These records are converted to SQL statements and executed as a single batch. If any error occurs, the flow file is routed to failure or retry; on successful transfer, the incoming flow file is routed to success. The statement type is specified via the Statement Type property, which accepts hardcoded values like INSERT, UPDATE, DELETE, and ‘Use statement.type Attribute’, which causes the processor to get the statement type from the flow file attribute. Important: if the statement type is UPDATE, the incoming record must not change the value of the primary key (or user-specified update key). If such a record is encountered, …

PutDistributedMapCache

Gets the FlowFile’s content and puts it into the distributed map cache using a cache key computed from FlowFile attributes. If the cache already contains the entry and the cache update strategy is “keep original”, the entry is not replaced.

PutDruidRecord

Sends records to Druid for indexing. Uses the Druid Tranquility Controller service.

PutDynamoDB

Extracts a document from DynamoDB based on hash and range keys. The table can use hash and range or hash key alone. Supported key types are string, number, and value can be a json document. In the hash-and-range case, both keys are required. FlowFile content must be JSON. FlowFile content maps to the Json Document attribute specified in the DynamoDB item.

PutElasticsearch

Writes the FlowFile’s content to Elasticsearch using specified parameters (e.g., index and document type to insert). A secure connection is possible if the cluster is configured for authorization and/or secure transport (SSL/TLS) and the Shield plugin is available. Supports Elasticsearch 2.x clusters.

PutElasticsearch5

Writes the FlowFile’s content to Elasticsearch using specified parameters. A secure connection is possible if the cluster is configured for authorization and/or secure transport (SSL/TLS) and the X-Pack plugin is available. Supports Elasticsearch 5.x clusters.

PutElasticsearchHttp

Writes the FlowFile’s content to Elasticsearch using specified parameters.

PutElasticsearchHttpRecord

Writes records from the FlowFile to Elasticsearch using specified parameters (e.g., index and document type) and operation type (index, upsert, delete, etc.). Note: the Bulk API is used to send records. This means the entire incoming flow file content is read into memory and each record converted to a JSON document added to a single HTTP request body. For very large flow files (e.g., with many records), this may cause memory issues.

PutEmail

Sends an email to the configured recipients for each incoming FlowFile.

PutFile

Writes the FlowFile’s content to the local file system.

PutFTP

Sends FlowFiles to an FTP server.

PutGCSObject

Puts a flow file into a Google Cloud Bucket.

PutGridFS

Writes a file to a GridFS bucket.

PutHBaseCell

Adds the FlowFile’s content to HBase as the value of a single cell.

PutHBaseJSON

Adds rows to HBase based on the content of an incoming JSON document. Each FlowFile must contain a UTF-8 encoded JSON document; any FlowFile whose root element is not a single document is routed to failure. Each JSON field name and value becomes a column qualifier and HBase row value. Any field with a null value is skipped, and fields with complex values are handled per the complex-field strategy. The row ID can be specified directly on the processor via the Row Identifier property, or extracted from the JSON document via the Row Identifier Field Name property. The processor saves the content of all FlowFiles in a given batch at once.

PutHBaseRecord

Adds rows to HBase from the flow file content using the configured record reader.

PutHDFS

Writes FlowFile data to the Hadoop Distributed File System (HDFS).

PutHiveQL

Executes HiveQL DDL/DML commands (e.g., UPDATE, INSERT). The incoming FlowFile content should be the HiveQL command to execute. HiveQL commands can use ? escaped parameters. In that case the parameters must exist as FlowFile attributes with naming convention hiveql.args.N.type and hiveql.args.N.value, where N is a positive integer. hiveql.args.N.type should be a number representing the JDBC Type. FlowFile content is expected in UTF-8.

PutHiveStreaming

Sends flow file data to an Apache Hive table using Hive Streaming. The incoming flow file should be in Avro format and the table must exist in Hive. See Hive docs for table requirements (format, partitioning, etc.). Partition values are extracted from the Avro record based on the partition column names specified in the processor. Note: if multiple concurrent tasks are configured for this processor, a single thread can write to only one table at a time. Other tasks intending to write to the same table will wait for the current task to finish writing.

PutHTMLElement

Places a new HTML element in an existing HTML DOM. The desired location of the new HTML element is specified using CSS selector syntax. The incoming HTML is first converted to an HTML DOM so the location can be selected like CSS selectors applying styles. Then the generated HTML DOM is “queried” with a user-defined CSS selector string to find where to add the new HTML element. After adding, the DOM is rendered to HTML and the result replaces the flow file content with the updated HTML. See http://jsoup.org/apidocs/org/jsoup/select/Selector.html for a comprehensive CSS selector reference.

PutIgniteCache

Streams the FlowFile’s content to the Ignite Cache using a DataStreamer. The processor uses the value of a FlowFile attribute (Ignite cache entry key) as the cache key and the FlowFile’s byte array as the cache entry value. Both a string key and a non-null byte array value are required, otherwise the FlowFile is transferred to the failure relationship. Note — the Ignite Kernel periodically outputs node performance stats to the log; set logger ‘org.apache.ignite’ to WARN in logback.xml to turn off.

PutInfluxDB

The processor writes the FlowFile’s content in “line protocol” form. See InfluxDB docs (https://www.influxdb.com/) for line protocol details. A flow file can contain a single measurement point or multiple separated by line delimiters. The timestamp (last field) should be nanosecond resolution.

PutJMS

This processor is deprecated and may be removed in a future release.

PutKafka

Sends the FlowFile’s content as a message to Apache Kafka, specifically version 0.8.x. The message can be individual FlowFiles or separated by a user-specified delimiter (e.g., newline). The complementary processor for receiving messages is GetKafka.

PutKinesisFirehose

Sends content to a specified Amazon Kinesis Firehose. To send data to the firehose, the firehose delivery stream name must be specified.

PutKinesisStream

Sends content to a specified Amazon Kinesis. The stream name must be specified.

PutKudu

Uses the provided Record Reader to read records from the incoming FlowFile and writes those records to the specified Kudu table. The table schema must be provided in the processor properties or source. If any error occurs reading records from input or writing to Kudu, the FlowFile is routed to failure.

PutLambda

Sends content to a specified Amazon Lambda function. The AWS credentials used for authentication must have permission to invoke the Lambda function (lambda:InvokeFunction). FlowFile content must be JSON.

PutMongo

Writes the FlowFile’s content to MongoDB.

PutMongoRecord

A record-aware processor for inserting data into MongoDB. It uses the configured record reader and schema to read the incoming record set from the flow file body, then bulk-inserts those records into the configured MongoDB collection. This processor does not support update, delete, or upsert. The number of documents inserted at once is controlled by the “Insert Batch Size” config property. This should be set to a reasonable size so MongoDB doesn’t insert too many at once.

PutParquet

Uses the provided Record Reader to read records from the incoming FlowFile and writes those records to a Parquet file. The Parquet file schema must be provided in the processor properties. The processor first writes a temporary dot file, and after successfully writing each record to the dot file, renames it to its final name. If the dot file can’t be renamed, it retries up to 10 times; if still unsuccessful, deletes the dot file and routes the flow file to failure. If any error occurs reading from input or writing to output, the entire dot file is deleted and the flow file is routed to failure or retry depending on the error.

PutRethinkDB

The processor writes the FlowFile’s JSON content to RethinkDB (https://www.rethinkdb.com/). The flow file should contain a JSON Object and an array of JSON documents.

PutRiemann

Sends events to Riemann (http://riemann.io) as FlowFiles pass through this processor. You can notify Riemann that a FlowFile passed through, or attach more meaningful metrics such as the time a FlowFile took to reach this processor. All attributes attached to an event support the NiFi Expression Language.

PutS3Object

Puts FlowFiles into an Amazon S3 bucket. Upload uses the PutS3Object method or PutS3MultipartUpload method. PutS3Object sends the file in a single synchronous call but has a 5GB size limit. Use PutS3MultipartUpload for larger files. This multipart process saves state after each step so a large upload can be resumed with minimal loss if the processor or cluster stops and restarts. Multipart upload has three steps: 1) initiate upload, 2) upload parts, 3) complete upload. For multipart, the processor saves state locally, tracking the upload ID and uploaded parts, both required to complete the upload. The AWS library selects the endpoint URL based on the AWS region, but this can be overridden via the “Endpoint Override URL” property to use other S3-compatible endpoints. The S3 API specifies a 5GB max file size for PutS3Object uploads. It also requires multipart upload parts to be at least 5MB except the last. These limits bound the “Multipart Upload Threshold” and “Part Size” properties.

PutSFTP

Sends FlowFiles to an SFTP server.

PutSlack

Sends a message to your team on slack.com.

PutSNS

Sends the FlowFile’s content as a notification to the Amazon Simple Notification Service.

PutSolrContentStream

Sends the FlowFile’s content to Solr as a ContentStream.

PutSolrRecord

Indexes records from the FlowFile into Solr.

PutSplunk

Sends logs to Splunk Enterprise via TCP, TCP+TLS/SSL, or UDP. If a message delimiter is provided, the processor reads messages from the incoming FlowFile by that delimiter and sends each to Splunk. If no delimiter is provided, the FlowFile content is sent directly to Splunk as if it were a single message.

PutSQL

Executes SQL UPDATE or INSERT commands. The incoming FlowFile content should be the SQL command to execute. SQL commands can use ? escaped parameters. In that case the parameters must exist as FlowFile attributes with naming convention sql.args.N.type and sql.args.N.value, where N is a positive integer. sql.args.N.type should be a number representing the JDBC Type. FlowFile content is expected in UTF-8.

PutSQS

Publishes a message to the Amazon Simple Queuing Service Queue.

PutSyslog

Sends Syslog messages to a given host and port via TCP or UDP. The message is constructed from the processor’s “Message ___” properties, which can use Expression Language to generate the message from incoming FlowFiles. These properties construct a message of the form: (PRI)(VERSION)(TIMESTAMP)(HOSTNAME)(BODY) where VERSION is optional. The constructed message is checked against a regex for RFC5424 and RFC3164 formats. The timestamp can be RFC5424 “yyyy-MM-dd’T’HH:mm:ss.SZ” or “yyyy-MM-dd’T’HH:mm:ss.S+hh:mm”, or RFC3164 “MMM d HH:mm:ss”. If a message is constructed that doesn’t form a valid Syslog message, it is routed to the invalid relationship. Valid messages are sent to the Syslog server; success routes to success, failure to failure.

PutTCP

Receives a FlowFile and sends its content via a TCP connection to the configured TCP server. By default FlowFiles are transmitted over the same TCP connection (or a TCP connection pool if multiple input threads are configured). To help the TCP server determine message boundaries, an optional “Outgoing Message Delimiter” string can be configured, appended to each FlowFile’s content end when transmitted. An optional “Connection Per FlowFile” parameter can change behavior so each FlowFile’s content is transmitted over a single TCP connection opened on receipt and closed after sending. This option is only for low-volume scenarios.

PutUDP

Receives a FlowFile and packages its content into a single UDP datagram packet, then sends it to the configured UDP server. The user must ensure the FlowFile content provided is no larger than the underlying UDP transport’s max size. The max varies by platform but is usually under 64KB. If the content is larger than the max, it is marked failed.

PutWebSocket

Uses the WebSocket session established by ListenWebSocket or ConnectWebSocket to send a message to the WebSocket remote endpoint.

QueryCassandra

Executes the provided Cassandra Query Language (CQL) select query on a Cassandra 1.x, 2.x, or 3.0.x cluster. Results can be converted to Avro or JSON. Streaming supported. Can be scheduled on timer/cron or triggered by incoming FlowFile. If triggered, the FlowFile’s attributes are available when evaluating the select query. FlowFile attribute ‘executecql.row.count’ indicates rows selected.

QueryDatabaseTable

Generates a SQL select query, or uses the provided statement, and executes it to fetch all rows whose value in the specified Maximum Value column is greater than the previously seen maximum. Results converted to Avro. Several properties support Expression Language but incoming connections are not allowed. The Variable Registry can provide values for properties with Expression Language. If you want to use flow file attributes for these queries, use GenerateTableFetch and/or ExecuteSQL. Streaming supported. Intended to run only on the primary node. FlowFile attribute ‘querydbtable.row.count’ indicates rows selected.

QueryDatabaseTableRecord

Generates a SQL select query, or uses the provided statement, and executes it to fetch all rows whose value in the specified Maximum Value column is greater than the previously seen maximum. Results converted to the format specified by the Record Writer. Several properties support Expression Language but incoming connections are not allowed. The Variable Registry can provide values for properties with Expression Language. If you want to use flow file attributes for these queries, use GenerateTableFetch and/or ExecuteSQL. Streaming supported. Intended to run only on the primary node. FlowFile attribute ‘querydbtable.row.count’ indicates rows selected.

QueryDNS

A powerful DNS query processor mainly used to enrich a DataFlow with DNS-based APIs (e.g., RBL, ShadowServer’s ASN lookup), but also usable for general DNS lookups.

QueryElasticsearchHttp

Queries Elasticsearch using specified connection properties. Note the full body of each page of documents is read into memory before being written to flow files for transfer. Also note the Elasticsearch max_result_window index setting is the upper bound on records retrievable with this query. To retrieve more, use the ScrollElasticsearchHttp processor.

QueryRecord

Evaluates one or more SQL queries against the FlowFile’s content. The SQL query results become the output FlowFile’s content. Useful for field-specific filtering, transformation, and row-level filtering. Columns can be renamed, simple calculations and aggregations performed, etc. The processor is configured with a Record Reader controller service and a Record Writer service for flexible input/output formats. The processor must have at least one user-defined property. The property name is the Relationship the data is routed to, and the value is a SQL SELECT statement specifying how to transform/filter the input. The SQL statement must be valid ANSI SQL, powered by Apache Calcite. If transformation fails, the original FlowFile is routed to “failure”; otherwise the selected data is routed to the associated relationship. If the Record Writer chooses to inherit the schema from the Record, note the inherited schema comes from the ResultSet, not the input Record. This lets a single QueryRecord instance have multiple queries, each returning a different set of columns and aggregations. Thus the derived schema has no schema name, so if inheriting from Record, the configured Record Writer won’t try to write the schema name as an attribute.

QuerySolr

Queries Solr and outputs results as a FlowFile in XML format or using a Record Writer.

QueryWhois

A powerful whois query processor mainly used to enrich a DataFlow with whois-based APIs (e.g., ShadowServer’s ASN lookup), but also usable for general whois lookups.

ReplaceText

Updates the FlowFile’s content by evaluating a regular expression (regex) against the FlowFile and replacing the content portions that match the regex with a replacement value.

ReplaceTextWithMapping

Updates the FlowFile’s content by evaluating a regex against it and replacing matched content portions with a replacement value provided in a mapping file.

ResizeImage

Resizes an image to user-specified dimensions. Uses image codecs registered in the NiFi environment. By default includes JPEG, PNG, BMP, WBMP, and GIF.

RouteHL7

Routes incoming HL7 data based on user-defined queries. To add a query, add a new property to the processor. The property name becomes a new relationship of the processor, and the value is an HL7 query language query. If a FlowFile matches the query, a copy is routed to the associated relationship.

RouteOnAttribute

Routes FlowFiles based on their attributes using the Attribute Expression Language.

RouteOnContent

Applies a regular expression to the FlowFile’s content and routes a copy of the FlowFile to each destination whose regex matches. Regexes are added as user-defined attributes, where the property name is the relationship name and the value is the regex matching the FlowFile content. User-defined attributes do support the Attribute Expression Language, but the result is interpreted as a literal value, not a regex.

RouteText

Routes text data based on a set of user-defined rules. Each line in the incoming FlowFile is compared against values specified by user-defined Properties. The mechanism for comparing text to these properties is defined by the “Matching Strategy”. Data is then routed per these rules, routing each line of text separately.

RunMongoAggregation

A processor that runs an aggregation query whenever a flow file is received.

ScanAttribute

Scans specified attributes of FlowFiles, checking whether any value exists in a specified dictionary of terms.

ScanContent

Scans FlowFile content for terms found in a user-provided dictionary. If a term matches, the UTF-8 encoded version of that term is added to the FlowFile with a ‘matching.term’ attribute.

ScanHBase

Scans and extracts rows from an HBase table. Can fetch rows by specifying a range of rowkey values (start and/or end), a time range, a filter expression, or any combination. The order of records can be controlled by a property; Reversed can limit the number of rows retrieved.

ScrollElasticsearchHttp

Scrolls an Elasticsearch query using specified connection properties. Intended to run on the primary node and designed for scrolling large result sets, as in reindexing. State must be cleared before running another query. Each result page is returned wrapped in a JSON object like: {“hits”:[…]}. Note the full body of each page of documents is read into memory before being written to the flow file for transfer.

SegmentContent

Segments a FlowFile into multiple smaller segments at byte boundaries. Each segment has these attributes: fragment.identifier, fragment.index, fragment.count, segment.original.filename; the MergeContent processor can then use them to reconstruct the original FlowFile.

SelectHiveQL

Executes the provided HiveQL SELECT query against a Hive database connection. Results converted to Avro or CSV. Streaming supported. Can be scheduled on timer/cron or triggered by incoming FlowFile. If triggered, the FlowFile’s attributes are available when evaluating the select query. FlowFile attribute ‘selecthiveql.row.count’ indicates rows selected.

SetSNMP

Based on incoming FlowFile attributes, the processor executes an SNMP Set request. When establishing attributes named like snmp$, the processor attempts to set the attribute value to the corresponding OID given in the attribute name.

SplitAvro

Splits a binary-encoded Avro data file into smaller files based on a configured output size. The output strategy determines whether smaller files are Avro data files or bare Avro records with metadata in FlowFile attributes. Output is always binary-encoded.

SplitContent

Splits incoming FlowFiles by a specified byte sequence.

SplitJson

Splits a JSON file into multiple individual FlowFiles for the array elements specified by a JsonPath expression. Each resulting FlowFile consists of the specified array’s element and is transferred to the “split” relationship, the original to the “original” relationship. If the specified JsonPath is not found or doesn’t evaluate to an array element, the original is routed to “failure” and no file is generated.

SplitRecord

Splits an input FlowFile in a record-oriented data format into multiple smaller FlowFiles.

SplitText

Splits a text file into multiple smaller text files at line boundaries limited by a maximum line count or fragment total size. Each output split file contains no more than the configured number of lines or bytes. If both “Line Split Count” and “Maximum Fragment Size” are specified, splitting occurs at the first limit reached. If the first line of a fragment exceeds the maximum fragment size, that line is output in a single split file exceeding the configured maximum. This component also allows specifying that each split should contain a header row. The header row can be computed by specifying how many lines form the header or by matching a header marker against read lines. If such a match occurs, the corresponding line is treated as the header. Remember that once a header marker match fails for the first time, no further matching is done and the remaining data is parsed as regular lines for the given split. If no more data remains after computing the header, the resulting split contains only the header row.

SplitXml

Splits an XML file into multiple individual FlowFiles, each containing a child or descendant of the original root element.

SpringContextProcessor

A processor that supports sending and receiving data from an application defined in a Spring Application Context via predefined input/output MessageChannels.

StoreInKiteDataset

Stores Avro records in a Kite dataset.

TagS3Object

Sets tags on a FlowFile in an Amazon S3 bucket. If the file to tag does not exist, the FlowFile is routed to success.

TailFile

“Tails” a file or list of files, ingesting data from the file as it is written. The file should be text. Data is ingested only when a new line (carriage return or newline or combination) is encountered. If the tailed file periodically “rolls over” (usually the case for log files), an optional Rolling Filename Pattern can retrieve data from rolled-over files, even if the rollover happened while NiFi wasn’t running (as long as the data still exists when NiFi restarts). Generally recommend setting the run schedule to a few seconds rather than the default 0 seconds, since an aggressive schedule consumes many resources. At this time, this processor does not support retrieving files that were compressed at the time of “rollover”.

TransformXML

Applies the provided XSLT file to the flow file XML payload. Creates a new FlowFile with the transformed content and routes it to the “success” relationship. If XSL transformation fails, the original FlowFile is routed to “failure”.

UnpackContent

Unpacks the content of FlowFiles packaged with one of several different wrapping formats, sending one to many FlowFiles per input FlowFile.

UpdateAttribute

Updates the FlowFile’s attributes using the Attribute Expression Language and/or removes attributes based on a regular expression.

UpdateCounter

Lets the user set specific counters and key points in their flow. Useful for debugging and basic counting.

UpdateRecord

Updates the content of a FlowFile containing record-oriented data (i.e., data readable by a RecordReader and writable by a RecordWriter). This processor requires at least one user-defined property. The property name should indicate a RecordPath determining the field to update. The property value is the replacement value (optionally using Expression Language) or itself a RecordPath extracting a value from the Record. Whether the property value is a RecordPath or literal is determined by the property’s configuration.

ValidateCsv

Validates the FlowFile’s content against a user-specified CSV schema. See the processor’s other docs for schema examples.

ValidateRecord

Validates the incoming FlowFile’s records against a given schema. Records that follow the schema are routed to the “valid” relationship, those that don’t to the “invalid” relationship. Thus a single incoming FlowFile can be split into two if some records are valid and others not. Any FlowFile routed to “invalid” emits a ROUTE Provenance Event with the Details field explaining why the record is invalid. Additionally, to further explain invalid records, DEBUG-level logging can be enabled for the “org.apache.nifi.processors.standard.ValidateRecord” logger.

ValidateXml

Validates the FlowFile’s content against a user-specified XML Schema file.

Wait

Routes incoming FlowFiles to the “wait” relationship until a matching release signal is stored in the distributed cache from the corresponding Notify processor. When a matching release signal is recognized, the waiting FlowFile is routed to “success”, with attributes copied from the FlowFile that produced the release signal at the Notify processor. The release signal entry is then removed from the cache. If a Waiting FlowFile exceeds the Expiration Duration, it is routed to “expired”. If multiple signals must be waited for, specify the desired count via the “Target Signal Count” property. This is especially useful for processors that split a source FlowFile into multiple fragments (like SplitText). To wait for all fragments to be processed, connect the “original” relationship to the Wait processor along with the “split” relationship to the corresponding Notify processor. Configure the Notify and Wait processors to use ’${fragment.identifier}’ as the “Release Signal Identifier” value and specify ’${fragment.count}’ as the “Target Signal Count” in the Wait processor. When using the ‘wait’ relationship as a loop, a prioritizer (e.g., first-in-first-out) is recommended.

YandexTranslate

Translates content and attributes from one language to another.