Run Apache NiFi Cluster in Docker with SSL Enabled

Welcome to the fourth article in the series of Apache NiFi. The last article explained how to set up an Apache NiFi Docker container with a self-signed SSLcertificate. This article addresses the next pain point: how to create an Apache NiFi cluster in Docker with SSL enabled. Unlike HTTP cluster, setting up Apache NiFi cluster with SSL enabled in Docker introduces a new challenge: Hostname verification.

For added security, if HTTPS connection is enabled, Apache NiFi will verify the Hostname of requests. Therefore each request sent to Apache NiFi must have a predefined hostname. Not only the external requests, but peer-to-peer communication of NiFi nodes in a cluster also go through HTTPS and are subject to hostname verification. If the hostname provided in the HTTPS request does not match the hostname defined in the SSL certificate, NiFi will throw a javax.net.ssl.SSLPeerUnverifiedException.

If you are traditionally deploying Apache NiFi: individual servers with known IP addresses, it is easy to create certificates with those IP addresses. However, in a dynamic environment like Docker, the hostname of a container is defined at the runtime if you need flexible scaling options. Since Docker doesn't provide an option to define the hostname pattern in a scalable cluster, we have to stick to hard-coded Apache NiFi containers with predefined hostnames to create a cluster. The disadvantage of this method is that it is hard to scale up/down a cluster with hard-coded containers. Instead, you can also set up an HTTP cluster and create a load balancer with HTTPS frontend and SSL Termination between the client and NiFi UI. However, in this article, we will stick to the SSL configuration at the cluster level.

Read More

Run Apache NiFi in Docker with SSL Enabled

The last two articles in the Apache NiFi series discussed how to run Apache NiFi standalone server and NiFi cluster in Docker. However, those are far from production-ready because they are not secured. The next step in setting up a secured NiFi cluster is spinning up an Apache NiFi instance with SSL enabled in Docker. Though we are moving towards production-ready, this article will use self-signed certificates. In production, you should not use a self-signed certificate. In addition, you may also require additional safety measures like firewall and proxy.

Read More

Run Apache NiFi Cluster in Docker

The last article on Apache NiFi: Run Apache NiFi in Docker was for those who want to start playing with Apache NiFi. Though it was a good start to play with NiFi, it is far from production deployment. This article introduces the second stage of deployment: a NiFi cluster running in Docker using Docker Compose.

To begin with, you must have Docker installed in your system and also install Docker Compose as we are going to use Docker Compose to setup the Apache NiFi cluster. 

Read More

Run Apache NiFi in Docker

Running Apache NiFi in Docker is a simple and hassle-free process if you know the port to access. This article explains how to get Apache NiFi running in Docker on a Linux machine. If you don't have already, install Docker first.

Read More

Read and Write ORC Files in Core Java

The Optimized Row Columnar (ORC) file format provides a highly efficient way to store Hive data. It was designed to overcome the limitations of the other Hive file formats. Using ORC files improves performance when Hive is reading, writing, and processing data. There are hundreds of computing engine from Hive to Presto to read and write ORC files. When it comes to reading or writing ORC files using core Java, there is no enough help except the official document. This article is for you if you are looking forward to writing your own code to read or write ORC files.

 

In this article, we will create a simple ORC writer and reader to write ORC files and to read from ORC files. Later the ORC writer and the reader will be enhanced to support any common ORC types with some minor optimizations.

Requirements:

• Oracle JDK 8 or the latest version
• Apache Maven
• IntelliJ IDEA or Eclipse with Maven support
  

Read More