Run Apache NiFi Cluster in Docker with SSL Enabled

Welcome to the fourth article in the series of Apache NiFi. The last article explained how to set up an Apache NiFi Docker container with a self-signed SSLcertificate. This article addresses the next pain point: how to create an Apache NiFi cluster in Docker with SSL enabled. Unlike HTTP cluster, setting up Apache NiFi cluster with SSL enabled in Docker introduces a new challenge: Hostname verification.

For added security, if HTTPS connection is enabled, Apache NiFi will verify the Hostname of requests. Therefore each request sent to Apache NiFi must have a predefined hostname. Not only the external requests, but peer-to-peer communication of NiFi nodes in a cluster also go through HTTPS and are subject to hostname verification. If the hostname provided in the HTTPS request does not match the hostname defined in the SSL certificate, NiFi will throw a javax.net.ssl.SSLPeerUnverifiedException.

If you are traditionally deploying Apache NiFi: individual servers with known IP addresses, it is easy to create certificates with those IP addresses. However, in a dynamic environment like Docker, the hostname of a container is defined at the runtime if you need flexible scaling options. Since Docker doesn't provide an option to define the hostname pattern in a scalable cluster, we have to stick to hard-coded Apache NiFi containers with predefined hostnames to create a cluster. The disadvantage of this method is that it is hard to scale up/down a cluster with hard-coded containers. Instead, you can also set up an HTTP cluster and create a load balancer with HTTPS frontend and SSL Termination between the client and NiFi UI. However, in this article, we will stick to the SSL configuration at the cluster level.

Read More

Run Apache NiFi in Docker with SSL Enabled

The last two articles in the Apache NiFi series discussed how to run Apache NiFi standalone server and NiFi cluster in Docker. However, those are far from production-ready because they are not secured. The next step in setting up a secured NiFi cluster is spinning up an Apache NiFi instance with SSL enabled in Docker. Though we are moving towards production-ready, this article will use self-signed certificates. In production, you should not use a self-signed certificate. In addition, you may also require additional safety measures like firewall and proxy.

Read More

Run Apache NiFi Cluster in Docker

The last article on Apache NiFi: Run Apache NiFi in Docker was for those who want to start playing with Apache NiFi. Though it was a good start to play with NiFi, it is far from production deployment. This article introduces the second stage of deployment: a NiFi cluster running in Docker using Docker Compose.

To begin with, you must have Docker installed in your system and also install Docker Compose as we are going to use Docker Compose to setup the Apache NiFi cluster. 

Read More

Run Apache NiFi in Docker

Running Apache NiFi in Docker is a simple and hassle-free process if you know the port to access. This article explains how to get Apache NiFi running in Docker on a Linux machine. If you don't have already, install Docker first.

Read More

Read and Write ORC Files in Core Java

The Optimized Row Columnar (ORC) file format provides a highly efficient way to store Hive data. It was designed to overcome the limitations of the other Hive file formats. Using ORC files improves performance when Hive is reading, writing, and processing data. There are hundreds of computing engine from Hive to Presto to read and write ORC files. When it comes to reading or writing ORC files using core Java, there is no enough help except the official document. This article is for you if you are looking forward to writing your own code to read or write ORC files.

 

In this article, we will create a simple ORC writer and reader to write ORC files and to read from ORC files. Later the ORC writer and the reader will be enhanced to support any common ORC types with some minor optimizations.

Requirements:

• Oracle JDK 8 or the latest version
• Apache Maven
• IntelliJ IDEA or Eclipse with Maven support
  

Read More

Presto SQL for Newbies

In the series of Presto SQL articles, this article explains what is Presto SQL and how to use Presto SQL for newcomers. Presto is a high performance, distributed SQL query engine for big data. Its architecture allows users to query a variety of data sources such as Hadoop, AWS S3, Alluxio, MySQL, Cassandra, Kafka, and MongoDB. One can even query data from multiple data sources within a single query.

Let's begin with what is Presto. Presto is a massively parallel programming engine that allows users to execute against any databases. If you define a database as software that stores data and processes it, Presto does not fall under the database category. Rather I prefer to call it a data or computing engine because Presto itself does not provide a storage solution. Instead, Presto focuses on how to query different data sources such as MySQL, SQLServer, Hive, Cassandra even possibly CSV files. Presto achieves such flexibility of querying anything using its plugin architecture as shown below:

In the future if you find a new database to be supported by Presto, you only need to write a new connector to connect that database with Presto. Though it looks like connectors doing the heavy lifting here, actually connectors only provide simple API to connect to the database. For example, connectors tell Presto what are the tables available in the underlying database and how to read raw data from them. Given that information, Presto decides how to process those data and respond to a user's request. The coolest thing here is that you can join a table from one database with a table in another database. For example, consider a bank has account details in MySQL database and transaction history in Hive, they don't need to migrate data from one database to another to join them. Presto supports SQL like the following query out of the box:

SELECT acc.account_no as account_no, trans.amount
FROM mysql.bank.accounts acc LEFT JOIN hive.bank.transactions trans
    ON acc.account_no = trans.account_no
WHERE trans.amount > 1000;

Read More

Setup Presto SQL Development Environment

Presto SQL a massively parallel processing big-data engine grasps the attention of many big-data developers. This article is for those who like to set up a development environment for the Presto SQL community edition. The below-mentioned steps are applicable for any Presto variations including Presto DB with class names and file names replaced by equivalent names.

Requirements:

• Java Development Kit 8 (after March 2020, Presto SQL is using Java 11)
• IntelliJ IDEA Community/Ultimate Edition
• Latest Apache Maven (Older versions may not build Presto)
• Git

Read More

Presto SQL: Join Algorithms

Presto is a distributed big data SQL engine initially developed by Facebook and later open-sourced and being led by the community. The last article Presto SQL: Types of Joins covers the fundamentals of join operators available in Presto and how they can be used in SQL queries. With that knowledge, you can now learn the internals of Presto and how it executes join operations internally. This article presents how Presto executes join operations and the algorithms used to join tables.

Read More

Presto SQL: Types of Joins

SQL Join is one of the most important and expensive SQL operation and require deep understanding from database engineers to write efficient SQL queries. From database engineers' perspective, understanding how join operation works help them to optimize them for efficient execution. This article, explains the join operations supported in the open source distributed computing engine: Presto SQL. This article is based on now archived prestodb.rocks blog which I referred to learn the Join algorithms of Presto.

Read More

Read Carbondata Table from Apache Hive

Apache Carbondata an indexed columnar data store heavily depends on Apache Spark but also supports other Big Data frameworks like Apache Hive and Presto. This article explains how to read a Carbondata table created in Apache Spark from Apache Hive in two sections: 1. How to create a table in HDFS using Apache Spark, 2. How to read the Carbondata table from Apache Hive.

Requirements:

• Oracle JDK 1.8
• Apache Spark
• Apache Hadoop (Carbondata officially support Hive 2.x. In this article, Apache Hadoop 2.7.7 is used)
• Apache Hive (Carbondata officially support Hive 2.x. So better to stick to 2.x version. In this article, Apache Hive 2.3.6 is used to demonstrate the integration)
• Carbondata libraries

Please follow the Integrate Carbondata with Apache Spark Shell article to compile Carbondata from source and integrate it with Apache Spark. This article is written based on the assumption that you have already followed all the steps from the above-mentioned article.

Read More

Integrate Carbondata with Apache Spark Shell

Apache Carbondata an indexed columnar data store solution for fast analytics on big data platform, e.g.Apache Hadoop, Apache Spark, etc. This article is written to provide a quick start guide on how to integrate Carbondata with Apache Spark Shell. Why another article while there is a quick start guide on the official website? Things are not always as smooth as expected. In my experience, integrating Carbondata with Apache Spark using pre-built binaries didn't work as expected. So here is the quick start tutorial.

Requirements:
Carbondata requires Java 1.7 or 1.8 to run and Apache Maven to build from source. Please make sure that you have Oracle JDK 1.8, supporting Apache Maven and Git to setup Carbondata. If you don't have Oracle JDK or Apache Maven installed in your system, please follow the given links below to install them first.

• Install Oracle JDK 8 on Linux
• Install Apache Maven on Linux

Read More

Spark 06: Broadcast Variables

If you read the Spark 04: Key-Value RDD and Average Movie Ratings article, you might wonder what to do with popular movie IDs printed at the end. A data analyst cannot ask his/her users to manually check those IDs in a CSV file to find the movie name. In this article, you will learn how to map those movie IDs to movie names using Apache Spark's variable broadcasting.

Suppose you want to share a read-only data that can fit into memory with every worker in your Spark cluster, broadcast that data. The broadcasted variable will be distributed only once and cached in every worker node so that it can be reused any number of times. More about broadcasting will be covered later in this article after the code example.

Read More

Spark 05: List Action Movies with Spark flatMap

Welcome to the fifth article in the series of Apache Spark tutorials. In this article, you will learn the application of flatMap transform operation. After the introduction to flatMap operation, a sample Spark application is developed to list all action movies from the MovieLens dataset.

In the previous articles, we have used the map transform operation which transforms an entity into another entity where the transformation is one-to-one. For example, suppose you have a String RDD named lines, applying lines.map(x => x.toUpperCase) operation creates a new String RDD with the same number of records but with uppercase string literals as shown below:

Read More

Spark 04: Key-Value RDD and Average Movie Ratings

In the first article of this series: Spark 01: Movie Rating Counter, we created three RDDs (data, filteredData and ratingData) each contains a singular datatype. For example, data and filteredData were String RDDs and the ratingRDD was a Float RDD. However, it is common to use an RDD which can store complex datatypes especially Key-Value pairs depending on the requirement. In this article, we will use a Key-Value RDD to calculate the average rating of each movie in our MoviLens dataset.  Those who don't have the MovieLens dataset, please visit the Spark 01: Movie Rating Counter article to setup your environment.

As you already know, the ratings.csv file has the fields movieId and rating. A given movie may get different ratings from different users. To get the average ratings of each movie, we need to add all ratings of each movie individually and divide the sum by the number of ratings.

Read More

Spark 03: Understanding Resilient Distributed Dataset

You are not qualified as an Apache Spark developer until you know what is a Resilient Distributed Dataset (RDD). It is the fundamental technique to represent data in the Spark memory. There are advanced data representation techniques like DataFrame built on top of RDD. However, it is always better to start with the most basic dataset: RDD. RDD is nothing other than a data structure with some special properties or features.

We all know that Apache Spark is a distributed general-purpose cluster-computing framework. There are some common problems faced in a distributed environment including but not limited to:

1. Remote access of data is expensive
2. High chance of failure
3. Runtime errors are expensive and hard to track
4. Wasting computing power is way too expensive

RDD is designed to address the abovementioned problems. In the following section, you will see the properties of RDD and how it solves these problems.

Read More

Spark 02: Scala Cheat Sheet for Java Developers

This article introduces Scala to those Java developers who don't know Scala. I assume here that you already know Java (preferably Java 8) so that you can compare the features of Scala with Java. Please be informed that this article is not an end to end Scala tutorial. I am covering only the fundamentals of Scala which are used in my Apache Spark tutorials.

First of all, remember that Scala is a JVM based language which is running on top of your regular Java Virtual Machine. The whole purpose of Scala is providing a convenient functional programming language (at that time Java 8 wasn't there). However since Scala is built on top of Java, you can access Java libraries and API from your Scala code.

To play with Scala, please setup Scala on IntelliJ IDEA or install a command line Scala version in your system.

Read More

Spark 01: Movie Rating Counter

Apache Spark is a must to know framework for big data developers. This is an attempt to write a series of articles on Apache Spark to train you from zero to hero. In this series of articles, I will use the latest Apache Spark release which is 2.4.0 as of 2019 January. In the first few articles, we will code and test Apache Spark on IntelliJ Idea. As you may already know, Apache Spark is developed using Scala and of course, there are APIs available for other languages including Java and Python. Still, Scala has preferred over other languages for its performance and compact code. Therefore, you need to prepare the environment first.

My articles will be based on Frank Kane's course on Udemy: Apache Spark 2 with Scala - Hands On with Big Data! I highly recommend his tutorial if you prefer for a video tutorial.

Setup the Environment

Apache Spark 2.4.0 depends on Scala 2.12 which in turn depends on Java 1.8. Scala is unlike Java not very version compatible language. Therefore, please take special care on choosing versions.

Step 1:

Install Oracle Java Development Kit 1.8 on your system. Linux users can follow this article: Install Oracle JDK 8 on Linux

Step 2:

Install the latest IntelliJ IDEA. Again, Linux users can follow my article: Install IntelliJ IDEA on Linux

Step 3:
Install Scala plugin in IntelliJ IDEA. Regardless of your operating system, you can follow the article Setup Scala on IntelliJ IDEA.

Read More

Detect Absence of Events - WSO2 Siddhi Pattern

WSO2 Siddhi an opensource complex event processing engine which is used to power the WSO2 Analytics Server received a new feature from GSoC project: Non-Occurrence of Events for Siddhi Patterns. Until Siddhi 3.x, patterns can detect only the events that have arrived but building patterns based on events that have not arrived is an essential requirement and even there were questions on StackOverflow regarding this problem:

• Using Siddhi patterns to detect events that did not occur
• Using Siddhi patterns for events that haven't happened

This article introduces the new pattern to detect absence of events along with its limitations and sample use cases. In order to get a clear idea, let's begin with a sample use case.

This feature is available from Siddhi v4.0.0-M50. Those who prefer stable version have to wait until Siddhi v4.0.0.

Read More

WSO2 DAS - Hello World!

WSO2 Data Analytics Server is a smart analytics platform for both real-time and batch analytics. The real-time analytics is provided through their powerful open source Complex Event Processing Engine engine Siddhi. This article focuses on the complex event processing capability of the DAS server and provides a quick start guide on how to setup an event stream and process events generated by an HTTP source.

Prerequisite:

Oracle JDK 1.7 or latest

Memory – Minimum 2GB

Disk – Minimum 1 GB

For more details: Installation Prerequisites

Read More

Is WSO2 CEP Dead? No! Here’s Why…

During the WSO2 Con US 2017, a major business decision is announced. Due to some business decisions, WSO2 promotes the Data Analytic Server (DAS) (They may change this name very soon) over the complex event processor. For those who haven’t heard about DAS even though it has been there for a long period, it is another product of WSO2 which contains the Complex Event Processor for real-time analytics and some other components for batch processing and predictive analytics.

Then what is the future of WSO2 CEP?

Still, WSO2 CEP is available on their site but users are recommended to migrate to DAS. Since all the WSO2 products are developed using the same framework called Carbon and also due to their similar aesthetic design across all the platforms, the transition from CEP to DAS should be easy sledding even for the beginners. Let me repeat, DAS is a creamed version of CEP which provides all the features of CEP as it was in CEP and some additional features for batch processing. This short post is to announce that the existing articles about WSO2 CEP are not going to be outdated or useless due to this transition. You can simply download and extract the DAS pack and test all the features I’ve discussed so far using DAS.

If you still have confusion, just go and download the DAS pack from the official site. Extract the pack and start the server using the following command:

./bin/wso2server.sh

Go to the admin dashboard and you will see all the features of WSO2 CEP except the color of the top banner ;-)

Therefore, my upcoming articles about complex event processing will use WSO2 DAS not CEP.

Read More