Spring boot Admin

Featured

When the actuator introduced, the first thing came into my mind – What if we have a common place where I can see my all applications endpoints and I can manage them from there itself?

Thanks to the code-centric team they make this possible. if you are not aware of the Actuator, I would suggest you to first read about the actuator library.

Managing application using actuator endpoint is quite difficult, because if you have bunch of applications you won’t have a common place to see them all. But now with the help of Spring boot admin, it’s easier to view the dashboard for multiple microservices at a single place.

In this articles I’m going to describe you basic example of Spring boot admin client and server. CodeCentric has Introduce Spring boot Admin Server and it has inbuilt UI Dashboard that show the clients details. Isn’t it really amazing where you can see all your available clients at one place? Let’s see how we can implement this.

How it works?

Let’s do server and client setup

Spring boot admin server setup

First go to spring initialiser and generate sample project

start.spring.io screenshot to generate project

Once project generated you will find below dependencies into your build.gradle file.

'de.codecentric:spring-boot-admin-starter-server'

Now let’s enable Admin server in project using @EnableAdminServer annotation

@SpringBootApplication
@EnableAutoConfiguration
@EnableAdminServer
public class SbadminserverApplication {

	public static void main(String[] args) {
		SpringApplication.run(SbadminserverApplication.class, args);
	}

}

Logically dashboard should be secure, So let’s include basic authentication using Spring Security

implementation 'org.springframework.boot:spring-boot-starter-security'
import org.springframework.context.annotation.Configuration;
import org.springframework.security.config.annotation.web.builders.HttpSecurity;
import org.springframework.security.config.annotation.web.configuration.WebSecurityConfigurerAdapter;

@Configuration(proxyBeanMethods = false)
public class SecuritySecureConfig extends WebSecurityConfigurerAdapter {

    @Override
    protected void configure(HttpSecurity http) throws Exception {
        http.formLogin().loginPage("/login").permitAll();
        http.logout().logoutUrl("/logout").permitAll();
        http.csrf().ignoringAntMatchers("/actuator/**", "/instances/**", "/logout");
        http.authorizeRequests().antMatchers("/**/*.css", "/assets/**", "/third-party/**", "/logout", "/login")
                .permitAll();
        http.authorizeRequests().anyRequest().authenticated();
        http.httpBasic(); // Activate Http basic Auth for the server
    }
}

Now assign default password to your server, if we don’t define Spring security password it will generate a random password at runtime and you can see the generated password in console.

spring.security.user.name=admin
spring.security.user.password=admin

If you want to update server ‘Titles’ you can use below properties.

spring.boot.admin.ui.title=AdminConsole

Spring boot admin client setup

Generate client project

Once the project generated, configure server details into client properties files, make sure if you running both on the same machine, the port should be different to avoid a port binding error.

spring.application.name=sb-client
server.port=8888
management.endpoints.web.exposure.include=*

spring.boot.admin.client.url=http://localhost:8080
spring.boot.admin.client.username=admin
spring.boot.admin.client.password=admin

Now run both of the project and login into the server.

Spring boot admin server dashboard

Source link

Summary : If you are looking for a prebuilt dashboard for your microservices, you can consider it one of the options. As this project is under an open-source umbrella you can update some of the features as per your requirement.

Hope you like this 🙂

Featured

Spring boot actuator

Actuator

An actuator is a spring boot sub-project that helps to expose production-ready support features against Spring boot application.

Key features offered by actuator

  • Health check : You can use health endpoint to check the status of your running application.
  • Monitoring and Management over HTTP/JMX : Actuator support HTTP endpoint as well as Java Management Extensions (JMX) to provide a standard mechanism to monitor and manage applications.
    • Logger: It provide feature to view and update the logs level.
    • Metrics: Spring Boot Actuator provides dependency management and auto-configuration for Micrometer, an application metrics facade that supports numerous monitoring systems.
    • Auditing: Once Spring Security is in play, Spring Boot Actuator has a flexible audit framework that publishes events (by default, “authentication success”, “failure” and “access denied” exceptions). This feature can be very useful for reporting and for implementing a lock-out policy based on authentication failures.
    • Http Tracing: HTTP Tracing can be enabled by providing a bean of type HttpTraceRepository in your application’s configuration. For convenience, Spring Boot offers an InMemoryHttpTraceRepository that stores traces for the last 100 request-response exchanges
    • Process Monitoring

Enable Actuator into Spring boot project

You can enable Actuator into Spring boot project by including below dependency.

//Gradle
org.springframework.boot:spring-boot-starter-actuator:2.3.1.RELEASE
//Maven
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-actuator</artifactId>
    <version>2.3.1.RELEASE</version>
</dependency>

Endpoint offer by Actuator

By default ‘health’ and ‘info’ endpoint are enabled

Default exposed endpoint

Other endpoints are sensitive and not advisable to expose to the production environment without security. As we are demonstrating, let’s expose all the APIs.

management.endpoints.web.exposure.include=*

Include and Exclude Endpoint

Even you can include or exclude endpoint by defining below properties

# wild card to include/exclude all
management.endpoints.web.exposure.include=* 
management.endpoints.web.exposure.exclude=* 

# you can include specific properties like below
management.endpoints.web.exposure.include=env,beans
management.endpoints.web.exposure.exclude=heapdump

Customise management server address

You can customize the management server port, it will help you define the limited scope to the ports.

management.server.port=8081
management.server.address=127.0.0.1

Expose custom endpoint

Any methods annotated with @ReadOperation@WriteOperation, or @DeleteOperation are automatically exposed over JMX and HTTP. Even you can expose technology specific endpoint by using @JmxEndpoint or @WebEndpoint.

Here i’m share you example for exposing endpoint using Spring boot 2.x

import org.springframework.boot.actuate.endpoint.annotation.ReadOperation;
import org.springframework.stereotype.Component;

@Component
@org.springframework.boot.actuate.endpoint.annotation.Endpoint(id = "say-hello")
public class Endpoint {

    @ReadOperation
    public String sayHello()
    {
        return "Hello World";
    }

}

Summary : Spring boot actuator is one of the best libraries you can add in your application to enable production-ready features in less effort. it offers key features that can be used in day to day production support.

Featured

Secure properties with spring cloud config

Overview: Earlier post I was demonstrating basic of spring cloud config. Now in this post, we will see how we can use secure properties in the spring cloud config. Before continuing on this, I recommend you to first go through the previous post.

Secure properties: Almost every application has some kind of configuration that can’t be exposable, some are very sensitive and should be limited access. we generally pronounce it secure properties. 

There are multiple ways you can secure your properties like using Cerberus, Hashicorp vault with consul backend, Cyberark password vault, and aim, confidant, Credstash etc. Here we are going to use the simplest way that may not as powerful as the above tools but secure and very easy to use.

Let’s implement this, in a sample code,

First, generate a key store. assuming you are aware of keytool.

keytool -genkeypair -alias mytestkey -keyalg RSA \ -dname "CN=Web Server,OU=Unit,O=Test,L=City,S=State,C=IN" \ -keypass changeme -keystore server.jks -storepass testPassword

Then place server.jks to resource folder in cloud config server project, after that edit bootstrap.properties(or .yaml). and add below properties.

Properties Description
encrypt.keyStore.locationContains a resource(.jks file) location
encrypt.keyStore.passwordHolds the password that unlocks the keystore
encrypt.keyStore.aliasIdentifies which key in the store to use
encrypt.keyStore.secret secret to encrypt or decrypt

e.g

encrypt.keyStore.location=server.jks encrypt.keyStore.password=testPassword encrypt.keyStore.alias=mytestkey encrypt.keyStore.secret=changeme

That’s it and now restart config server. 

There is encryption and decryption endpoint expose by config server. 

Let’s take a simple example, where we will try to encrypt and decrypt ‘testKey’.

#curl localhost:8888/encrypt --data-urlencode testKey

output:

AQAYqm8ax79kPFGT0sOvV8i8uN0GDLsToULmflVNYKf95bpyAKLIV4eCFVdNJpgb7SyS808a3uTjvQBj1SrIwFlQktRpln8ykpWUG3NdPM6aPf5k4yRhNkG43S5lCckmyLTH8CIzoSSFQeKoFuk4zPiAPTMchTP9qtAYG2EwbdWU1/a9xqoDJb9OQbSsEr0wp2Ud+HlG02NGF2qmhxL7kW5BJxTsGdZG2J8qwhkPYreYF6UQlehmheWCAJBzfBw4peT9LOxi7rA0sHD78xle7Bahziyc+WOETADloKfSowERNY5FCOe4/ywhcHpJuCk+6NPok3KVI+jMTXdSpqMmfxBNc764hHjlhpablwNcRPDv8XGCdstdy4Esb9/eXTZgh0g=

#curl localhost:8888/decrypt --data-urlencode AQAYqm8ax79kPFGT0sOvV8i8uN0GDLsToULmflVNYKf95bpyAKLIV4eCFVdNJpgb7SyS808a3uTjvQBj1SrIwFlQktRpln8ykpWUG3NdPM6aPf5k4yRhNkG43S5lCckmyLTH8CIzoSSFQeKoFuk4zPiAPTMchTP9qtAYG2EwbdWU1/a9xqoDJb9OQbSsEr0wp2Ud+HlG02NGF2qmhxL7kW5BJxTsGdZG2J8qwhkPYreYF6UQlehmheWCAJBzfBw4peT9LOxi7rA0sHD78xle7Bahziyc+WOETADloKfSowERNY5FCOe4/ywhcHpJuCk+6NPok3KVI+jMTXdSpqMmfxBNc764hHjlhpablwNcRPDv8XGCdstdy4Esb9/eXTZgh0g=

output:

testKey

As our keystore file containing public and private key so we can able to encrypt and decrypt properties.

In the case of config client, we do not have to do any extra step except below one, whenever we use encrypted property that has to start with ‘{cipher}’, for example

user.password={cipher}5lCckmyLTH8CIzoSSFQeKoFuk4zPiAPTMchTP9qtA

Caution: encrypted data should not be within single or double quotes.

In a case when a client wants to decrypt configuration locally

First,

If you want to encrypt and decrypt endpoint not work, Then comment all properties that start with encrypt.* and include the new line as below

spring.cloud.config.server.encrypt.enabled=false

Include keystore(.jks) file in client project and update below properties in bootstrap.properties(or .yaml) file

encrypt.keyStore.location=server.jks
encrypt.keyStore.password=testPassword
encrypt.keyStore.alias=mytestkey
encrypt.keyStore.secret=changeme

That’s all, now client project not going to connect with the server to decrypt properties.

Summary: We can secure our external properties using spring cloud config in fewer efforts, That may easily fulfill small or mid-scale project requirement.

Hope you like this 🙂

Featured

Spring cloud config

Overview: In this tutorial, we will cover the basics of Spring cloud config server and client, where you will set up your cloud config server and access configuration through client services.

What is Spring cloud config?

Spring cloud config provides server and client side support for externalized configuration a distributed system.

Why Spring cloud config?

External configuration is the basic need of almost any application. Now we are leaving in the microservices world, that need external configuration more often. Almost every individual application has some external and dynamic properties that keep updating based on the environment or some period of time.  In case of the massive size of services, it is hard to manage or left with creating a custom solution for it.

The dilemma is how we can make our application to adopt dynamic changes more often without investing time into it and with zero downtime. Here spring cloud config comes into the picture. It will help a user to manage their external properties from multiple sources like git, local file system etc. Transfer data into encrypted form and lot more.

How it works?

Screen Shot 2018-08-26 at 9.22.06 PM.png

Above diagram, you can see multiple services that get configuration from config server where config server sync with GIT or any other VCS to get new changes.

Let’s create a simple application, that will take a message from the config server and return through the endpoint.

Implementation of spring cloud config server:

Now I’m going to visit  spring initializer page to generate demo project, where I’ve included dependencies of spring ‘web’ and ‘config server’.

Once project imported into IDE, enable config server with ‘@EnableConfigServer’ annotation

@SpringBootApplication
@EnableConfigServer
public class DemoApplication {

   public static void main(String[] args) {
      SpringApplication.run(DemoApplication.class, args);
   }
}

In this example, I’m using local git  for demo purpose

#mkdir cloud-config
#git init
#touch config-client-demo.properties
#vi config-client-demo.properties 
message=Hello World
//wq (write and quite vi editor)
#git add config-client-demo.properties
#git commit -m "initial config"

By default, config server starts on port 8080 as spring boot application.

Here is config server application.properties

server.port=8888
spring.cloud.config.server.git.uri=${HOME}/Desktop/cloud-config

server.port: We use this property to define server port if you do not define it will set 8080 as a default port. Here I’m using the same machine for client and server so one of the port needs to be updated.

spirng.cloud.config.server.git.uri: We define this property to fetch configuration detail from git repository.

Implementation of client application:

Now I’m again going to generate a template application using spring initializer where included ‘web’, ‘client config’ and ‘actuator’ as dependencies. Here ‘actuator’ help us to refresh configuration.

Below is message controller that return message and that message came from config properties, you will notice @RefreshScope annotation that helps to refresh configuration.

@RestController
@RefreshScope
public class MessageRestController {

    @Value("${message}")
    private String message;

    @GetMapping("/message")
    String getMessage() {
        return this.message;
    }
}

Then rename your ‘applcation.properties‘ file to ‘bootstrap.properties‘ and include below config,  I’ve defined application name(name of application should accurate to fetch specific data), Configuration URI(which your config server URL that holds properties details) and expose all actuator endpoint that by default disabled

spring.application.name=config-client-demo
spring.cloud.config.uri=http://localhost:8888

management.endpoints.web.exposure.include=*

spring.application.name: Accurate name of the configuration file that defined on config server.

spring.cloud.config.uri: Base url of config server

management.endpoints.web.exposure.include: By default actuator’s most features are disabled so use a wildcard to enable all. That actually not needed in case of production.

Run your application and you will see below message.

GET http://localhost:8080/message

Response :
Hello world

Let’s update the properties file, I’m going to update ‘Hello World’ to ‘Hello Test’

#vi config-client-demo.properties
message= Hello Test
...
#git add config-client-demo.properties
#git commit -m "update message"
//Once file updated then 'refresh' configuration

Once you included the @RefreshScope annotation that will expose endpoint for you to refresh application, check below.

POST http://localhost:8080/actuator/refresh

Now again hit your message API

GET http://localhost:8080/message
Response:
Hello Test

checkout example at git

Conclusion: We can use spring cloud config to any kind of application for external, distributed and centralized configuration server. There is no limitation of technology and programming languages. You can apply spring cloud config in other languages as well.

Hope you like this tutorial.

 

Featured

Getting started with MongoDB

mongodb_logo.jpg

Hello Everyone,

In this post, I am going to present some of the basic syntax and example of MongoDB to get started with it. For basic detail of NoSQL db visit this link.

What is MongoDB?

Mongo db is an open source, cross-platform NoSql database. It is a document-oriented db which is written in C++.

Mongo db stores its data on the filesystem. It stores all the data in BSON (Binary JSON). The format of BSON documents is very similar to the Object Oriented Programming. In MongoDB we can store complete information in one document rather than creating different tables for them and then define a relationship between them.

Let’s take a brief look at terms which mongo db uses to store the data:

  • Collections: You need to create collections in each Database. Each DB can contain multiple collections.
  • Documents: Each Collection contains multiple Documents.
  • Fields: Each Document contains multiple Fields.

 

Now get started with the commands:

Database:

  • Create db or Use db: There is no command to create a db in Mongo. Whenever we want to create a new db use following command.
    • Syntax: use <dbname>
    • Example: use customerdb
  • Show current db : This is very simple and small command
    • Syntax: db
  • Show db : This command will return the existing dbs only if it contains at least one collection.
    • Syntax: show dbs

At this time we did show dbs it will return only a default db. For this, we need to add a collection to it. We will see how to add collection in collection section. But for now, presenting an example:

    • Example: db.customer({first_name:”Robin”}
    • show dbs

Now we can see our db customerdb in the list.

  • Drop db: To drop database following is the command. Before deleting database first select the db.
    • Syntax: db.dropDatabase()
    • Example: use customerdb
      db.dropDatabase()

Collections

  • Create collection : Mongo db normally we do not need to create collection explicitly. When we write a command to insert the document it will create the collection if does not exist. But there is a way to create collection explicitly an define it as expected:
    • Syntax: db.createCollection(<collectionName>, option)
      db.createCollection(<name>, { capped: <boolean>,
      autoIndexId: <boolean>,
      size: <number>,
      max: <number>,
      storageEngine: <document>,
      validator: <document>,
      validationLevel: <string>,
      validationAction: <string>,
      indexOptionDefaults: <document>,
      viewOn: <string>,
      pipeline: <pipeline>,
      collation: <document> } )
    • Example: db.createCollection(customer)

Collection name’s type is String and option type is Document.
Some of the important fields I am describing below:

Field(optional) Type Description
capped boolean If it sets to true it creates capped collection. For capped collection we need to define the size as well.
size number It defines the size of the capped collection. If the documents size reached to its limit then on each insert mongo db started deleting the old entries.
max number This defines the max size of capped collection. If the size is defined less then max size mongo db will start deleting the old document. So we need to ensure that max size is always less then size.
autoIndexId boolean It automatically creates index on id.

 

  • Drop Collection: We can drop a collection by using the following index but before dropping any collection we should be in the same db.
    • Syntax: db.<collectionName>.drop()
    • Example: use customerdb
      db.customer.drop()

CRUD Operations:

Mongo db provides very flexibility for CRUD operations. We can insert or update document on the fly.

  • Insert Document:
    • Syntax: db.<collectionName>.insert(<documents>)
    • Example:  db.customer.insert([{first_name:”Robin”, last_name:”Desosa”}, {first_name:“Kanika”, last_name:”Bhatnagar”},{first_name:”Rakesh”, last_name:”Sharma”, gender:”male”}]);

In the above example we are adding 3 document, first 2 are having the same fields but the third document has an additional field gender. Mongo db provides a functionality to insert the non structural data.
When you insert a document mongo db will automatically create a unique Id for each document.

  • Update Document:
    • Syntax: db.<collectionName>.update({<documentIdentifier>}, {$set:{<update value>}})
    • Example: db.customer.update({first_name:”Robin”},  {$set:{gender:”male”}});
      db.customer.update({first_name:”Kanika”},  {$set:{gender:”female”}});db.customer.update({first_name:”Rakesh”}, {$set:{age:”25”}})The above example will add a new field in corresponding document.
  • Update or Insert: Upsert command updates the document if it already exists or inserts a new one.
    • Syntax: db.<collectionName>.upsert({<docIdentifier>}, {<document>}, {upsert:true})
    • Example: db.customer({first_name:”Amita”}, {first_name:” Amita”, last_name:”Jain”, gender:”female”}, {upsert: true});
  • Rename Field in Document: We can rename field of a specific document by using $rename in update command.
    • Syntax: db.<collectionName>. update({<documentIdentifier>}, {$rename:{<update value>}})
    • Example: db.customer.update({first_name:”Rakesh”}, {$rename:{“gender”:”sex”}});After this we renamed the gender field to sex only for the document whose first_name is “Rakesh”.
  • Remove a field: To remove a field $unset needs to be used in update command.
    • Syntax: db.<collectionName>.update({<documentIdentifier >}, {$unset:{<field:1>}})
    • Example: Db.customer.update({first_name:”Rakesh”}, {$unset:{age:1}});
  • Remove Document:
    • Syntax: db.<collectionName>.remove({<documentIdentifier >})
    • Example: db.customer.remove({first_name:”Amita”});
      (If we have multiple entries with first_name Amita and want to remove 1.)
      db.customer.remove({first_name:”Amita”}, {justOne:true});
  • Find Document: We can find the document in collection by using following command. The output of that command is an object in json form.
    • Syntax: db.<collectionName>.find()
    • Example: db.customer.find();

The output of the above command will be all the json object stored in that collection.
To see it in a formatted way like each object and field in new line we can use pretty on find.

Example: db.customer.find().pretty();

  • Find Specific: By passing the documentIdentifier value in find method.
    • Syntax: db.<collectionName>.find({<documentIdentifier >})
    • Example: db.customer.find({first_name:”Kanika”});
  • Or condition: 
    • Example: db.customer.find({$or:[{first_name:”Kanika”}, {first_name:”Robin”}]});

In the above example we have a document in find as parameter, and in that document we have give an array of first_name. $or is defining the operation which is going to be performed on the array.

  • Greater than, Less than: We can directly jump on the example of the greater than and less than.

Example:

  • db.customer.find({age:{$gt:26}});
    In the above example $gt defines that > operation need to be perform on age. It will find and print all the documents who has this field age and age >26.
  • db.customer.find({age:{$lt:26}});
    In the same way this $lt will help us to find all documents which have age field and age < 26.
  • db.customer.find({age:{$gte:26}});
    We can perform >= or <= operations as well by using $gte and $lte.

Following are some more example on features provided by Mongo db:

  • Sort:
    • db.customer.find().sort({first_name:1}); //descending order
      db.customer.find().sort({first_name:-1}); //ascending order
  • Count:
    • db.customer.find().count();
  • ForEach:
    • db.customer.find().forEach(function(doc){print(“Customer Name:”+ doc.first_name)});

These all are the basic syntax for getting started with MongoDb.

 

 

 

 

Featured

Relational to NoSQL database

Hey Everyone,

Today I’m going to share you the other experience that I’ve struggled with. I know after reading this post most of you will relate your issues with my experience and you will find a new way to deal with them.

The pain of relational database in a small project where you no need of transaction support and also not defined any complex relationship between the multiple entities.  We are very habitual with relation database we always trying to go with them. But here I’m trying to stop you, first check NoSQL, if they not meet your requirement. Then go with relational one.

Let me explain!!

What is NoSQL?

NoSql is basically a database used to manage huge sets of unstructured data, where data is not stored in a tabular relation like a relational database.

Why NoSQL?

A traditional database prefers more predictable and structured data. The technologies are increasing and we want to automate everything, in such situations we want to gather all type of data to be more predictable. Every time getting structural data is a bit difficult. Many times we need to deal with such situations where we need to store unstructured data. Here comes NoSQL as a savior.

NoSQL as per name says NOT ONLY SQL that comes with lots of more added benefit. It provides a completely new way then the Relational DB to deal with data. Managing any type of data is very simple in NoSQL then relational.

Types of NoSQL database

  • Document type: In this database, the key is paired with a complex data structured called as Document. Example: MongoDB
  • Graph stores: This types of a database is usually used to store networked data. Where we can relate data based on some existing data. Example: Neo4J
  • Key-Value stores: These are the simplest NoSql database. In this database each record store with a unique key to identify it. Example: Aerospike
  • Wide column-stores: This type of database store large data set. Example: Cassandra.

Let’s see one of the use cases (using MongoDB and MySQL):

Imagin that we have beer store database that holds beer store basic information, like beer style, beer type, and beer store addresses.

Let’s see a basic schema design in case of the relational database.

Here I have created beer_store table that holds store names then created address table that holds beer store addresses (one store can be available in multiple places)  along with country and city table. then I’ve defined beer_style and beer_type that provided by the beer stores. then defined many to many relationships between beer type and beer_store and the same in case of beer_style and beer store also.

Screen Shot 2018-04-15 at 9.06.53 PM.png

Now same beer store we can achieve in case of NoSQL (JSON-like documents). below schema.

There is two way you can design your NoSQL schema.

  1. You can store all the data into the same collection. but there is some disadvantage when some data repeat multiple times.
  2. You can store a reference to data little bit similar to SQL.

Here to avoiding confusion, I’ve separated few tables where information is fixed and that will repeat in case of another beer store record.

beer_store

Screen Shot 2018-04-15 at 2.47.28 PM.png

and other tables that hold values and references

beer_style

Screen Shot 2018-04-15 at 4.32.42 PM.png

beer_type

Screen Shot 2018-04-15 at 4.33.26 PM.png

country

Screen Shot 2018-04-15 at 2.51.41 PM.png

Query Example SQL VS NoSQL

#SQL

select * from beer_store;
//output
+------------+-------------+
| pkstore_id | name        |
+------------+-------------+
|          1 | Magic Beer  |
|          2 | Beerland    |
|          3 | Uptown Beer |
+------------+-------------+

#NoSQL

db.beer_store.find()
//output
{
	"_id" : ObjectId("5ad31853e5ce6fe732eb7300"),
	"name" : "The Beera",
	"address" : [
		{
			"address1" : "sec 4",
			"address2" : "shastri cercle",
			"city" : "jaipur",
			"country" : "India"
		}
	],
	"beerType" : [
		"Ales",
		"Lagers"
	],
	"beerStyle" : [
		"Amber",
		"Blonde"
	]
}
{
	"_id" : ObjectId("5ad4bfa2b20cf32e1fa3cb6d"),
	"name" : "BeerLand",
	"address" : [
		{
			"address1" : "sec 4",
			"address2" : "hiran magri",
			"city" : "udaipur",
			"country" : "India"
		}
	],
	"beerType" : [
		"Ales",
		"Lagers"
	],
	"beerStyle" : [
		"Amber",
		"Blonde"
	]
}

Above both SQL and NoSQL query requested for beer_store but as per both are different way persisted result return based on it.

Same NoSQL response we can achieve with relation database but for this, we have to do multiple joins, subqueries and inbuilt JSON converter.

Conclusion: Relation database has their own importance like supporting the ACID properties. they are well organized.  But many cases NoSQL appreciable, like when we do search operation, sorting operation filtering on one-to-many and many-to-many relationships. 

Hope you like this. Thanks for reading 🙂

Kafka: Publish and Consume messages

In my earlier posts, I have explained about Kafka and the how to install and run Kafka on your system.

Now we will see how to publish and consume messages in Kafka.

Step 1: Create a Topic

As we know in Kafka publisher publish messages to topic and Kafka will decides this message will be assigned to which partition in a topic.

So first we will create a topic named greetings. For this lets open a new command prompt and navigate to bin/windows folder. Then by using kafka-topic.bat we can create a topic.

Notice the bootstrap server port 9092. This is the default port of Kafka server.

$ kafka-topics.bat --create --topic greetings --bootstrap-server localhost:9092

So now we have successfully created the topic.

We can pass the –describe parameter to kafka-topic.bat to get the information about the topic.

$ kafka-topics.bat --describe --topic greetings --bootstrap-server localhost:9092

Step 2: Publish some events

Now let’s write some message or publish some event to the topic.

To do so open a new command prompt and navigate to bin/windows and type below command.

$ kafka-console-producer.bat --topic greetings --bootstrap-server localhost:9092

Then type the message you want to publish. By default each line you enter will trigger a separate event to the topic.

We can stop the publisher any time by pressing Ctrl + C.

Step 3: Consume events

Open an another terminal and by using kafka-console-consumer.bat you will be able to consume messages.

$ kafka-console-consumer.bat --topic greetings --from-beginning --bootstrap-server localhost:9092

Great 👏

Now you are publishing and consuming messages using Kafka.

Summery :

In this article we have demonstrate how you can create topic in Kafka and produce and consume messages by using Kafka’s producer and consumer console library.

Prev -> Kafka: Install and Run Apache Kafka on windows

Kafka: Install and Run Apache Kafka on windows

Install Apache Kafka on Windows

STEP 1: Install JAVA SDK >8

For this we need java-jdk installed on our system.

STEP 2: Download and Install Apache Kafka binaries

You can download the Apache Kafka binaries from Apache kafka official page:

https://kafka.apache.org/downloads

STEP 3: Extract the binary

Extract the binary to some folder. Create a ‘data‘ folder at bin level.

Inside data folder create zookeeper and kafka folder.

STEP 4: Update configuration value

Update zookeeper data directory path in “config/zookeeper.Properties” configuration file.

With the zookeeper folder path that you have created in data.

Update Apache Kafka log file path in “config/server.properties” configuration file.

STEP 5:  Start Zookeeper

Now we will start zookeeper from command prompt. Go to kafka bin\windows and execute zookeeper-server-start.bat command with config/zookeeper.Properties configuration file.

Here we are using default properties that already bundled with kafka bindary and persist into the config folder. later we can update this according to our uses.

To validate if zookeeper starts successfully check for below logs.

STEP 6:  Start Apache Kafka

Finally we will start Apache Kafka from command prompt just in the same way we started zookeeper. Open an another command prompt, run kafka-server-start.bat command with kafka config/server.properties configuration file.

Summery:

To proceed with kafka you need install and run kafka and zookeeper server on your machine. with the above steps.

Next-> Kafka: Publish and Consume messages

Prev-> Kafka: Introduction to Kafka

Kafka: Introduction to Kafka

In this world of data where things and systems started depending on data, it is very important to get the right data at a right time to get the most of it. In this a great architecture of data streaming – “Apache Kafka” has introduced in 2011

Here I am brining a short course for Kafka where try to provide a basic understanding of Kafka with it’s core architecture and some hands-on on the producer consumer code.

So let’s get started 😊

What is Kafka?

Apache Kafka was originated at LinkedIn and later became an open-sourced Apache project in 2011,  then a first-class Apache project in 2012. Kafka is written in Scala and Java.

Apache Kafka is a publisher-subscriber concept based on a fault-tolerant messaging system. It is fast, scalable, and distributed by design.

“Kafka is an Event Streaming architecture.”

Event streaming is capturing data in real-time from various event sources like databases, cloud services, software applications, etc.

Why Kafka?

Kafka is a messaging system. This is typically suits for the application that requires high throughput and low latency. It can be used for real-time analytics.

Kafka can work with Flume/Flafka, Spark Streaming, Storm, HBase, Flink, and Spark for real-time ingesting, analysis and processing of streaming data. Kafka is a data stream used to feed Hadoop BigData lakes. Kafka brokers support massive message streams for a low-latency follow-up analysis in Hadoop or Spark.

Basics of Kafka:

Apache.org states that:

  • Kafka runs as a cluster on one or more servers.
  • The Kafka cluster stores a stream of records in categories called topics.
  • Each record consists of a key, a value, and a timestamp.

Key Concepts :

Events and Offset :

Kafka uses Log data structure to store the Event/Messages. Each message/Event has a unique Key. Kafka ensures that the message should not be duplicate and must be in sequence.

Offsets are the pointers to understand from where data needs to be picked.

Events/Messages can stay in the partition for very long period and even forever.

Topic and Partitions :

Topic is a uniquely defined category in which producer publishes messages.

Each topic contain one or many partitions. Partitions contains messages.

Messages are written to topics and kafka uses round robin to selects which partition to write the message to.

To make sure that some particular type of messages should go to same partition we can assign Key to the messages, attaching a key to messages will ensure messages with the same key always go to the same partition in a topic. Kafka guarantees order within a partition, but not across partitions in a topic.

Cluster and Broker :

Kafka cluster can have multiple brokers inside it, to maintain load balancing. A single Kafka server is called as Kafka broker. Kafka cluster is stateless hence to maintain cluster state Kafka uses Zookeeper.

I’ll cover zookeeper in the next point. For now let’s understand what is broker.

Broker receives messages from producer and assign offset to it and then store it on local disk.

Broker is also responsible to serve message fetch request coming from consumer.

Each broker contains one or more Topics. Each topic along with their partitions can be assigned to multiple broker but the owner or leader will be only one.

For example in the below diagram Partition 0 is replicated along with topic X in Broker 1 and Broker 2, but the leader will always be only one. The replica is used as a backup of partition. So that if any particular broker fails then the replicator takes leadership.

Producer and consumer only connects to the Leader partition.

Zookeeper:

Kafka uses Zookeeper to maintain and coordinate between brokers.

Zookeeper is also sends notification to the Producer and consumer about the presence of any new broker or if any new leader created. So that according to that they can make decision and start coordinating  the task accordingly.

Consumer Group:

A consumer group is a platform where we can have multiple consumers. Each consumer group has one unique Id.

Only one consume in the group can pull the messages from a particular partition. Same consumer group can not have multiple consumers of same partition.  

Multiple consumers can consume messages from same partition but they must be from different consumer groups.

If the consumers are more in same group and partitions are less then there are changes to have some inactive consumers in the group.

Summery:

Kafka is an event based messaging system. Mostly suited for applications where big amount of real time data needs to be processed.

In the complete architecture of Kafka it provides load balancing, data backup, maintain message order, facility to read messages from a particular position, message storage for longer period, message can be fetched by multiple consumers of different groups.

Next -> Kafka: Install and Run Apache Kafka on windows

Install spring boot application on linux

Today most of the people prefer to deploy the standalone application and easy to manageable instead of doing too much overhead or get expert advice on it to manage a specific web server for rest application.

dilemma: When we have a standalone jar there are few minor hacks we have to do this with it. like, configure log path, start with a specific promised user. run it with standalone process instead of inline execution.

A fully executable jar can be executed like any other executable binary or it can be registered with ‘systemd‘.  This makes it very easy to install and manage spring boot applications in common production environments.

#systemd: systemd is system and service manager for linux operating system. When run as the first process on boot (as PID 1), it acts as init system that brings up and maintains userspace services. it is a replacement of ‘Unix system v’ and BSD (Berkeley Software Distribution) init system. for more detail #man systemd

Assume that you have a linux server that has Java installed, now place your executable spring boot jar file in a /opt/apps/ directory

Now create a app.service file in a /etc/systemd/system
directory and execute below command also make sure your permission before creating a file at a particular location
#vi /etc/systemd/system/app.service

and paste script

[Unit]
Description=sample application
After=syslog.target

[Service]
User=appUser
ExecStart=/opt/apps/app.jar
SuccessExitStatus=143

[Install]
WantedBy=multi-user.target

now save above script using “:wq” for more detail check vi editor helps(#man vi)

let’s enable service

#systemctl enable app.service
(check #man systemctl for more detail.)

once service enables now you can start, stop and check status of your service.

#systemctl start app.service
#systemctl stop app.service

Hope you like this. Thanks for reading 🙂