Rahul Singhai Blog

wget and curl behind corporate proxy throws certificate is not trusted or certificate doesn't have a known issuer

2020-12-15T20:45:00.000+05:30

If you try to run wget or curl in Ununtu/Debian behind corporate proxy, you might receive errors like: ERROR: The certificate of 'apertium.projectjj.com' is not trusted. ERROR: The certificate of 'apertium.projectjj.com' doesn't have a known issuer. wget https://apertium.projectjj.com/apt/apertium-packaging.public.gpg ERROR: cannot verify apertium.projectjj.com's certificate, issued by '

ElasticSearch [FORBIDDEN/12/index read-only / allow delete (api)]

2019-06-03T21:29:00.001+05:30

While trying to index a document, if ElasticSearch throws following exception: elasticsearch.exceptions.AuthorizationException: AuthorizationException(403, 'cluster_block_exception', 'blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];') then, this means that ElasticSearch has cluster-wide locked writes to all indices. As a temporary solution, you can unlock writes to the cluster (

Securing ElasticSearch / Kafka clusters with SSL

2018-05-24T20:53:00.000+05:30

By default, there is no encryption, authentication, or ACLs configured in Elasticsearch/Kafka. Any client can communicate to ES nodes / Kafka brokers via the PLAINTEXT port. It is critical that access via this port is restricted to trusted clients only. Network segmentation and/or authorization ACLs can be used to restrict access to trusted IPs in such cases. If neither is used, the cluster is

Logstash throws error while installing plugins

2018-05-06T00:21:00.001+05:30

While trying to install logstash plugin, I was getting below error: $ /work/logstash/logstash-5.5.2/bin/logstash-plugin install logstash-input-cloudwatch WARNING: A maven settings file already exist at ~/.m2/settings.xml, please review the content to make sure it include your proxies configuration. Validating logstash-input-cloudwatch Installing logstash-input-cloudwatch Error Bundler::

Read and parse CSV containing Key-value pairs using Akka Streams

2018-01-30T20:29:00.000+05:30

Kafka performance tuning

2018-01-01T17:14:00.000+05:30

Performance Tuning of Kafka is critical when your cluster grow in size. Below are few points to consider to improve Kafka performance: Consumer group ID: Never use same exact consumer group ID for dozens of machines consuming from different topics. All of those commits will end up on the same exact partition of __consumer_offsets, hence the same broker, and this might in turn cause performance

Migrating ElasticSearch 2.x to ElasticSearch 5.x

2017-09-29T19:58:00.000+05:30

In my previous blog post, I described how to install and configure an ElasticSearch 5.x cluster. In this blog post, we will look at how to migrate data. Consult this table to verify that rolling upgrades are supported for your version of Elasticsearch. Full cluster upgrade (2.x to 5.x)We will have to do full cluster upgrade and restart. Install Elasticsearch Migration Helper on old cluster.

ElasticSearch max file descriptors too low error

2017-09-28T23:24:00.000+05:30

ElasticSearch 5.x requires a minimum of Max file descriptors 65536 and Max virtual memory areas 262144. It throws an error on start-up if these are set to very low value. ERROR: bootstrap checks failed max file descriptors [16384] for elasticsearch process is too low, increase to at least [65536] max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144] Check

ElasticSearch Curator

2017-09-28T22:43:00.000+05:30

Curator is a tool from Elastic to help manage your ElasticSearch cluster. For certain logs/data, we use one ElasticSearch index per year/month/day and might keep a rolling 7 day window of history. This means that every day we need to create, backup, and delete some indices. Curator helps make this process automated and repeatable. InstallationCurator is written in Python, so will need pip to

ElasticSearch cluster SSL/TLS configuration

2017-09-28T22:41:00.000+05:30

ElasticSearch X-pack documentation a good description on how to secure your ElasticSearch cluster using SSL/TLS. I used certgen to generate certificates for all the nodes as below: Create a instances.yml file: vim /work/elk/elasticsearch-5.6.2/config/x-pack/instances.yml instances: - name: "hostname-00" ip: - "192.126.0.163" - "192.0.2.2" - "198.51.100.1" dns:

Setting up ELK 5.x (ElasticSearch, Logstash, Kibana) cluster

2017-09-28T18:15:00.000+05:30

We recently upgraded from ElasticSearch 2.4.3 to 5.6.2. Below are the steps that we used to install and configure new ELK cluster. ElasticSearchInstallationmkdir -p /work/elk/data/data-es562 mkdir -p /work/elk/data/logs-es562 mkdir -p /work/elk/data/repo-es562 cd /work/elk/ curl -O https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.6.2.tar.gz tar -zxvf elasticsearch-5.6.2.

A sample Logstash config to connect to ElasticSearch with TLS

2017-09-27T20:26:00.000+05:30

Following up my previous blog post below is a sample Logstash config that can be used to connect to the ElasticSearch for the output of data: cd /work/elk/logstash-5.2.6/ vim ./config/twitter_feeds_consumer/twitter_feeds_consumer.conf input { kafka { topics => ["twitter_feeds_kafka_topic_name"] bootstrap_servers => "kafka-broker-1.domain.name:9092,kafka-broker-2.domain.name:9092" #

Aggregation based comparison in Watch condition and hard-coded variables

2017-07-27T20:24:00.000+05:30

Watch: Compare Vs Script conditionIf you need to do comparisons with aggregation buckets in watchers, then instead of this: "condition": { "compare": { "ctx.payload.aggregations.agg1.buckets.0.doc_count": { "gte": 5 } } } please prefer this, as this takes care of "Index out of bound" exceptions: "condition": { "script": { "source": "return (ctx.payload.aggregations.agg1.

Kafka MirrorMaker in Kafka 0.10.0.1+

2016-10-12T01:30:00.000+05:30

Check MirrorMaker.scala for more details. Target cluster setupDownload and install Kafka (target cluster). Select appropriate version and download its tgz from Kafka Downloads page. tar -zxf kafka_2.11-0.10.0.1.tgz cd kafka_2.11-0.10.0.1 Configure Target Kafka cluster's ZooKeeper vim ./config/zookeeper.properties # the directory where the snapshot is stored. dataDir=/work/kafka_2.11-0.10.0.1/

Generating Multi-Domain (SAN) Certificates

2016-09-29T20:31:00.000+05:30

The Subject Alternative Name field lets you specify additional host names (sites, IP addresses, common names, etc.) to be protected by a single SSL Certificate, such as a Multi-Domain (SAN) or Extend Validation Multi-Domain Certificate. Benefits Secure Host Names on Different Base Domains in One SSL Certificate: A Wildcard Certificate can protect all first-level subdomains on an entire domain,

ElasticSearch pipeline bucket selector aggregation

2016-07-29T21:02:00.000+05:30

ElasticSearch has a concept of bucket selection generated from aggregation. This works as a pipeline, where first aggregation generates buckets, and then bucket selection further filters out buckets. We have an ElasticSearch index 'daily_reports', where a row represents a particular version of report. When a report is created a new row is inserted in the index with a new 'reportId' field value

Logstash: Looping through nested JSON in ruby filter

2015-10-15T00:00:00.000+05:30

Suppose we have a JSON payload (may be a stream coming from Kafka) that looks like this: {"regionName":"asia","nodes":[{"node":"router.dc1.singhaiuklimited.com","weight":"0"},{"node":"router.dc2.singhaiuklimited.com","weight":"100"}]} {"regionName":"asia","nodes":[{"node":"router.dc1.singhaiuklimited.com","weight":"0"},{"node":"router.dc1.singhaiuklimited.com","weight":"0"},{"node":"router.dc2.

Add Environment variables and aliases to Bash on Mac OSX

2015-02-25T20:21:00.000+05:30

Refrences: 1. http://www.cyberciti.biz/tips/howto-linux-unix-bash-shell-setup-prompt.html 2. http://www.cyberciti.biz/faq/bash-shell-change-the-color-of-my-shell-prompt-under-linux-or-unix/ Add following in ~/.profile (or ~/.bash_profile, or ~/.bashrc, or /etc/profile if you have access): export JAVA_VERSION=1.8 export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_40.jdk/Contents/Home

mongoDB

2014-05-11T20:44:00.000+05:30

MongoDBA cross-platform open-source document-oriented NoSQL database with an Enterprise level distribution that includes additional security and management features. MongoDB is supported and steered by the company MongoDB. What do we mean by NoSQL?The NoSQL database category loosely incorporates all databases which do not have a rigid schema and are not queryable via a structured query language

Spark Tutorial and Cheatsheet

2013-07-01T16:05:00.000+05:30

Main resources: Scala Cheat Sheet Reactive Cheat Sheet Spark Cheat sheet Spark Quick start Spark programming guide Spark Streaming: processing real-time data streams Spark SQL and DataFrames: support for structured data and relational queries MLlib: built-in machine learning library GraphX: Spark’s new API for graph processing Scala programming examples: Define a object with main function --

Making Spray based ReST API publicly / remotely accessible

2013-03-16T18:29:00.000+05:30

Recently I developed a demo Spray based ReST API and ran it on Windows. While I was able to access it locally using http://localhost:port/service link, but remotely I was not. I made following two changes to fix the issue: 1. In your application's application.conf file make hostname as 0.0.0.0 - 127.0.0.1 or localhost is foe listening only on the loopback interface. - 0.0.0.0 for listening on all

SecurityException with Bouncy Castle

2013-02-16T19:16:00.000+05:30

I am using BouncyCastle library to encrypt the data transfer with Kakfa. While my program run fine from IntelliJ, but when I package a fat JAR and run, it throws following exception: java.lang.SecurityException: JCE cannot authenticate the provider BC at javax.crypto.Cipher.getInstance(Cipher.java:657) at javax.crypto.Cipher.getInstance(Cipher.java:596) at

Akka: Printing list of actors

2012-10-24T21:10:00.000+05:30

To print the list of actors that are running in the actor system, you can add following scheduler that will print actor path every 10 seconds: import akka.actor.{Actor, ActorIdentity, ActorLogging, ActorRef, Cancellable, Props} class TestActor extends Actor with ActorLogging { private var scheduler: Cancellable = _ @scala.throws[Exception](classOf[Exception]) override def preStart():

Scala simple concepts

2012-10-11T21:14:00.000+05:30

A simple iteration over Array in Scala can be done as: val xs:Array[String] = Array[String]("a", "b", "c", "d", "e") for((x,i) <- xs.iterator.zipWithIndex) println(i + "th element is " + x) for((x,i) <- xs.view.zipWithIndex) println(i + "th element is " + x) for (i <- xs.indices) println(i + "th element is " + xs(i)) for (i <- 0 until xs.length) println(i + "th element is " + xs(i))

Terminating a run or test in Scala Build Tool (SBT)

2012-09-19T23:00:00.000+05:30

When running a program or test in SBT, if you want to exit from it (and not from SBT) try adding following lines in your build.sbt file: fork := true To directly add forking in an SBT session, type set fork := true This will not detach and open a new consule from the original sbt console. But, users can kill this separately. The way to do this in Max OS or Windows is to search a Java process