Simplicity is always a choice! Keep it simple.

Ubuntu 24.0 LTS - Install OpenConnect

2024-09-18T14:13:00.000-07:00

My organization uses Cisco AnyConnect to connect to the VPN. Problem with using AnyConnect client is that it doesn't save the password. So I try to use OpenConnect instead.

I just upgraded to Ubuntu 24 LTS. When I do Settings --> Network --> VPN --> Add, it doesn't show Multi-protocol VPN client option which will let me connect to my Org VPN.

So I have to do this

sudo apt-get install openconnect network-manager-openconnect network-manager-openconnect-gnome

After a quick restart, I can see that option.

Golang - How to "extend" an interface while using Generics

2023-01-05T07:56:00.001-08:00

Here is a code snippet that shows how to do it:

package main

import "fmt"

type Parent[T any] interface {
    ParentMethod1(t T)
    ParentMethod2(t T)
}

type Child[T string] interface {
    Parent[T]
    ChildMethod1(t string)
    ChildMethod2(t string)
}

type parent[T any] struct {
}

func NewChild() Child[string] {
    return & child[string]{
        parent[string]{ },
    }
}

func(a * parent[T]) ParentMethod1(t T) {
    fmt.Println("ParentMethod1", t)
}

func(a * parent[T]) ParentMethod2(t T) {
    fmt.Println("ParentMethod2", t)
}

type child[T string] struct {
    parent[T]
}

func(b * child[T]) ChildMethod1(t T) {
    fmt.Println("ChildMethod1", t)
}

func(b * child[T]) ChildMethod2(t T) {
    fmt.Println("ChildMethod2", t)
}

func(a * child[T]) ParentMethod1(t T) {
    fmt.Println("Overrided ParentMethod1", t)
}

func main() {
    ch:= NewChild()
    AcceptParent[string](ch)
}

func AcceptParent[T any](a Parent[T]) {
    fmt.Println("Works", a.ParentMethod2)
}

Golang - How to dynamically set and get values using reflection

2022-11-10T08:53:00.001-08:00

This code shows how to set and get values dynamically:

package main

import (
    "fmt"
    "reflect"
)

type Car struct {
    Id int
    Color string
    Make string
    Model string
    Miles int64
}

func main() {
    c: = new(Car)
    SetAttributeValue(c, "Miles", int64(50000))
    SetAttributeValue(c, "Id", 100)
    SetAttributeValue(c, "Make", "Toyota")
    SetAttributeValue(c, "Model", "Camry")
    fmt.Println("Car:", c)
    fmt.Println("Miles:", GetAttributeValue(c, "Miles"))
}

func GetAttributeValue(s any, fieldName string) any {
    r: = reflect.ValueOf(s)
    va: = reflect.Indirect(r).FieldByName(fieldName)
    return va.Interface().(any)
}

func SetAttributeValue(s any, fieldName string, v any) {
    r: = reflect.ValueOf(s)
    va: = reflect.Indirect(r).FieldByName(fieldName)
    if va.CanSet() {
        va.Set(reflect.ValueOf(v))
    }
}

VSCODE - How to navigate cursor to previous location

2022-11-09T13:52:00.008-08:00

On Windows:

Alt + ← ... navigate back

Alt + → ... navigate forward

On Mac:

Ctrl + - ... navigate back

Ctrl + Shift + - ... navigate forward

On Ubuntu Linux:

Ctrl + Alt + - ... navigate back

Ctrl + Shift + - ... navigate forward

Upgraded to ubuntu 22.x and can't ssh into some of your servers?

2022-09-02T20:38:00.004-07:00

Well, I upgraded to Ubuntu 22.x and I couldn't ssh into some of my servers. I ran the following command to see why:

ssh -vvv user@host

The output of above command showed why I wasn't able to ssh. Saw this line:

debug1: send_pubkey_test: no mutual signature algorithm

This gives a hint about the algorithm you used to create your ssh keys is no longer supported.

Fix is to add the following to /etc/ssh/ssh_config on the client machine:

PubkeyAcceptedKeyTypes +ssh-rsa

This should fix the issue.

How to retrieve schema-free MongoDB documents

2020-12-08T14:57:00.002-08:00

If you are on consumer side of MongoDB, there is a very good chance that the schema of the documents being written into MongoDB is not under your control. If you are as unlucky as me, the producer of the documents will add random fields without telling you. There are 2 options here. First options is that you only retrieve set number of fields and drop everything else. This is also called schema-on-read. If you cannot afford to drop any fields (even if they are randomly added), you will need the second option which is what I try to explain here.

The solution is to read the documents as a Map. To be specific org.bson.Document (implements java.util.Map). You cannot use MongoRepository anymore. You need to use MongoTemplate that will allow you to pass entity class type like this. Bonus: The snippet also shows how to do pagination and sorting.

String userId = "kamal.bahadur@gmail.com";

// Sort by "timestamp" (desc - latest first) and return first doc
PageRequest pageRequest = new PageRequest(0, 1, new Sort(Sort.Direction.DESC, "timestamp"));

List<Document> users = mongoTemplate.find(new BasicQuery("{'userId':'"+userId+"'}").with(pageRequest), Document.class, "users");
if (users != null && users.size() > 0) {
	System.out.println(users.get(0));
}

How to write enum type with a dash (hyphen) or a space (whitespace) into Postgres

2020-05-04T08:00:00.000-07:00

I am currently working on an ETL project that reads from Kafka and writes into Postgres (12.x). This is a Spring boot app with Spring JPA for DB access.

Disclaimer: I have never worked with enum types before. If you find a better solution, let me know in the comments.

First, I thought I will just write the data as String and as long as the value matched the enum values it should be ok. I was wrong. I quickly realized that I need to use enum as the type in my model. Like this (Note the attribute answerIndic):

import lombok.Data;
import javax.persistence.*;import java.math.BigDecimal;import java.sql.Timestamp;
@Entity@Data@Table(name = "call_log")
public class CallLogCdr {
    @Id    @Column(name = "record_id")
    private String recordId;
    @Column(name = "user_id")
    private String userId;
    @Column(name = "start_time")
    private Timestamp startTime;
    @Column(name = "answer_time")
    private Timestamp answerTime;
    @Column(name = "release_time")
    private Timestamp releaseTime;
    @Column(name = "direction")
    @Enumerated(EnumType.STRING)
    private Direction direction;
    @Column(name = "calling_number")
    private String callingNumber;
    @Column(name = "called_number")
    private String calledNumber;
    @Column(name = "answer_indic")
    private AnswerIndic answerIndic;
}

In Postgres, the accepted values for enum are:

Yes
No
Yes-PostRedirection

I tried to create a Java enum type with those values and I got compile errors. Turns out I can't use space or dash in the name. It wasn't hard to figure out I should be using AttributeConverter to convert between Java Enum and DB Enum. Thanks to google!

So, here is my converter:

import javax.persistence.AttributeConverter;import javax.persistence.Converter;
@Converterpublic class AnswerIndicConverter implements AttributeConverter<AnswerIndic, String> {

    @Override    public String convertToDatabaseColumn(AnswerIndic answerIndic) {
        if (answerIndic == null) {
            return null;        }
        return answerIndic.toString();

    }

    @Override    public AnswerIndic convertToEntityAttribute(String name) {
        if (name == null) {
            return null;        }
        return AnswerIndic.fromString(name);    }
}

and, here is my enum:

public enum AnswerIndic {
    YES("Yes"),    NO("No"),    YES_POSTREDIRECTION("Yes-PostRedirection");
    AnswerIndic(String name) {
        this.name = name;    }

    private final String name;
    @Override    public String toString() {
        return this.name;    }

    public static AnswerIndic fromString(String name ) {
        if ("No".equals(name)) {
            return AnswerIndic.NO;        } else if ("Yes".equals(name)) {
            return AnswerIndic.YES;        } else if ("Yes-PostRedirection".equals(name)) {
            return AnswerIndic.YES_POSTREDIRECTION;        }
        return null;    }
}

Now, I had to annotate the attribute answerIndic in the model to use the converter like this:

@Column(name = "answer_indic")
@Convert(converter = AnswerIndicConverter.class)
private AnswerIndic answerIndic;

At this point, I thought I was done. But no....the universe was working against me that day. I got this error message:

Caused by: org.postgresql.util.PSQLException: ERROR: column "answer_indic" is of type answer_indic_enum but expression is of type character varying
Hint: You will need to rewrite or cast the expression.

I tried to google that error message and most of the posts stated that I should be using an AttributeConverter which I was already doing. I spent 5 hours trying different things and nothing worked. I went back and read every single post/article/stackoverflow that was returned by google search. I must have gone through 2 or 3 pages of search results and finally found the solution.

Well, at first I didn't think it would work, but then I got nothing to lose so I tried and voila!

The solution is to add

stringtype=unspecified

to the datasource url. Like this:

spring.datasource.url=jdbc:postgresql://localhost:5432/postgres?sslmode=require&stringtype=unspecified

I hope this helps someone. See you in the next Post.

How to convert your training mask to deal with Corona Virus

2020-03-23T12:27:00.001-07:00

Things needed:

Training Mask https://amzn.to/2UeOHXW
Air Filter https://amzn.to/2xhSD1d
Scissors

Make sure the filter you buy filters out Bacteria and Virus.

Remove the filter material from the packaging. It is bit tricky but doable.

Cut three 4X4 inch squares of the filter.

Remove the 3 white thingy on the mask.

This white thingy cam be opened to remove a little rubber flap. Remove this and keep it safe. We won't need it to make Corona mask :)

Place the 4X4 inch filter paper on the mask and push the white thingy back in like this:

Put the outer sleeve of the mask back on

That's all!

Be safe out there!

Stitch Multiple MP4 files together in Linux

2020-01-01T19:49:00.000-08:00

To stitch multiple MP4 files into one big MKV file, you will have to install mkvtoolnix

sudo apt install mkvtoolnix

Then, run this command:

mkvmerge -o final.mkv file1.mp4 \+ file2.mp4 \+ file3.mp4

Send and Reply Message using Gmail Api

2019-12-31T10:30:00.000-08:00

I recently worked with Gmail API to send and receive emails. The complex part was sending and replying to messages. The following class has all necessary code to be able to get the following features:

Compose a new message.
Reply to a message. Parameters “threadId”, “messageId” and “references” are required to create a reply. Read this to understand how this works.
Supports messages in plain text.
Supports HTML messages.
Supports messages in HTML with alternative plain text as fallback.
Supports multiple attachments.

import com.google.api.services.gmail.Gmail;
import com.google.api.services.gmail.model.Message;

import javax.activation.DataHandler;
import javax.activation.DataSource;
import javax.activation.FileDataSource;
import javax.mail.MessagingException;
import javax.mail.Multipart;
import javax.mail.Session;
import javax.mail.internet.InternetAddress;
import javax.mail.internet.MimeBodyPart;
import javax.mail.internet.MimeMessage;
import javax.mail.internet.MimeMultipart;
import java.io.ByteArrayOutputStream;
import java.io.File;
import java.io.IOException;
import java.nio.file.Path;
import java.util.Base64;
import java.util.List;
import java.util.Objects;
import java.util.Properties;

public class GmailApiClient {

    private static final String SUBJECT = "Subject";
    private static final String FROM = "From";
    private static final String MESSAGE_ID = "Message-ID";
    private static final String REFERNCES = "References";
    private static final String IN_REPLY_TO = "In-Reply-To";

    public void sendMessage(Gmail client, String clientId, String clientSecret, String accessToken,
                            String refreshToken, String email, String to, String cc, String bcc,
                            String subject, String text, String html, String threadId, String messageId,
                            String references, List<Path> localFiles) {
        try {
            Message message = createMessageWithEmail(createEmail(to, cc, bcc, email, subject, text, html, messageId, references, localFiles), threadId);
            client.users().messages().send("me", message).execute();
        } catch (Exception e) {
            e.printStackTrace();
        }
    }

    private MimeMessage createEmail(String to, String cc, String bcc, String from, String subject,
                                    String text, String html, String messageId, String references,
                                    List<Path> localFiles) throws MessagingException {

        MimeMessage message = getMimeMessage(to, cc, bcc, from, subject, messageId, references);

        // Without attachments
        if (localFiles == null) {
            // Text
            if (text != null && html == null) {
                message.setText(text);
            }
            // Html
            else if (html != null && text == null) {
                message.setContent(getHtmlMultiPart(html));
            }
            // Html with alternative text
            else if (html != null && text != null) {
                message.setContent(getPlainTextAndHtmlMultiPart(text, html, false));
            }
        }
        // With attachments
        else if (localFiles != null) {

            Multipart multipart = null;
            // Text
            if (text != null && html == null) {
                multipart = getPlainTextMultiPart(text);
            }
            // Html
            else if (html != null && text == null) {
                multipart = getHtmlMultiPart(html);
            }
            // Html with alternative text
            else if (text != null && html != null) {
                multipart = getPlainTextAndHtmlMultiPart(text, html, true);
            }

            // Add attachments
            for (Path path : localFiles) {
                File file = path.toFile();
                MimeBodyPart attachmentPart = new MimeBodyPart();
                DataSource source = new FileDataSource(file);

                attachmentPart.setDataHandler(new DataHandler(source));
                attachmentPart.setFileName(file.getName());

                multipart.addBodyPart(attachmentPart);
            }

            message.setContent(multipart);
        }
        return message;
    }

    private Multipart getPlainTextMultiPart(String textBody) throws MessagingException {
        Multipart multipart = new MimeMultipart();
        MimeBodyPart mimeBodyPart = new MimeBodyPart();
        mimeBodyPart.setContent(textBody, "text/plain");
        multipart.addBodyPart(mimeBodyPart);
        return multipart;
    }

    private Multipart getPlainTextAndHtmlMultiPart(String text, String html, boolean hasAttachments) throws MessagingException {
        Multipart multipart = null;
        if (hasAttachments) {
            multipart = new MimeMultipart("mixed");

            Multipart innerMultiPart = new MimeMultipart("alternative");
            MimeBodyPart mbp = new MimeBodyPart();
            mbp.setContent(innerMultiPart);
            multipart.addBodyPart(mbp);
            addAlternativeTexts(innerMultiPart, text, html);
        } else {
            multipart = new MimeMultipart("alternative");
            addAlternativeTexts(multipart, text, html);
        }

        return multipart;
    }

    private void addAlternativeTexts(Multipart multipart, String text, String html) throws MessagingException {
        MimeBodyPart textPart = new MimeBodyPart();
        textPart.setText(text, "utf-8");

        MimeBodyPart htmlPart = new MimeBodyPart();
        htmlPart.setContent(html, "text/html; charset=utf-8");

        multipart.addBodyPart(textPart);
        multipart.addBodyPart(htmlPart);
    }

    private Multipart getHtmlMultiPart(String htmlBody) throws MessagingException {
        Multipart multipart = new MimeMultipart();
        MimeBodyPart htmlPart = new MimeBodyPart();
        htmlPart.setContent(htmlBody, "text/html; charset=utf-8");
        multipart.addBodyPart(htmlPart);
        return multipart;
    }

    private MimeMessage getMimeMessage(String to, String cc, String bcc, String from, String subject, String messageId, String references) throws MessagingException {
        Properties props = new Properties();
        Session session = Session.getDefaultInstance(props, null);

        MimeMessage message = new MimeMessage(session);

        if (Objects.nonNull(messageId)) {
            message.addHeader(IN_REPLY_TO, messageId);

            if (Objects.nonNull(references)) {
                message.addHeader(REFERNCES, references + " " + messageId);
            } else {
                message.addHeader(REFERNCES, messageId);
            }
        }

        message.setFrom(new InternetAddress(from));

        if (Objects.nonNull(to)) {
            message.addRecipients(javax.mail.Message.RecipientType.TO, to);
        }

        if (Objects.nonNull(cc)) {
            message.addRecipients(javax.mail.Message.RecipientType.CC, cc);
        }

        if (Objects.nonNull(bcc)) {
            message.addRecipients(javax.mail.Message.RecipientType.BCC, bcc);
        }

        // Set subject
        if (Objects.nonNull(messageId)) {
            if (subject.toUpperCase().startsWith("RE: ")) {
                message.setSubject(subject);
            } else {
                message.setSubject("Re: " + subject);
            }
        } else {
            message.setSubject(subject);
        }

        return message;

    }

    private Message createMessageWithEmail(MimeMessage emailContent, String threadId)
            throws MessagingException, IOException {
        ByteArrayOutputStream buffer = new ByteArrayOutputStream();
        try {
            emailContent.writeTo(buffer);
        } catch (Exception e) {
            e.printStackTrace();
            throw e;
        }
        byte[] bytes = buffer.toByteArray();
        String encodedEmail = Base64.getUrlEncoder().encodeToString(bytes);
        Message message = new Message();
        message.setRaw(encodedEmail);
        if (Objects.nonNull(threadId)) {
            message.setThreadId(threadId);
        }
        return message;
    }
}

GTX 1060 GPU HDMI Autio Unplugged

2019-09-21T11:26:00.001-07:00

I have been trying to get a system 76 laptop work with my TV. The video worked just fine but could not get the audio to work with HDMI. PulseAudio would show the audio card but all the HDMI options would be "Unplugged". After lot of trial and error, I found a solution that worked for me.

First I ran this command to see the different audio cards I have in the laptop:

pacmd list-cards

Output of the above command will look something like this:

2 card(s) available.
    index: 0
    name: <alsa_card.pci-0000_01_00.1>
    driver: <module-alsa-card.c>
    owner module: 7
    properties:
        alsa.card = "1"
        alsa.card_name = "HDA NVidia"
        alsa.long_card_name = "HDA NVidia at 0xdc080000 irq 17"
        alsa.driver_name = "snd_hda_intel"
        device.bus_path = "pci-0000:01:00.1"
        sysfs.path = "/devices/pci0000:00/0000:00:01.0/0000:01:00.1/sound/card1"
        device.bus = "pci"
        device.vendor.id = "10de"
        device.vendor.name = "NVIDIA Corporation"
        device.product.id = "10f1"
        device.product.name = "GP106 High Definition Audio Controller"
        device.string = "1"
        device.description = "GP106 High Definition Audio Controller"
        module-udev-detect.discovered = "1"
        device.icon_name = "audio-card-pci"
    profiles:
        output:hdmi-stereo: Digital Stereo (HDMI) Output (priority 5400, available: unknown)
        output:hdmi-surround: Digital Surround 5.1 (HDMI) Output (priority 300, available: unknown)
        output:hdmi-surround71: Digital Surround 7.1 (HDMI) Output (priority 300, available: unknown)
        output:hdmi-stereo-extra1: Digital Stereo (HDMI 2) Output (priority 5200, available: no)
        output:hdmi-surround-extra1: Digital Surround 5.1 (HDMI 2) Output (priority 100, available: no)
        output:hdmi-surround71-extra1: Digital Surround 7.1 (HDMI 2) Output (priority 100, available: no)
        output:hdmi-stereo-extra2: Digital Stereo (HDMI 3) Output (priority 5200, available: no)
        output:hdmi-surround-extra2: Digital Surround 5.1 (HDMI 3) Output (priority 100, available: no)
        output:hdmi-surround71-extra2: Digital Surround 7.1 (HDMI 3) Output (priority 100, available: no)
        off: Off (priority 0, available: unknown)
    active profile: <output:hdmi-surround71>
    sinks:
        alsa_output.pci-0000_01_00.1.hdmi-surround71/#3: GP106 High Definition Audio Controller Digital Surround 7.1 (HDMI)
    sources:
        alsa_output.pci-0000_01_00.1.hdmi-surround71.monitor/#3: Monitor of GP106 High Definition Audio Controller Digital Surround 7.1 (HDMI)
    ports:
        hdmi-output-0: HDMI / DisplayPort (priority 5900, latency offset 0 usec, available: yes)
            properties:
                device.icon_name = "video-display"
                device.product.name = "SAMSUNG
     "
        hdmi-output-1: HDMI / DisplayPort 2 (priority 5800, latency offset 0 usec, available: no)
            properties:
                device.icon_name = "video-display"
        hdmi-output-2: HDMI / DisplayPort 3 (priority 5700, latency offset 0 usec, available: no)
            properties:
                device.icon_name = "video-display"
    index: 1
    name: <alsa_card.pci-0000_00_1f.3>
    driver: <module-alsa-card.c>
    owner module: 8
    properties:
        alsa.card = "0"
        alsa.card_name = "HDA Intel PCH"
        alsa.long_card_name = "HDA Intel PCH at 0x2ffff20000 irq 133"
        alsa.driver_name = "snd_hda_intel"
        device.bus_path = "pci-0000:00:1f.3"
        sysfs.path = "/devices/pci0000:00/0000:00:1f.3/sound/card0"
        device.bus = "pci"
        device.vendor.id = "8086"
        device.vendor.name = "Intel Corporation"
        device.product.id = "a171"
        device.product.name = "CM238 HD Audio Controller"
        device.form_factor = "internal"
        device.string = "0"
        device.description = "Built-in Audio"
        module-udev-detect.discovered = "1"
        device.icon_name = "audio-card-pci"
    profiles:
        input:analog-stereo: Analog Stereo Input (priority 60, available: unknown)
        output:analog-stereo: Analog Stereo Output (priority 6000, available: unknown)
        output:analog-stereo+input:analog-stereo: Analog Stereo Duplex (priority 6060, available: unknown)
        output:iec958-stereo: Digital Stereo (IEC958) Output (priority 5500, available: unknown)
        output:iec958-stereo+input:analog-stereo: Digital Stereo (IEC958) Output + Analog Stereo Input (priority 5560, available: unknown)
        off: Off (priority 0, available: unknown)
    active profile: <output:analog-stereo>
    sinks:
        alsa_output.pci-0000_00_1f.3.analog-stereo/#1: Built-in Audio Analog Stereo
    sources:
        alsa_output.pci-0000_00_1f.3.analog-stereo.monitor/#1: Monitor of Built-in Audio Analog Stereo
    ports:
        analog-input-internal-mic: Internal Microphone (priority 8900, latency offset 0 usec, available: unknown)
            properties:
                device.icon_name = "audio-input-microphone"
        analog-input-mic: Microphone (priority 8700, latency offset 0 usec, available: no)
            properties:
                device.icon_name = "audio-input-microphone"
        analog-output-lineout: Line Out (priority 9900, latency offset 0 usec, available: no)
            properties:

        analog-output-speaker: Speakers (priority 10000, latency offset 20000 usec, available: unknown)
            properties:
                device.icon_name = "audio-speakers"
        analog-output-headphones: Headphones (priority 9000, latency offset 0 usec, available: no)
            properties:
                device.icon_name = "audio-headphones"
        iec958-stereo-output: Digital Output (S/PDIF) (priority 0, latency offset 0 usec, available: unknown)
            properties:

Note the index number of the card that has hdmi. In my case, it was the first one with index 0.

I then ran this command:

pacmd set-card-profile 0 output:hdmi-stereo

"0" in the above command is the index number. "output:hdmi-stereo" came from the list of profiles.

Once I ran the above command, all the hdmi options became available.

Java Swing UI Looks Tiny

2019-09-16T10:11:00.000-07:00

You probably have very high resolution screen and everything looks tiny.

Just add this JVM parameter when you run the application and things will start looking normal.

-Dsun.java2d.uiScale=2

If you are starting the app from within Intellij, your run configuration should look like this:

I tested this on my laptop running Ubuntu 18.x.

Install protoc 2.5.0 on Ubuntu 18.x

2019-09-11T15:59:00.000-07:00

sudo apt-get install build-essential
mkdir /tmp/protobuf_install
cd /tmp/protobuf_install
wget https://github.com/protocolbuffers/protobuf/releases/download/v2.5.0/protobuf-2.5.0.tar.gz 
tar xzvf protobuf-2.5.0.tar.gz
cd  protobuf-2.5.0
./configure
make
make check
sudo make install
sudo ldconfig
protoc --version

Docker Cheatsheet

2019-08-16T13:31:00.002-07:00

Command to list the running containers:

docker ps

Command to check the logs from the host server:

docker logs <container name>

Command to ssh into a running docker container:

docker exec -it <container name> /bin/bash

Command to stop a running container: (get the container id using above command)

docker stop <container_id>

Linux - Connect to SFTP Sever

2018-12-14T12:57:00.005-08:00

sftp -oPort=<port if different than default> user@host:/

Slack App for Mac stuck in overtype mode instead of usual insert mode

2018-11-16T06:18:00.000-08:00

Use ⌘+R to do a quick refresh and reset the mode.

How to Unset a TTL from HBase Table

2018-09-27T08:25:00.000-07:00

Here is the command:

alter '<Table Name>', {NAME => '<Column Family>', TTL => org.apache.hadoop.hbase.HConstants::FOREVER}

Pandas Dataframe - Easy way to group by and calculate the count, mean and standard deviation

2018-07-18T08:30:00.002-07:00

The following command groups by col_1 and calculates the count, mean and standard deviation for the values in col_2

df[['col_1', 'col_2']].groupby(['col_1']).agg(['count', 'mean', 'std'])

Official documentation: http://pandas.pydata.org/pandas-docs/stable/groupby.html#applying-multiple-functions-at-once

Spring Boot - Embeded Tomcat : How to limit the number of requests processed and queued

2018-06-19T10:14:00.002-07:00

own voteacceptedS

One of my REST applications ran into a problem where the requests started timing out. I had this parameter set:

server.tomcat.max-threads = 500

And everything else was default value. I was running embedded Tomcat 8.5.x.

Some of my APIs were really long running ones (could very well take over a minute). When a lot of long running api requests came in they clogged up the server and occupied all 500 worker threads. In an ideal situation, I would have expected the server to return HTTP 503 (Service Unavailable) to any new request beyond the set 500, but it didn't.

I started digging into various options to set in tomcat and found these:

server.tomcat.max-connections
server.tomcat.accept-count

I created a simple rest application with an api endpoint which just sleeps for 2 minutes and started testing using jmeter to find out how the above parameters behave.

I found that the total number of requests accepted for processing will be

server.tomcat.max-connections + server.tomcat.accept-count

However, only server.tomcat.max-threads were being processed concurrently.

I am still not sure how to return HTTP 506. If you do, please comment below. I would appreciate!

Using these parameters, I now have a workaround to my problem, which is to fail-fast requests exceeding max-threads + accept-count.

Here is my workaround:

set server.tomcat.max-threads and server.tomcat.max-connections to the same value (which is the number of concurrent requests to be processed). And set server.tomcat.accept-count to a number that you want to be "queued". This number depends on how much time you want your client to wait if the worker threads are busy for a long time. Also, ask your clients to set connect timeout to something like 100 milli seconds to fail-fast. I think the default is 1 minute.

Cheers!

Error loading coprocessor from HBase Shell

2018-05-24T09:39:00.000-07:00

Are you getting this when loading your coprocessor: "cannot be loaded Set hbase.table.sanity.checks to false at conf or table descriptor if you want to bypass sanity checks"?

I got that error message when I tried to load my second endpoint coprocessor on a table.

I started with the following check-list:

Does the jar file exist
Permissions of the jar file
Checked fully qualified class name

Damn, it still doesn't work. I spent hours reading books and articles about how to load coprocessor. I tried every possible method but I still kept getting that error.

In my case, I had multiple coprocessors in the same jar file. That was the problem (even though I don't agree why it matters). I moved coprocessors to their own fat jar file and everything worked!

I hope this saves someone's day. Cheers!

Java Client To Read Data from HA Enabled HDFS

2018-02-07T08:19:00.003-08:00

If you have enabled HA (High Availability) in HDFS, you have 2 Name Nodes now and only one of them is active at any given time. First option to read data from HDFS is to connect to one of the Name Nodes and if it isn't active (got exception), connect to the second Name Node. Alternatively, you could use the following, which is a better option IMO:

Start by creating an xml file like this (call it hdfs-site.xml) and add this to your classpath. Replace namenode1.mycompany.com with the host or IP of your first name node and replace namenode2.mycompany.com with the host or IP or your second name node.

<?xml version="1.0" encoding="UTF-8"?>

<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://nameservice1</value>
    </property>
    <property>
        <name>dfs.nameservices</name>
        <value>nameservice1</value>
    </property>
    <property>
        <name>dfs.client.failover.proxy.provider.nameservice1</name>
        <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
    </property>
    <property>
        <name>dfs.ha.automatic-failover.enabled.nameservice1</name>
        <value>true</value>
    </property>
    <property>
        <name>dfs.ha.namenodes.nameservice1</name>
        <value>namenode1,namenode2</value>
    </property>
    <property>
        <name>dfs.namenode.rpc-address.nameservice1.namenode1</name>
        <value>namenode1.mycompany.com:8020</value>
    </property>
    <property>
        <name>dfs.namenode.servicerpc-address.nameservice1.namenode1</name>
        <value>namenode1.mycompany.com:8022</value>
    </property>
    <property>
        <name>dfs.namenode.http-address.nameservice1.namenode1</name>
        <value>namenode1.mycompany.com:50070</value>
    </property>
    <property>
        <name>dfs.namenode.https-address.nameservice1.namenode1</name>
        <value>namenode1.mycompany.com:50470</value>
    </property>
    <property>
        <name>dfs.namenode.rpc-address.nameservice1.namenode2</name>
        <value>namenode2.mycompany.com:8020</value>
    </property>
    <property>
        <name>dfs.namenode.servicerpc-address.nameservice1.namenode2</name>
        <value>namenode2.mycompany.com:8022</value>
    </property>
    <property>
        <name>dfs.namenode.http-address.nameservice1.namenode2</name>
        <value>namenode2.mycompany.com:50070</value>
    </property>
    <property>
        <name>dfs.namenode.https-address.nameservice1.namenode2</name>
        <value>namenode2.mycompany.com:50470</value>
    </property>
</configuration>

Now, use the above file to create your client like this:

org.apache.hadoop.conf.Configuration conf = new org.apache.hadoop.conf.Configuration();
conf.addResource(com.google.common.io.Resources.getResource("hdfs-site.xml"));
org.apache.hadoop.fs.FileSystem fs = org.apache.hadoop.fs.FileSystem.get(conf);
System.out.println("Home directory :" + fs.getHomeDirectory());

If you are trying to read a snappy compressed file in your java code (Not mapreduce), you might see an error like this:

native snappy library not available: this version of libhadoop was built without snappy support.

It means that the native snappy files (like libsnappy.so) has not been added to the java library path. To fix this, add the following right after the java command:

-Djava.library.path=/opt/cloudera/parcels/CDH/lib/hadoop/lib/native

Note: /opt/cloudera/parcels/CDH/lib/hadoop is the HADOOP_HOME for Cloudera distribution. This location might change for different distributions. Update that accordingly.

Example command:

java -Djava.library.path=/opt/cloudera/parcels/CDH/lib/hadoop/lib/native -cp .:lib/*:/opt/cloudera/parcels/CDH/lib/hbase/lib/*:/opt/cloudera/parcels/CDH/lib/hadoop/client/*:/opt/cloudera/parcels/CDH/lib/hadoop-yarn/*:/opt/cloudera/parcels/CDH/lib/hadoop-yarn/lib/*.jar:/opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/* com.mycompany.StandaloneHdfsReader /data/example.snappy

Oracle JDK 8 Download Link to be used in Ansible

2018-01-23T11:10:00.001-08:00

I use Ansible for deployments. As part of the deployment, I install Oracle JDK. If you are like me, you are as frustrated as I am because Oracle seems to move around their rpms or tarballs a lot. I changed the URL 3 times already and I was on JDK 8 all the time.

Here is the latest URL that works. (As of 01/23/2018)

http://download.oracle.com/otn-pub/java/jdk/8u162-b12/0da788060d494f5095bf8624735fa2f1/jdk-8u162-linux-x64.rpm

oracle_java_version: 8
oracle_java_version_update: 162
oracle_java_version_build: '12'

oracle_java_hashcode: '0da788060d494f5095bf8624735fa2f1'

oracle_java_version_string: "1.{{ oracle_java_version }}.0_{{ oracle_java_version_update }}"

oracle_java_home: "/usr/java/jdk1.{{ oracle_java_version }}.0_{{ oracle_java_version_update }}"

oracle_java_rpm_filename: "jdk-{{ oracle_java_version }}u{{ oracle_java_version_update }}-linux-x64.rpm"

oracle_java_rpm_url: "https://download.oracle.com/otn-pub/java/jdk/{{ oracle_java_version }}u{{ oracle_java_version_update }}-b{{ oracle_java_version_build }}/{{oracle_java_hashcode}}/{{ oracle_java_rpm_filename }}"

Reducer Creating Empty Files in HDFS When using MultipleOutputs??

2017-12-29T08:38:00.002-08:00

I might be the only person on the planet that still uses Mapreduce. I started writing any new jobs with Spark, but I still need to enhance my MR job from time to time.

I had a requirement to generate files with different names (prefix) for different keys. So I decided to use MultipleOutput. I finished up coding and deployed in Dev cluster. Everything seemed to be hunky dory. When time came to deploy in Prod, things went south. Some of my files (with a specific key) were always empty. If you have ever worked with Mapreduce, you can understand my pain to debug. I spent almost 2 days and was on the verge to give up and just rewrite the whole damn thing using Spark. I even created git repo for the new project using Spark and was about to start coding. Before I went that route, I got an idea of trying to write data just using context.write() instead of multipleOutputs.write(). Guess what? now I see data in those files. This made me focus on MultipleOutputs rather than my business logic (even though it worked perfectly in Dev). In less than 2 minutes, I found that MultipleOutputs needs to be closed. Thanks to Google search. So, I added multipleOutputs.close() in my cleanup method of Reducer and everything started working. I am still puzzled how it worked fine in Dev though. Any guess?? Leave your comments.

What a painful way to learn this silly thing! I am recording this here in the hope to help someone someday. Happy Holidays!

Copy HBase Table Data Between Different Clusters

2017-12-26T08:33:00.001-08:00

Here is the command you'll need to copy data from one cluster to another:

bin/hbase org.apache.hadoop.hbase.mapreduce.CopyTable --peer.adr=<Comma Separated Target Zookeeper Ip Address>:2181:/hbase <Table Name>

If you are running YARN and need to specify a specific queue name, add this to the command (just before the --peer.adr):

-Dmapreduce.job.queuename=<Queue Name>

Manually Set Kafka Offset In Zookeeper

2017-12-05T13:08:00.002-08:00

If you are running a version of Kafka which stores the consumer details (like offset) in zookeeper, and if you want to manually set the offset, follow this procedure:

You will need to run zookeeper-client.sh to view and make changes. The location of this script depends on your environment. I have a Cloudera CDH 5.8.3 cluster and the script is located in /opt/cloudera/parcels/CDH/bin

Zookeeper stores the data like this:

/consumers/<Group ID>/offsets/<Topic>/<Partition>

To connect to Zookeeper cluster, CD in to the directory containing the script and run the following command:

./zookeeper-client -server <one of the zk host>:<zk port>

Your prompt should change to something like: [zk: Host:Port(CONNECTED) 0]

Now, you can run commands like these to browse data in Zookeeper

To list all consumers: ls /consumers
To list all partitions: ls /consumers/<group id>/offsets/<topic>
To view current offset: get /consumers/<group id>/offsets/<topic>/<partition>

Now, to manually set the offset run command like this:

set /consumers/<group id>/offsets/<topic>/<partition> <offset>

Once all changes are done, run "close" command to close connection to the zookeeper. Then, run "quit" to get back to normal command prompt.