tag:blogger.com,1999:blog-60130193330816650532025-11-27T10:32:01.361-08:00I write about all things technology related. Currently focused on Kafka, Avro and related ecosystem of services.Feroze Daudhttp://www.blogger.com/profile/02314158942083686821noreply@blogger.comBlogger32125tag:blogger.com,1999:blog-6013019333081665053.post-28152139876140835652024-07-07T13:30:00.000-07:002025-05-31T20:24:37.304-07:00

Introduction Kafka is a highly scalable and durable distributed streaming platform that has become the go-to choice for handling large amounts of data across a variety of industries. Its ecosystem of services is vast and powerful, enabling businesses to solve complex problems and uncover insights from their data in real-time.  One of the key services in the Kafka

Feroze Daudhttp://www.blogger.com/profile/02314158942083686821noreply@blogger.com0tag:blogger.com,1999:blog-6013019333081665053.post-34095932846282587012024-06-06T13:51:00.000-07:002025-05-31T21:00:23.129-07:00

Introduction Apache Avro Apache Avro is the leading serialization format for record data, and first choice for streaming data pipelines. It offers excellent schema evolution. In this article, we discuss how to author Avro enums so that data written with the new version of the schema can be read by consumers using the previous version of the schema. In other

Feroze Daudhttp://www.blogger.com/profile/02314158942083686821noreply@blogger.com0tag:blogger.com,1999:blog-6013019333081665053.post-81882168236244364042024-04-17T16:59:00.000-07:002024-07-23T18:36:11.334-07:00

IntroductionConfluent provides on-prem and hosted services of the Apache Kafka ecosystem. As part of this they also provide a custom serializer/deserializer to enable writing and reading messages with schema from a topic. The KafkaAvroSerializer provides an efficient mechanism for serializing messages to a kafka topic. The wire format of the KafkaAvroSerializer is described hereSometimes,

Feroze Daudhttp://www.blogger.com/profile/02314158942083686821noreply@blogger.com0tag:blogger.com,1999:blog-6013019333081665053.post-5839804326299918962023-06-24T19:55:00.007-07:002024-04-30T15:00:57.578-07:00

{ "@context": "https://schema.org", "@type": "BlogPosting", "headline": "Mask Json Field Transform", "datePublished": "2023-06-24T08:00:00+08:00", "dateModified": "2023-07-0509T09:20:00+08:00", "author": [{ "@type": "Person", "name": "Feroze Daud", "url": "https://ferozedaud.blogspot.com" }], "abstract": "

Feroze Daudhttp://www.blogger.com/profile/02314158942083686821noreply@blogger.com2tag:blogger.com,1999:blog-6013019333081665053.post-46622148402267412892021-12-04T15:09:00.006-08:002023-07-02T18:23:49.211-07:00

Avro Logical Types with Parquet, Hive and Spark In this post, I am going to talk about the fidelity of types between Avro and big data technologies like Parquet, Hive and Spark. Specifically, we will discuss which logical types maintain fidelity when the data is eventually written as parquet. We will also include querying that data using Hive and Spark. Avro Apache Avro  is a

Feroze Daudhttp://www.blogger.com/profile/02314158942083686821noreply@blogger.com0tag:blogger.com,1999:blog-6013019333081665053.post-41600914919966867952020-07-04T23:26:00.010-07:002025-03-02T11:11:07.130-08:00

IntroductionApache Kafka has become the platform of choice for doing large scale messaging. It supports a variety of serialization formats for messages, and also allows integration with external systems using a wide variety of connectors.Apache Kafka allows messages to be stored in a variety of formats in the topic:- Opaque bytes.- Avro- Json- Protocol BuffersIn this post, we take a deep

Feroze Daudhttp://www.blogger.com/profile/02314158942083686821noreply@blogger.com1tag:blogger.com,1999:blog-6013019333081665053.post-43971403101948205202016-10-04T19:49:00.006-07:002020-08-22T19:37:07.039-07:00

Introduction This is Part-3 of the N Part Series where we develop a Ping client in Python. Implementing a Ping client in Python - Part I Implementing a Ping client in Python - Part II Until now, we understood the structure of the ICMP Packet, and also looked at python code to serialize and deserialize the packet from the network. In the current installment, we will complete the

Feroze Daudhttp://www.blogger.com/profile/02314158942083686821noreply@blogger.com0tag:blogger.com,1999:blog-6013019333081665053.post-75294522895003371202016-10-03T20:55:00.001-07:002020-07-13T15:18:22.129-07:00

Introduction In Implementing a Ping client in Python - Part I, we gave an introduction to Ping, and showed how it is implemented using the ICMP protocol. Specifically, it uses the ICMP Echo Request and Response mechanism. The client sends the server an ICMP Echo packet, and the server responds with an ICMP Response packet. We also defined the python class structure of the ICMP packet. In

Feroze Daudhttp://www.blogger.com/profile/02314158942083686821noreply@blogger.com0tag:blogger.com,1999:blog-6013019333081665053.post-61817843470826145172016-09-26T18:40:00.000-07:002016-10-01T08:49:17.678-07:00

Introduction In this series of posts,  I will demonstrate how to implement a Ping client using Python. Ping is one of the most useful tools that a computer programmer might use. It is used to figure out if a target machine is reachable or not. Ping is implemented using the ICMP Protocol ICMP Protocol The Internet Control Message Protocol (ICMP) is defined by the Internet

Feroze Daudhttp://www.blogger.com/profile/02314158942083686821noreply@blogger.com0tag:blogger.com,1999:blog-6013019333081665053.post-68885424157433450982014-01-05T18:46:00.002-08:002020-08-22T19:39:11.065-07:00

This article is a bit of history about the various Microsoft HTTP client stacks. WININET Originally, there was one - the WININET implementation, that was written for the Internet Explorer browser. The implementation for this was encapsulated in Wininet.dll. In it's original implementation (as shipped in IE 2.0/IE 3.0/IE 4.x), it supported the following protocols - HTTP, FTP, GOPHER. It

Feroze Daudhttp://www.blogger.com/profile/02314158942083686821noreply@blogger.com0tag:blogger.com,1999:blog-6013019333081665053.post-11050813924917401822011-09-04T19:52:00.001-07:002020-08-22T19:50:56.770-07:00

When writing an application that talks to a server using NTLM authentication, HttpWebRequest does a three legged authentication handshake that looks as follows: Client Server GET / HTTP/1.1 HTTP/1.1 401 Access Denied Connection: close WWW-Authenticate: NTLM <token> GET / HTTP/1.1 Authorization: NTLM <token1> HTTP/1.1 401 Access Denied Connection: close WWW-Authenticate:

Feroze Daudhttp://www.blogger.com/profile/02314158942083686821noreply@blogger.com0tag:blogger.com,1999:blog-6013019333081665053.post-18761060779185108022011-08-09T16:32:00.001-07:002020-08-22T19:50:00.346-07:00

Introduction I have a service that runs on Linux under JBOSS. This service uses JMX (RMI) to talk to a windows box running a java service. RMI is the equivalent of the windows Distributed COM that allows a client to invoke a method on a remote service. Setup My setup is as follows. I have a Linux machine (Client) that is running a Java service in Tomcat, hosted in JBOSS. It is trying

Feroze Daudhttp://www.blogger.com/profile/02314158942083686821noreply@blogger.com0tag:blogger.com,1999:blog-6013019333081665053.post-29263235201348675502011-03-22T09:56:00.000-07:002016-12-10T19:07:08.083-08:00

Ars Technica has a great article giving the reasons why HTTPS is not used more on the Web. The summary is as follows: 1) HTTPS adds some latency to initial connection establishment. 2) HTTPS documents cannot be cached by the browser. While this is not a big issue when the client and server are on the same continents, 3) HTTPS websites cannot be virtual hosts on the same server, and this causes

Feroze Daudhttp://www.blogger.com/profile/02314158942083686821noreply@blogger.com0tag:blogger.com,1999:blog-6013019333081665053.post-66282898476213833842010-06-25T15:42:00.000-07:002010-06-25T15:42:43.555-07:00

Brian Jones analyzes Apple's claim that the Iphone4's display resolution is better than a human eye. Interesting reading.

Feroze Daudhttp://www.blogger.com/profile/02314158942083686821noreply@blogger.com0tag:blogger.com,1999:blog-6013019333081665053.post-18859822418249777102010-03-21T22:58:00.003-07:002020-08-22T19:40:15.995-07:00

Implementing Multipart-MIME upload helper using WebClient The WebClient class is a higher level abstraction on top of HttpWebRequest, that abstracts away the common tasks of uploading files, as well as doing Form Posts to HTTP servers. However, it does not provide an easy way to do multipart form uploads. Multipart form upload is defined by the W3C specification. Multipart HTML Form Design

Feroze Daudhttp://www.blogger.com/profile/02314158942083686821noreply@blogger.com0tag:blogger.com,1999:blog-6013019333081665053.post-7456425087427989392009-12-06T14:05:00.000-08:002016-12-10T19:14:24.341-08:00

Implementing fast file transfer with Socket.SendFile When writing network applications, we usually have a need to implement file transfer between two hosts. For eg, imaging an FTP client, where the client is downloading or uploading a file from an FTP server. Similarly, you could be uploading an image file (probably a photo) as an attachment to a Blog or a website like Facebook or Flickr.

Feroze Daudhttp://www.blogger.com/profile/02314158942083686821noreply@blogger.com6tag:blogger.com,1999:blog-6013019333081665053.post-47313454067342642622009-11-15T15:45:00.001-08:002020-08-22T19:40:58.874-07:00

In the last article, I described how to send an object from a Java application to a .NET application over a Socket connection. https://ferozedaud.blogspot.com/2009/11/howto-serialize-data-from-object-from.html The Java implementation was relying on ByteBuffer class to serialize the object in LittleEndian format that the .NET code understands. However, it turns out that using a ByteBuffer is

Feroze Daudhttp://www.blogger.com/profile/02314158942083686821noreply@blogger.com3tag:blogger.com,1999:blog-6013019333081665053.post-82142525981531560822009-11-15T00:34:00.000-08:002009-11-15T00:34:36.678-08:00

Problem: You want to send data across the network, from a .NET application to a Java application (or vice versa). The data is actually an object that you want to send over the network. Since these are heterogenous platforms, you cannot use the Runtime specific serialization methods (for eg BinaryFormatter in .NET). You need to roll your own. You basically have two choices. You can manually

Feroze Daudhttp://www.blogger.com/profile/02314158942083686821noreply@blogger.com3tag:blogger.com,1999:blog-6013019333081665053.post-39124474135046853622009-11-01T17:26:00.002-08:002023-07-03T10:08:46.479-07:00

A lot of people have blogged about System.NET tips and tricks. Esp from Microsoft, prior System.Net team members (Jon Cole, Durgaprasad Gorti, Malar Chinnusamy) have authored blog posts on how to do stuff with System.Net. Non microsoft folks have also stepped up to the plate. For eg, Stephen Cleary has written a very good concise blog on System.Net HOWTOs and FAQs. In this post, I aim to

Feroze Daudhttp://www.blogger.com/profile/02314158942083686821noreply@blogger.com0tag:blogger.com,1999:blog-6013019333081665053.post-13792301972652366392009-10-24T11:51:00.000-07:002009-10-24T11:51:24.252-07:00

Recently, I have been playing around with Lucene (http://incubator.apache.org/lucene.net/. Lucene is an Open Source project, which is sponsored by the Apache foundation, that gives you all the components necessary to create your own search engine. I downloaded the latest build, which is versioned 2.0.004 and located at http://incubator.apache.org/lucene.net/download/

Feroze Daudhttp://www.blogger.com/profile/02314158942083686821noreply@blogger.com2tag:blogger.com,1999:blog-6013019333081665053.post-60875662848752463492009-10-02T19:59:00.000-07:002009-10-02T19:59:12.518-07:00

On public newsgroups, I have seen a lot of postings where people complained that their managed code application, written with HttpWebRequest, and using NTLM auth to talk to a server, would fail. However, Internet explorer running on the same machine would work fine. Here are some of the threads that show this problem: http://social.msdn.microsoft.com/Forums/en-US/netfxnetcom/thread/

Feroze Daudhttp://www.blogger.com/profile/02314158942083686821noreply@blogger.com2tag:blogger.com,1999:blog-6013019333081665053.post-43907895717709239982009-09-29T18:26:00.000-07:002009-09-29T18:26:39.195-07:00

In the previous parts, I showed how to implement traceroute using System.Net.Sockets. We used the Socket class to create a Raw socket, and use that to do a traceroute. Implementing Traceroute with System.Net Part III Implementing Traceroute with System.Net Part II Implementing Traceroute with System.Net Part I It turns out, that System.Net.NetworkInformation already has a Ping class, that

Feroze Daudhttp://www.blogger.com/profile/02314158942083686821noreply@blogger.com2tag:blogger.com,1999:blog-6013019333081665053.post-59118148254859066292009-09-17T18:25:00.000-07:002009-09-29T16:40:04.723-07:00

In the previous part http://ferozedaud.blogspot.com/2009/09/implementing-traceroute-with-systemnet_07.html I showed a simple implementation of traceroute, that built upon my earlier implementation of Ping. As promised, in this article, we will see how to make the utility more robust. One of the things I didnt like about the previous implementation, was that there was no verification of the

Feroze Daudhttp://www.blogger.com/profile/02314158942083686821noreply@blogger.com1tag:blogger.com,1999:blog-6013019333081665053.post-65454685308609140642009-09-11T18:09:00.001-07:002009-09-11T18:32:00.583-07:00

In this post, I want to talk about some of the tools that a programmer doing network programmer might want to learn, in order to do his job better.First, let us talk about logging tools. No matter which library you use, whether it is System.Net from .NET framework, or CommonsHTTP from the Apache Commons project, you need to have a way of enabling logging to see what is going on in the library.If

Feroze Daudhttp://www.blogger.com/profile/02314158942083686821noreply@blogger.com0tag:blogger.com,1999:blog-6013019333081665053.post-7289343049914830612009-09-10T08:55:00.000-07:002024-02-08T14:37:27.398-08:00

A reader posed a question on Stack Overflow about a problem with authentication. The scenario was as follows:This was a typical 3-tier application, in which the middle-tier (ASP.NET web server) was making a HttpWebRequest to a backend IIS-7 server that required authentication.When authentication method was set to Digest/Basic/Negotiate, the server worked fine, and the authentication succeeded.

Feroze Daudhttp://www.blogger.com/profile/02314158942083686821noreply@blogger.com0