tag:blogger.com,1999:blog-71654322026-10-16T17:27:57.746+11:00Ojitha Hewa KumanayakaSoftware Engineer from AustraliaOjitha Hewa Kumanayakahttp://www.blogger.com/profile/17299136133671661631noreply@blogger.comBlogger141125tag:blogger.com,1999:blog-7165432.post-53417450988459367062023-06-17T14:06:00.026+10:002023-07-18T17:24:49.090+10:00SQL Analytic Functions table, td, th { border: 1px solid; } table { border-collapse: collapse; font-size: small; } Window functions play a very important part in analytic queries. I explore some of the window functions using Postgres 13.4. Window function defaults differ from Postgres to Redshift, although the results are the same. The OVER () default for Redshift is (ROWS BETWEEN UNBOUNDED PRECEDING AND Ojitha Hewa Kumanayakahttp://www.blogger.com/profile/17299136133671661631noreply@blogger.com0tag:blogger.com,1999:blog-7165432.post-91255891582554123692023-04-23T13:25:00.011+10:002023-04-23T21:35:20.873+10:00VSCode for Haskell DevelopmentHaskell is a purely functional programming language. The VSCode extension Haskell for Visual Studio Code has been configured to develop Haskell code in the Docker container remotely. This blog explains how to use the VSCode editor to develop Haskell programs. The features, where you can find the table, summarises the features that HLS supports. Many of these are standard LSP features. Install Ojitha Hewa Kumanayakahttp://www.blogger.com/profile/17299136133671661631noreply@blogger.com0tag:blogger.com,1999:blog-7165432.post-64590325284870773542022-02-21T19:29:00.015+11:002023-02-16T06:55:31.071+11:00PySpark relational operationsIn this blog I explore the PySpark DataFrame API to create SQL JOIN and Windowing functionality. I am using MovieLens1 dataset to explain. In addition to SQL JOIN and CASE, advanced topic such as SQL Window and Pivot will be discussed here. In my last blog post2, I've explained how to use Hive metastore which is more intent to use SQL only. For the full list, see the Spark SQL documentation3. Ojitha Hewa Kumanayakahttp://www.blogger.com/profile/17299136133671661631noreply@blogger.com0tag:blogger.com,1999:blog-7165432.post-65316280527273094402022-02-20T11:48:00.005+11:002023-04-09T12:58:26.450+10:00Spark SQL using Hive metastoreThis is the second part of the PySpark Dataframe DSL basics1. The main objectives are to create a Hive metastore based database and tables. Create Databases and tables Metadata Data Sources SQL User Define Functions First, create Spark Session manually as the first step. By default, Spark uses the Apache Hive metastore to persist all the tables metadata. I have to change the default locationOjitha Hewa Kumanayakahttp://www.blogger.com/profile/17299136133671661631noreply@blogger.com0tag:blogger.com,1999:blog-7165432.post-77258272307069396312022-02-19T12:27:00.015+11:002022-02-20T16:28:15.180+11:00PySpark Dataframe DSL basicsIn this blog post, I explore the PySpark DataFrame structured API and DSL operators. Typical tasks you can learn: Connection to remote PostgreSQL database Create DataFrame from above database using PostgreSQL Sample Database Create DataFrame using CSV (movieLens) files. In addition to that the equivalent SQL has been provided to compare with the DSL. Preperation Configure Database in the Ojitha Hewa Kumanayakahttp://www.blogger.com/profile/17299136133671661631noreply@blogger.com0tag:blogger.com,1999:blog-7165432.post-8291833601523713502022-02-16T16:46:00.036+11:002023-04-09T12:19:32.258+10:00PySpark environment for the Postgres databaseIn this blog, I am going to Create Postgres 13.4 docker environment Create Spark enabled Jupyter docker environment Run remote Jupyter notebooks via Visual Studio Code And test the PySpark Jupyter notebook1 or follow the PySpark Dataframe DSL basics which is the second part of this blog. As shown in Fig.1, Jupyter Server and Postgres databases run in the Docker environment. Jupyter and PostgresOjitha Hewa Kumanayakahttp://www.blogger.com/profile/17299136133671661631noreply@blogger.com0tag:blogger.com,1999:blog-7165432.post-43173679558180626192021-06-26T11:14:00.005+10:002023-02-16T07:14:28.775+11:00AWS Cloudformation to create AWS VPCThis blog will build AWS Virtual Private Cloud (VPC) creation using AWS Cloudformation template, step by step. However, it is necessary to plan your network with Classless Inter-Domain Routing (CIDR). Simplest VPC Planning Attach AWS Internet Gateway and Route Table Testing the VPC Simplest VPC Here this simplest workable CFN: Fig 1: Creates only VPC This creates only the VPC, as shown in Ojitha Hewa Kumanayakahttp://www.blogger.com/profile/17299136133671661631noreply@blogger.com0tag:blogger.com,1999:blog-7165432.post-16470764490418495522021-06-09T21:22:00.004+10:002021-06-22T13:31:20.031+10:00AWS Glue Workflow: Getting startedCreate a fundamental Glue workflow using the AWS Cloudformation template. The Glue workflow replaces the use of the Step functions, which have been used to maintain Glue flow states. However, if you plan to automate your build deployment, here is the blog post1 to help you. In this post, I completely ignore the AWS BuildPipeline, which is the recommended CI/CD pipeline explained in the above Ojitha Hewa Kumanayakahttp://www.blogger.com/profile/17299136133671661631noreply@blogger.com0tag:blogger.com,1999:blog-7165432.post-42227033421988130662021-03-18T12:27:00.007+11:002021-06-28T11:23:05.777+10:00Java Streaming API recipesIn Java, elements in the stream are subdivided into subsets (can be subdivided into further) which are processed in parallel using different cores of the CPU. Therefore, only stateless, non-interfering and associative subsets are eligible to run parallel. These subsets are combined for the short-circuit/terminal process for the final result. The sequential(default) or parallel mode depends on theOjitha Hewa Kumanayakahttp://www.blogger.com/profile/17299136133671661631noreply@blogger.com0tag:blogger.com,1999:blog-7165432.post-69236471311875453062021-01-01T17:27:00.002+11:002021-01-01T17:34:23.028+11:00Java 9 ParallelismA current computer programming paradigm is not object-oriented anymore; it is about parallelism1. In the programming perspective, concurrency is the composition of independently executing processes. Parallelism is the simultaneous execution of computations. Concurrency is about dealing with lots of things at once. Parallelism is about doing lots of things at once. Concurrency is about Ojitha Hewa Kumanayakahttp://www.blogger.com/profile/17299136133671661631noreply@blogger.com0tag:blogger.com,1999:blog-7165432.post-10295004953648763002020-09-29T10:40:00.002+10:002021-06-28T11:23:52.719+10:00Python my workflowMy Flow I combined two softwares using pyenv-virtualenv : pyenv manages multiple versions of Python itself virtualenv (Python Virtual Environments: A Primer) manages virtual environments for a specific Python version. pyenv-virtualenv manages virtual environments for across varying versions of Python. Here the way to create virtualenv pyenv virtualenv 3.7.2 p3 To activate the environment pyenvOjitha Hewa Kumanayakahttp://www.blogger.com/profile/17299136133671661631noreply@blogger.com0tag:blogger.com,1999:blog-7165432.post-2828907528706869212020-08-17T15:21:00.007+10:002022-12-18T17:22:46.343+11:00Structured data meaningGoogle uses structured data such as RDFa, Microdata or JSON-LD to understand the contents of the page. RDFa is based on the RDF and HTML5 extension. JOSN-LD can be created from the same RDF Turtle. Learn RDF Turtle to create structured data for SEO. RDFa1 (Resource Description Framework in Attributes) is attribute-based, for example, a, href in the HTML/XHTML. RDFa does not affect the HTML code Ojitha Hewa Kumanayakahttp://www.blogger.com/profile/17299136133671661631noreply@blogger.com0tag:blogger.com,1999:blog-7165432.post-20795325794256610742020-08-14T14:45:00.079+10:002022-12-18T17:37:45.460+11:00Apache Jena to learn RDF and SPARQL RDF is one of the semantic web technology as well as the foundation for Turtle, N-Triples including JSON-LD. SPARQL is the query language for RDF. Use Apache Jena tools to learn RDF. For example, if you see the web page, that is human readable, because the end-user is human. However, there are search engines who choose the page on behalf of the consumer. Therefore, a search engine is a machine Ojitha Hewa Kumanayakahttp://www.blogger.com/profile/17299136133671661631noreply@blogger.com0tag:blogger.com,1999:blog-7165432.post-12288333007221060282020-08-08T09:16:00.012+10:002020-08-25T15:48:54.031+10:00JSF and RichFacesRecently, I got a project to work on JSF RichFaces1. RichFaces project reached end-of-life in June 2016. Thanks to Java backward compatibility I managed to run the RichFaces application on TomEE 7 using Intellij Idea as IDE. Run the following maven archytype in your command prompt as explained in the JBoss Community guide2: mvn archetype:generate -DarchetypeGroupId=org.richfaces.archetypes Ojitha Hewa Kumanayakahttp://www.blogger.com/profile/17299136133671661631noreply@blogger.com0tag:blogger.com,1999:blog-7165432.post-65158617768778040402020-06-17T21:47:00.040+10:002020-08-25T15:49:43.691+10:00Mac keyboard shortcut to copy file path as markdownThe macOS productivity1 shortcut to Copy file/folder markdown path. How to create macOS Automator Quick Action to copy the file/folder markdown path using shortcut keys as the same way you copy the path name of the file/folder. I give you an idea of how to use JXA in the Automator. As additional information, I introduce how to use macOS Cocoa that will help you to make better logging using macOSOjitha Hewa Kumanayakahttp://www.blogger.com/profile/17299136133671661631noreply@blogger.com1tag:blogger.com,1999:blog-7165432.post-40904897816707661642020-05-31T19:10:00.009+10:002020-06-01T12:34:17.746+10:00How to create step guides Flow to create step guide Screen annotation in Mac Flow to create step guide Most of the bloggers need to add stories to their blog posts. For tech bloggers, that is most probably the journey of clicks over browser, application and so on. However, the blogger has to capture the screen and record with the annotations. In this blog, I explain how to create step by step guide: graph TDOjitha Hewa Kumanayakahttp://www.blogger.com/profile/17299136133671661631noreply@blogger.com0tag:blogger.com,1999:blog-7165432.post-21150959899800129372020-05-30T21:01:00.022+10:002020-05-31T19:46:34.394+10:00Toolbox for bloggersCONTENTS Animated GIF for terminal recording Diagrams and Charts Write a blog post Math notations Create step guides Animated GIF for terminal recording The LICECap is nice free little tool good to create animated gif show your terminal for example. Fig 1: Python version on terminal It is hard to find the similar Diagrams and Charts Although there are few diagrams types Ojitha Hewa Kumanayakahttp://www.blogger.com/profile/17299136133671661631noreply@blogger.com0tag:blogger.com,1999:blog-7165432.post-16921147296225144852020-05-17T21:10:00.000+10:002020-06-02T09:09:49.341+10:00AngularJs BasicsYou can download the AngularJs to use via CDN. I am using doing my examples in Visual Studio Code (VSC) Live Server plugin for the VSC Angular application based on the model-view-controller pattern. The view is based on the DOM. The controllers are JavaScript classes and the model data is stored in object properties1. The model (line# 10) represent the current state of the application. Ojitha Hewa Kumanayakahttp://www.blogger.com/profile/17299136133671661631noreply@blogger.com0tag:blogger.com,1999:blog-7165432.post-67415288058779917502020-04-08T19:49:00.009+10:002020-05-31T09:28:48.673+10:00MAC shortcutsHere the simple mac shortcuts found from the difference sources such as1. Here the general tips: Mac keyboard shortcuts from Apple Support. run the caffeinate in the terminal to keep the computer wake unlimitedly2. Expand print dialog box by default: defaults write -g PMPrintingExpandedStateForPrint -bool TRUE Enable or disable character accent menu: // disable defaults write -g Ojitha Hewa Kumanayakahttp://www.blogger.com/profile/17299136133671661631noreply@blogger.com0tag:blogger.com,1999:blog-7165432.post-64481727455650176462020-03-29T17:46:00.002+11:002020-03-29T22:55:05.655+11:00Calling AWS ServicesTo programmatically connect to the AWS Services, need to use endpoints. The AWS SDK and the CLI use the default endpoint of the region to access the service. The regional endpoint is protocol://service-code.region-code.amazonaws.com standard. But services such as IAM not support the region. Some example of calling1 to AWS services via API calls. All CLI calls are signed by Signature version 42.Ojitha Hewa Kumanayakahttp://www.blogger.com/profile/17299136133671661631noreply@blogger.com0tag:blogger.com,1999:blog-7165432.post-82093070302304153162019-08-31T08:08:00.000+10:002019-08-31T08:12:00.448+10:00Java Collectors recipesJava Collectors is a utility class with several reduction operations. Here a few of the example you can do with the Collectors class. For example, grouping functionality first. import java.util.stream.Collectors; import java.util.stream.Stream; // basic functionality of the collector Stream.of("one", "two", "three", "four", "five") .Ojitha Hewa Kumanayakahttp://www.blogger.com/profile/17299136133671661631noreply@blogger.com0tag:blogger.com,1999:blog-7165432.post-75000686896940162802019-04-03T23:15:00.001+11:002020-05-14T22:58:00.309+10:00Parse the namespace based XML using PythonIn this blog, I am considering how to parser and modify the xml file using python. For example, I need to parser the following xml1 (slightly modified for this blog) and need to write the modified xml to out.xml file. Here the country.xml <?xml version="1.0"?> <actors xmlns:fictional="http://characters.example.com" xmlns="http://people.example.com"Ojitha Hewa Kumanayakahttp://www.blogger.com/profile/17299136133671661631noreply@blogger.com0tag:blogger.com,1999:blog-7165432.post-25619935081834600402019-03-24T11:48:00.002+11:002020-05-15T17:17:00.873+10:00Terraform SSH connection to AWS EC2 The instances are created via Terraform, but Terraform can only used existing key pairs. First thing is to create the key pair as explained in the AWS documentation1. NOTE: However, if you are not familiar with Terraform, please go through the following blogs before this blog Basic example of creating AWS EC2 with Terraform Creating AWS S3 bucket with Terraform Connect to the EC2 instance, Ojitha Hewa Kumanayakahttp://www.blogger.com/profile/17299136133671661631noreply@blogger.com0tag:blogger.com,1999:blog-7165432.post-31309922523055959392019-03-23T21:51:00.002+11:002020-05-15T17:17:14.124+10:00Creating AWS S3 bucket with TerraformI recommend to read the first example1,because this is a extenstion to that. However, instead of AWS EC2, here the target resource is AWS S3 for the simplicity. Here the providers.tf file. provider "aws" { region = "${var.s3_region}" } terraform { required_version = ">= 0.11.13" backend "s3" { bucket = "ojitha" key Ojitha Hewa Kumanayakahttp://www.blogger.com/profile/17299136133671661631noreply@blogger.com0tag:blogger.com,1999:blog-7165432.post-2820067553686378472019-03-23T21:35:00.005+11:002020-05-15T17:17:29.649+10:00Basic example of creating AWS EC2 with TerraformHere the very basic example. This is just a note of creating single EC2 instance using Terraform. First you need to define the provider in the example.tf file provider "aws" { region = "ap-southeast-2" } # resource "aws_s3_bucket" "example" { # bucket = "ojithatest1" # acl = "private" # } resource "aws_instance" &quotOjitha Hewa Kumanayakahttp://www.blogger.com/profile/17299136133671661631noreply@blogger.com0