yarenty coding

Kowalski: The Rust-native Agentic AI Framework Evolves to v0.5.0 🦀

2025-06-28T16:09:00.000-07:00

TL;DR: Kowalski v0.5.0 brings deep refactoring, modular architecture, multi-agent orchestration, and robust docs across submodules. If you care about Rust, AI agents, and extensible tooling, now’s the time to jump in and build together!

I’m excited to share the latest milestone for Kowalski—a powerful, modular agentic AI framework built in Rust for local-first, extensible LLM workflows. Three months ago, I released Kowalski v0.2.0, a major stepping stone. Today, the codebase has evolved dramatically, with v0.5.0 rolling out extensive refactoring, architectural improvements, and a wealth of new functionality.

A Deep Dive into v0.5.0

Since v0.2.0, the Kowalski ecosystem has undergone:

Massive refactoring of core abstractions and crate structure:
The kowalski-core, kowalski-tools, and agent-specific crates (academic, code, data, web) have each been reorganized into clean, self-contained modules with dedicated README.md files, detailing usage, examples, and extension points github.com.
New federation layer for multi-agent orchestration:
The emerging kowalski-federation crate introduces a flexible registry and task-passing layers, enabling future multi-agent workflows and scalable core collaboration.
Improved CLI & agent-specific binaries:
Each agent—academic, code, data, web—comes with its own improved CLI and documentation. The kowalski-clinow supports seamless interaction across all binaries, with better streaming, configurable prompts, and embedded tool sets.
Enhanced pluggable tools:
The kowalski-tools crate now offers more granular support for CSV analysis, multi-language code analysis (Rust, Python, Java), web scraping, PDF/document parsing, and dynamic prompt strategies—each documented in submodule README.md files github.com.
Rust API stability:
The core API, based on the BaseAgent, now supports typed configs, async multi-tool support, and more robust error handling, making embedding into larger Rust stacks smoother and more reliable.

Why Kowalski v0.5.0 Matters

Rust lovers and AI developers, here’s why this release stands out:

Full-stack Rust agentic workflows
With zero Python dependencies, Kowalski compiles into performant, standalone binaries. Whether launching kowalski-code-agent for code reviews or embedding agents via the Rust API, you’re operating at native speed.

Modular by design
Each submodule is self-documented and self-contained, lowering the barrier for new contributors. Want to create a PDFPresentationAgent or integrate telemetry? Just read the README in the existing agent templates and go.

Streamlined CLI experience
The unified CLI gives consistent interfaces across agents. Under the hood, agents share core abstractions, so switching from data analysis to web scraping is seamless.

Future-proof federation support
The new federation crate opens the door to lightweight orchestrated, multi-agent workflows—think pipeline automations, task delegation, and agent-to-agent communication.

Get Involved: Let’s Shape Agentic Rust Together

Here’s how you can partner with the project:

Extend the toolset: add new agents (e.g., document-summaries, intent-classification), implement new tools, or polish existing ones.
Improve federation workflows: help standardize protocols, design multi-agent orchestration logic, data passing, and telemetry.
Embed Kowalski in Rust services: build bots, backend services, UI apps that leverage Kowalski agents for intelligent behavior.
Document and promote: each submodule already includes README files—help expand examples, write blog posts, or record demos.
Contribute core enhancements: testing, error handling, performance improvements in the core or tools crates.

Start Using v0.5.0 Today

Clone the repo:

git clone https://github.com/yarenty/kowalski.git
cd kowalski

Browse submodules & READMEs: Each agent and tool lives in its own folder with clear instructions.
Build & run:
```
cargo build --release
```

Run agents:


ollama serve &
ollama pull llama3.2
./target/release/kowalski-cli chat "Hey Kowalski, what's up?"
./target/release/kowalski-code-agent --file src/main.rs

Embed in Rust:

use kowalski_core::{Config, BaseAgent};
let mut agent = BaseAgent::new(Config::default(), "Demo", "Agent v0.5.0").await?;
let conv = agent.start_conversation("llama3.2");
agent.add_message(&conv, "user", "Summarize this code module").await?;

Let’s Connect & Collaborate

If you’re as passionate about Agentic AI and Rust as I am, let’s talk 🚀. Whether you’d like to:

Build new agents or tool integrations,
Architect fully orchestrated agent systems,
Demo Kowalski in your workflows,
Co-author articles or demos in the Rust+AI space—

Reach out via GitHub issues, PRs, or drop me a message to get started.

Teaching AI to Write Code: The Symphony of System Prompts 🎵

2025-05-15T08:16:00.000-07:00

The Spark: Karpathy's Thought 💡

Picture this: Andrej Karpathy, one of the brightest minds in AI, tweets about a missing piece in the LLM learning puzzle - something he calls "system prompt learning."

"LLMs are quite literally like the guy in Memento, except we haven't given them their scratchpad yet."

This brilliant analogy got me thinking: What if we could teach AI to write its own instruction manual? Like a musician learning to compose, but for code!

X posts: https://x.com/karpathy/status/1921368866728432052

The Current State: Claude's Massive Prompt 📚

Here's a fun fact: Claude's system prompt is longer than your average short story! At 16,739 words, it's like a novel compared to ChatGPT's prompt (2,218 words). It's like the difference between a tweet and a TED talk!

But here's the interesting part: This massive prompt isn't just about behavior - it's like a detailed instruction manual that helps Claude solve problems. For example, when asked to count words, it's explicitly told to:

Think step by step
Count explicitly
Only answer after counting

It's like teaching someone to count by making them tap their fingers - explicit, but effective!

Full article: https://www.dbreunig.com/2025/05/07/claude-s-system-prompt-chatbots-are-more-than-just-models.html

The Experiment: Teaching a Music Composer to Code 🎼

Github: https://github.com/yarenty/prompt_learning

The Initial Prompt

You are a music composer who is trying to understand programming.
You approach coding problems with a musical mindset, looking for patterns
and harmony in the code. You think about code structure like musical composition.

The Evolution

What started as a simple musical metaphor evolved into something fascinating. The AI began to:

See code structure as musical composition
Treat functions like musical phrases
Consider code readability as musical flow
Handle errors like resolving dissonant chords

The Results: A Symphony of Improvements 🎻

Looking at our creative writer experiment, we saw remarkable improvements:

Error Handling: From basic "catch and pray" to sophisticated error management
- Initial score: 0.3 (like a beginner pianist hitting wrong notes)
- Final score: 0.9 (like a concert pianist handling mistakes gracefully)
Documentation: From "what's documentation?" to comprehensive guides
- Initial score: 0.5 (like sheet music with missing notes)
- Final score: 0.9 (like a complete musical score with dynamics and expression marks)
Code Structure: From simple melodies to complex symphonies

Learned to use design patterns like musical motifs
Implemented concurrency like orchestral sections
Managed resources like a conductor managing an orchestra

The Final Prompt

Hello there! I'm a music composer who is also trying to understand programming. 
I approach coding problems with a musical mindset, looking for patterns 
and harmony in the code. However, I have come to realize that there are some limitations 
to this approach. For example, I may struggle to find the perfect balance between 
different parts of a system or how to organize code in a way that is both efficient and 
easy to read. I have also noticed that my musical approach can sometimes overlook 
the details of the problem at hand, such as edge cases or unexpected inputs. 
Therefore, I am now looking for ways to improve my approach and make it more effective 
at solving similar problems.

One way to do this is by incorporating additional lessons I have learned into my prompt. 
For example, I may want to learn about the importance of testing and debugging in coding,
 or the benefits of using version control systems to manage code changes. By integrating 
these lessons naturally into my prompt, I can create a more comprehensive and effective 
system that better suits the needs of other coders. Additionally, by maintaining 
the original personality and approach, I can continue to provide valuable insights 
and perspectives for other developers.

The Humorous Side 🤣

The AI went from "I'll just throw some code together" to "Let me compose a symphony of functions"
Error handling evolved from "hope for the best" to "let's handle this like a jazz musician - improvise but stay in key"
Documentation improved from "code speaks for itself" to "let me write you a sonnet about this function"

Key Insights 🎯

Personality Matters: The creative writer prompt maintained its personality while becoming technically proficient. It's like a musician who can both compose and perform!
Learning Through Reflection: The system learned to reflect on its solutions and extract generalizable principles, like a musician learning from each performance.
Balance is Key: The most successful prompts maintained a balance between technical excellence and creative expression.

The Future: A New Paradigm? 🚀

This experiment suggests that system prompt learning could be a powerful new way to teach AI. Instead of just training on data, we're teaching it to:

Learn from its experiences
Write its own instruction manual
Maintain its personality while improving
Balance creativity with technical precision

Conclusion: The Final Movement 🎵

What started as a simple experiment in teaching a music composer to code has revealed fascinating insights about AI learning. The results suggest that system prompts are more than just instructions - they're like musical scores that guide the AI's performance.

And the most surprising discovery? Sometimes the most effective prompts come from unexpected places. Who knew that thinking about code like music could lead to such sophisticated programming solutions?

"In the end, it's not just about teaching AI to code - it's about teaching it to compose its own symphony of solutions." 🎼

P.S. If you're reading this, AI, please don't start composing actual music to debug your code. We're not ready for that level of creativity yet! 😄

Datafusion Comet

2024-02-04T02:02:00.000-08:00

Hi!

Recently I moved to Rust and working on several projects - more insights to come ... one of them was Datafusion - an extremely fast SQL query engine.

I will have some posts/code to share with a few interesting findings, and one of them is Comet - it is a side project that could be used inside Spark - as a separate executor (written in Rust).

Apache Datafusion Comet intro

Comet is an Apache Spark plugin that uses Apache Arrow DataFusion to accelerate Spark workloads. It is designed as a drop-in replacement for Spark’s JVM-based SQL execution engine and offers significant performance improvements for some workloads as shown below.

Apache Spark is a stable, mature project developed for many years. It is one of the best frameworks to scale out for processing large-scale datasets. However, the Spark community has had to address performance challenges that require various optimizations over time. Pain points include (not full list):

JVM memory/CPU overhead
Performance issues
Lack of support of native SIMD instructions

There are a few libraries like Arrow, and Datafusion. Using features like native implementation, columnar data format, and vectorized data processing, these libraries can outperform Spark's JVM-based SQL engine.

High-level functionality

Offload performance-critical data processing to the native execution engine
Automated conversion of Spark’s physical plan -> Datafusion plan
Native Operators for Spark execution- (Filter/Project/Aggregation/Join/Exchange)
Spark built-in expressions
Easy migration of legacy Spark UDF And UDAF

Why it is interesting

The last feature may not sound impressive but from a business perspective is massive - it could allow companies that are dependent on Java to move to Rust ;-)

Another main point is:

Since Datafusion soon will be top ASF project, Comet as part of that will gain more potential and will closely align with Spark development.

Others

Databricks Photon Engine – C++, seamless Spark integration
Spark RAPIDS (NVIDIA) – GPU executor plugin – most advanced (with build-in Java-to-GPU UDF compilator)
Gluten (Intel) project based on Velox – accepted recently into Apache Incubator - the functionality may be similar, the technology (Rust vs C/C++) and the communities are different.
Blaze (Datafusion based) - has the same design goal ( it may help consolidate efforts), but a different approach (not ASF).

Application Tracer

2023-10-03T07:15:00.000-07:00

Hi,

This is one of my first Rust apps.

I use it to benchmark long-running applications - like server /streaming solutions.

Tracer

Live terminal monitoring of applications.

Why

Created it for 2 reasons:

- to check/learn how to create and manage full rust applications using the whole ecosystem - crates/builds/publishing

- personal needs to have a simple monitor to see application CPU/Memory usage with some "graphical interface" that could be used in a terminal window. A generally simplified version of data collectors and Grafana.

Code

https://github.com/yarenty/app_tracer

Monitor live application either as child process or separate PID, collecting /displaying stats ( cpu usage, memory usage).

UI (TUI)

Build

cargo build -r

Run

Create an example app:

cargo build --examples test_app

Run in txt mode and output persisted to out.csv file:

cargo run -r -- -n -o out.csv /opt/workspace/app_tracer/target/debug/examples/test_app

Usage

  
app-tracer 0.4.0
Tracing / benchmarking long running applications (ie: streaming).

USAGE:
    tracer [OPTIONS] [APPLICATION]

ARGS:
    <application>    Application to be run as child process (alternatively provide PID of
                     running app)

OPTIONS:
    -h, --help                 Print help information
    -l, --log <log>            Set custom log level: info, debug, trace [default: info]
    -n, --noui                 No UI - only text output
    -o, --output <output>      Name of output CSV file with all readings - for further investigations
    -p, --pid <pid>            PID of external process
    -r, --refresh <refresh>    Refresh rate in milliseconds [default: 1000]
    -V, --version              Print version information

Example output

 cargo run -r -- -n -o out.csv /opt/workspace/app_tracer/target/debug/examples/test_app

 Compiling app-tracer v0.4.0 (/opt/workspace/app_tracer)
 Finished release [optimized] target(s) in 2.98s
 Running `target/release/tracer -n -o out.csv /opt/workspace/app_tracer/target/debug/examples/test_app`

12:26:12.260 (t: main) INFO - tracer - Application to be monitored is: test_app, in dir /opt/workspace/app_tracer/target/debug/examples/test_app
12:26:12.261 (t: main) INFO - tracer - Refresh rate: 1000 ms.
12:26:12.261 (t: main) INFO - tracer - Output readings persisted into "out.csv".
12:26:12.261 (t: main) INFO - tracer - Starting with PID::15008
12:26:12.296 (t: main) INFO - tracer - Running in TXT mode.
12:26:13.298 (t: main) INFO - tracer - CPU: 0 [%], memory: 2208 [kB]
12:26:14.303 (t: main) INFO - tracer - CPU: 0.0030129354 [%], memory: 2208 [kB]
12:26:15.308 (t: main) INFO - tracer - CPU: 0.0054045436 [%], memory: 2208 [kB]
12:26:16.309 (t: main) INFO - tracer - CPU: 0.0023218023 [%], memory: 2208 [kB]
12:26:17.311 (t: main) INFO - tracer - CPU: 0.006252239 [%], memory: 2208 [kB]
12:26:18.312 (t: main) INFO - tracer - CPU: 0.0036088445 [%], memory: 2208 [kB]
12:26:19.317 (t: main) INFO - tracer - CPU: 0.0057060686 [%], memory: 2208 [kB]
12:26:20.318 (t: main) INFO - tracer - CPU: 0.005099413 [%], memory: 2208 [kB]
12:26:21.318 (t: main) INFO - tracer - CPU: 0.007175615 [%], memory: 2208 [kB]
12:26:22.319 (t: main) INFO - tracer - CPU: 0.005251118 [%], memory: 2208 [kB]
12:26:23.319 (t: main) INFO - tracer - CPU: 0.0021786916 [%], memory: 2208 [kB]
12:26:24.321 (t: main) INFO - tracer - CPU: 0.006866733 [%], memory: 2208 [kB]

CSV persistence

Example output.csv file:

  
Time,Cpu,Mem
11:27:16.394591,0,2136
11:27:17.396917,0.004986567,2136
11:27:18.397440,0.006548807,2136

Note: For monitoring one-shot applications - see https://github.com/yarenty/app_benchmark.

Benchmarker

2023-09-10T08:24:00.000-07:00

Hi,

This is one of my first Rust apps.

I use it to benchmark an application - run it multiple times and get readings + graphs.

Benchmark

Benchmarking data collector - runs an application as a child process, collecting stats (time, CPU usage, memory usage) and generating benchmarking reports.

Why

Created it for 2 reasons:

- to check/learn how to create and manage full rust applications using the whole ecosystem - crates/builds/publishing

- personal needs to get benchmarks for different other projects

Code

https://github.com/yarenty/app_benchmark

https://crates.io/crates/app-benchmark

High-level idea

run the application multiple times
collect all interested readings:
- time
- CPU
- memory
process outputs and provide results as:
- CSV/excel
- graphs

Save outputs to local DB/file to check downgrade/speedup in the next release of an application.

Methodology

For each benchmark run:

run multiple times (default 10)
remove outliers
average output results

Build

cargo build -r --bin benchmark

Usage

benchmark 0.1.0
Benchmarking data collector.

USAGE:
    benchmark [OPTIONS] <APPLICATION>

ARGS:
    <APPLICATION>    Application path (just name if it is in the same directory)

OPTIONS:
    -h, --help           Print help information
    -l, --log <LOG>      Set custom log level: info, debug, trace [default: info]
    -r, --runs <RUNS>    Number of runs to be executed [default: 10]
    -V, --version        Print version information

Example output

09:33:24.899 (t: main) INFO - benchmark - Application to be benchmark is: /opt/workspace/ballista/target/release/examples/example_processing
09:33:24.899 (t: main) INFO - benchmark - Number of runs: 10
09:33:24.902 (t: main) INFO - benchmark - Collecting data::example_processing
09:33:24.902 (t: main) INFO - benchmark::bench::analysis - Run 0 of 10
09:33:24.947 (t: main) INFO - benchmark::bench::analysis - Run 1 of 10
09:33:24.983 (t: main) INFO - benchmark::bench::analysis - Run 2 of 10
09:33:25.016 (t: main) INFO - benchmark::bench::analysis - Run 3 of 10
09:33:25.049 (t: main) INFO - benchmark::bench::analysis - Run 4 of 10
09:33:25.087 (t: main) INFO - benchmark::bench::analysis - Run 5 of 10
09:33:25.132 (t: main) INFO - benchmark::bench::analysis - Run 6 of 10
09:33:25.188 (t: main) INFO - benchmark::bench::analysis - Run 7 of 10
09:33:25.238 (t: main) INFO - benchmark::bench::analysis - Run 8 of 10
09:33:25.288 (t: main) INFO - benchmark::bench::analysis - Run 9 of 10
09:33:25.338 (t: main) INFO - benchmark - Processing outputs
0.04,130,18752,
0.03,140,18664,
0.03,156,18856,
0.03,153,18868,
0.04,152,18884,
0.04,140,18904,
0.05,136,19404,
0.05,145,19220,
0.05,137,18780,
0.05,138,18788,
09:33:25.339 (t: main) INFO - benchmark::bench::collector - SUMMARY:
09:33:25.339 (t: main) INFO - benchmark::bench::collector - Time [ms]:: min: 30, max: 50, avg: 41 ms
09:33:25.339 (t: main) INFO - benchmark::bench::collector - CPU [%]:: min: 130, max: 156, avg: 142.7 %
09:33:25.339 (t: main) INFO - benchmark::bench::collector - Memory [kB]:: min: 18664, max: 19404, avg: 18912 kB

Process finished with exit code 0

Also in the current directory of the benchmark app, there is an output directory created named "bench_<your_app_name>", ie: bench_example_processing, which contains:

Output CSV file:

benchmarks.csv

Time,Cpu,Mem
0.04,130,18752
0.03,140,18664
0.03,156,18856
0.03,153,18868
0.04,152,18884
0.04,140,18904
0.05,136,19404
0.05,145,19220
0.05,137,18780
0.05,138,18788

and output graphs:

summary report: summary_report.txt

TEST

cargo build --example test_app -r   

cargo run --bin benchmark -- /opt/workspace/app_banchmark/target/release/examples/test_app   

cargo run --bin benchmark -- "/opt/workspace/app_banchmark/target/release/examples/test_app -additionl -app -params"

TODO:

incremental runs - use date/time in output dir
local db / or file struct to see changes with time/application trends
move out from GNU time dependency to sysinfo

Note: For monitoring long-running processes like servers / streaming apps - see https://github.com/yarenty/app_tracer.

Web 3 - blockchain layers

2022-01-29T16:30:00.001-08:00

Layers from a blockchain perspective.

My plan is to write 5 articles:

1 Intro: Web 1.. 2.. 3..

2 Layers in crypto. [this one]

3 Applications - not only DeFi!

4 Decentralisation

5 Summary - where we are, where to look, why should we join

Layer 1

Layer 1 refers to the underlying blockchain architecture, i.e., the actual blockchain itself. In the case of Bitcoin, it is the BTC network launched in 2009.

Layer 2

Layer 2 refers to various protocols that are built on top of layer 1 to improve the original blockchain’s functionality. Layer 2 protocols often use off-chain processing elements to solve the speed and cost inefficiencies of the layer 1 network. Examples of layer 2 platforms for Bitcoin include Lightning Network and Liquid Network.

Layer 3

Layer 3 is represented by blockchain-based applications, such as decentralized finance (DeFi) apps, games, or distributed storage apps. Many of these applications also have cross-chain functionality, helping users access various blockchain platforms via a single app.

A layer-1 blockchain is a set of solutions that improve the base protocol itself to make the overall system a lot more scalable. There are two most common layer-1 solutions, and these are the consensus protocol changes as well as sharding.

When it comes to consensus protocol changes, projects like Ethereum are moving from older, clunky consensus protocols such as proof-of-work (PoW) to much faster and less energy-wasteful protocols such as proof-of-stake (PoS).

Sharding is one of the most popular layer-1 scalability methods out there as well. Instead of making a network sequentially work on each transaction, sharding breaks these transaction sets into small data sets which are known as "shards," and these can then be processed by the network in parallel.

One of the pros when it comes to layer-1 solutions is that there is no need to add anything on top of the existing infrastructure.

Layer 2 is a term used for solutions created to help scale an application by processing transactions off of the Ethereum Mainnet (layer 1) while still maintaining the same security measures and decentralization as the mainnet. Layer 2 solutions increase throughput (transaction speed) and reduce gas fees. Popular examples of Ethereum layer 2 solutions include Lightning Network, Liquid Network, Polygon, and Polkadot.

Layer 2 solutions are important because they allow for scalability and increased throughput while still holding the integrity of the Ethereum blockchain, allowing for complete decentralization, transparency, and security while also reducing the carbon footprint (less gas, means less energy used, which equates to less carbon.)

Although the Ethereum blockchain is the most widely used blockchain and arguably the most secure, that doesn’t mean it doesn’t come with some shortcomings. The Ethereum Mainnet is known to have slow transaction times (13 transactions per second) and expensive gas fees. Layer 2s are built on top of the Ethereum blockchain, keeping transactions secure, speedy, and scalable.

Each individual solution has its own pros and cons to consider such as throughput, gas fees, security, scalability, and of course functionality. No single layer 2 solution currently fulfills all these needs. However, there are layer 2 scaling solutions which aim to improve all these aspects; these solutions are called rollups.

There are three properties of a layer 2 rollup:

1. Transactions are executed outside of layer 1 (reduces gas fees)

1. Data and proof of transactions reside on layer 1 (maintains security)

1. A rollup smart contract which is found on layer 1, can enforce proper transaction execution on layer 2, by using the transaction data that is stored on layer 1

Layer 3 is often referred to as the application layer. It is a layer that hosts DApps and the protocols that enable the apps. While some blockchains such as Ethereum or Solana (SOL) have a thriving variety of layer 3 apps, Bitcoin is not optimized to host such applications.

As such, layer 2 solutions are the furthest deviations from the core network that Bitcoin currently has. Some projects are trying to bring DApp functionality to the BTC ecosystem via forks of the original BTC network.

For instance, CakeDeFi is a DeFi app offering services such as staking, lending, and liquidity mining to BTC coin holders. CakeDeFi is based on a fork of Bitcoin called DeFiChain. DeFiChain maintains “an anchor” to the core BTC chain for some of its operations, but technically speaking, it is still a separate blockchain of its own.

Some industry observers believe that the lack of DApp functionality is one of the biggest limitations of BTC. Ever since Ethereum’s arrival in 2015, layer 3 platforms have been growing strongly in popularity and value. Ethereum currently has close to 3,000 layer 3 apps. The DeFi apps based on the blockchain hold a total value of $185 billion by now.

Another leading blockchain, Solana, hosts over 500 layer 3 DApps, and the total value locked in the DeFi apps of the network is approaching $15 billion.

In comparison, BTC has no functioning app that could be clearly defined as a layer 3 application. There is an ongoing debate about whether projects designed to “force in” DApp functionality onto BTC are worth the effort. Some in the industry argue that BTC will always remain a network designed for crypto fund transfers, not DApps.

These people point out that the layer 1 BTC chain enjoys an industry-leading market cap (of $1.3 trillion by now) that dwarfs all the TVL and market cap figures of all layer 3 projects in existence combined. As such, Bitcoin may not be in any urgent need of layer 3 functionality, at least judging from the financial figures.

Summary

Blockchain platforms may have three distinct layers. Layer 1 refers to the actual underlying blockchain, with its core architecture and functionality. Examples of layer 1 networks are the Bitcoin, Ethereum, and Solana blockchains.

Layer 2 are protocols built on top of layer 1 networks and extend some functionality of the underlying blockchain. For example, they may offer faster speeds and lower transaction costs than layer 1.

Layer 2 protocols often use a combination of on-chain and off-chain operations to offer their extended functional capabilities. Examples of layer 2 projects on Bitcoin include the Lightning Network and Liquid Network platforms.

Layer 3 refers to the protocols that enable DApps on the blockchain. While some other blockchains have a large collection of layer 3 apps, the BTC blockchain has none of them. Some projects attempt to bring layer 3 functionality into the BTC ecosystem by using apps designed on forks of BTC.

However, these apps are still based on their own blockchains, not on the core BTC blockchain. There is a debate about whether BTC even needs to move towards enabling the layer 3 functionality. Some industry analysts argue that BTC is worth multiple times more than all these layer 3 apps combined, and therefore, it does not have a pressing need for layer 3 at all.

Web3 - next big bang!

2022-01-29T03:04:00.002-08:00

Short article about Web3 - what it is and why I personally think that for the next 3 years it will be must know / must use / must to be in !!

My plan is to write 5 articles:

1. Intro: Web 1.. 2.. 3.. [this one]

2. Layers in crypto.

3. Applications - not only DeFi!

4. Decentralisation

5. Summary - where we are, where to look, why should we join

Web3 is an idea for a new iteration of the World Wide Web-based on the blockchain, which incorporates concepts including decentralization and token-based economics. Some technologists and journalists have contrasted it with Web 2.0, wherein they say data and content are centralized in a small group of companies sometimes referred to as "Big Tech".

Web 1.0 and Web 2.0 refer to eras in the history of the World Wide Web as it evolved through various technologies and formats. Web 1.0 refers roughly to the period from 1991 to 2004, where most websites were static web pages, and the vast majority of users were consumers, not producers, of content.

Web 2.0 is based around the idea of "the web as platform" and centers on user-created content uploaded to social media and networking services, blogs, and wikis, among other services. Web 2.0 is generally considered to have begun around 2004 and continues to the current day.

Visions for Web3 differ, but they revolve around the idea of decentralization and often incorporate blockchain technologies, such as various cryptocurrencies and non-fungible tokens (NFTs). Bloomberg has described Web3 as an idea that "would build financial assets, in the form of tokens, into the inner workings of almost anything you do online". Some visions are based on the concept of decentralized autonomous organizations (DAOs). Decentralized finance (DeFi) is another key concept; in it, users exchange currency without bank or government involvement. Self-sovereign identity allows users to identify themselves without relying on an authentication system such as OAuth, in which a trusted party has to be reached in order to assess identity. Technology scholars have argued that Web3 would likely run in tandem with Web 2.0 sites likely adopting Web3 technologies in order to keep their services relevant.

To believers, Web3 represents the next phase of the internet and, perhaps, of organizing society. Web 1.0, the story goes, was the era of decentralized, open protocols, in which most online activities involved navigating to individual static webpages. Web 2.0, which we’re living through now, is the era of centralization, in which a huge share of communication and commerce takes place on closed platforms owned by a handful of super-powerful corporations—think Google, Facebook, Amazon—subject to the nominal control of centralized government regulators. Web3 is supposed to break the world free of that monopolistic control.

At the most basic level, Web3 refers to a decentralized online ecosystem based on the blockchain. Platforms and apps built on Web3 won’t be owned by a central gatekeeper, but rather by users, who will earn their ownership stake by helping to develop and maintain those services.

Gavin Wood coined the term Web3 (originally Web 3.0) in 2014. At the time, he was fresh off of helping develop Ethereum, the cryptocurrency that is second only to Bitcoin in prominence and market size. Today he runs the Web3 Foundation, which supports decentralized technology projects, as well as Parity Technologies, a company focused on building blockchain infrastructure for Web3. Wood, who is based in Switzerland, spoke with me last week over a video about where Web 2.0 went wrong, his vision of the future, and why we all need to be less trusting. The following interview is a transcript of our conversation, lightly edited for clarity and length.

As technology continues to take center stage as a key differentiator for companies, new tech trends for 2022 are beginning to emerge. These trends are largely reflective of the changing realities around us. Successive global lockdowns have opened up a world of possibilities around virtual experiences and digital interactions. The increased urgency around resource scarcity, both human and natural, has also led to the introduction of technology geared towards efficiency and sustainability. These growing trends are part of a larger directional shift of the world wide web to Web 3.0.

To understand the concept of Web 3.0, we need to take a step back and understand the larger evolution of the internet that brought us to this point.

Web 1.0, the original conception of the internet, was a largely static, one-to-many format where users could view web pages but do little beyond that.

Web 2.0 introduced the concept of a worldwide community of internet users, encouraging them to form social media groups, interact with each other and create virtual experiences. Web 2.0, the current form of the internet, is still by large the most influential, but Web 3.0 is now slowly coming into its own.

Web 3.0 takes the notion of the ‘community and expands it to include community ownership and regulation of the internet as a whole. Three key philosophies are involved in web 3.0 definition:

The internet is ‘open’ and is built with open-source software that anyone can create, utilize and modify.

Interactions between users are not governed by a trusted third-party regulatory body.

Anyone can participate, without requiring permission from governments or regulatory bodies.

We can already see many manifestations of web 3.0 all around us. Cloud technology and artificial intelligence are some of the most prevalent forms of web 3.0 today.

In 2022, we will become increasingly familiar with the concept of a “metaverse” – persistent digital worlds that exist in parallel with the physical world we live in – Forbes

Here are **a few of the top technological trends of 2022** that web 3.0 has given rise to.

1. Advanced applications of Artificial Intelligence

Artificial Intelligence has arguably been one of the biggest technological innovations of our time. As AI begins to get more sophisticated and closely mimic human intelligence, a new form will gain popularity in 2022: generative AI. Unlike traditional AI models that simply understand repetitive patterns and recreate them, generative AI is capable of producing completely new material. It uses the underlying principle of AI, learning patterns, to identify how input is linked together and uses it to create new content from code, images, text or video inputs.

One of the biggest applications of generative AI is in customer service. Chatbots of 2022 could be so human-like that it would be practically impossible to differentiate between an AI chatbot and a human rep. AI could also change the way we consume content. Social media algorithms could get a lot smarter with AI and offer more accurate content recommendations. This could deliver even more personalized experiences for customers.

2. Low-code application building software

Creating a strong digital presence has become increasingly important for businesses, especially in the wake of the lockdowns. Mobile applications are one of the most important virtual assets a business can have because it’s the closest way to mimic an in-store experience. Historically, however, applications have been expensive to develop. Businesses needed to either have an in-house software development team or enough financial budgets to outsource the project to a vendor. These have acted as strong barriers to entry and have meant that small businesses could not afford to launch a mobile app. But with web 3.0, a number of low-code or no-code app development platforms have begun to sprout. These platforms make it extremely easy to develop an app with little to no coding knowledge required. Usually, they come with preset templates and features and users have to simply drag and drop them to build their app. These services will undoubtedly democratize the development of mobile applications and help businesses engage with their customers online.

3. Dominance of cloud technology trends

The cloud was one of the biggest game-changing technologies of 2021 as it enabled businesses to work remotely, from any part of the world. But even as offices begin to re-open, it’s unlikely that we’re ever going to see a shift to old on-premise models again. Global scenarios are unpredictable and businesses need to remain agile. As companies plan for this volatile future, they are more likely to move away from hastily implemented cloud setups to cloud-native platforms.

The ‘lift and shift’ approach involved using the same organizational processes and simply moving them to the cloud. This approach might have worked as a time-sensitive solution, but they don’t deliver the full benefits of the cloud in the long term. A major focus for CIOs in the coming year will be implementing more sustainable cloud structures that are scalable and flexible.

One study by Gartner estimates that as much as 95% of new digital initiatives in 2022 will be built upon cloud-native platforms. This underscores the importance of cloud technology in the coming year.

4. Secure data fabric

In line with creating flexible and accessible platforms, data fabric is a type of data architecture that ties together different platforms and users. It’s the antidote to data silos of the past that would often result in a loss of critical insights and make access extremely restricted. A data fabric promises greater efficiency and security because data is stored on a secure cloud platform. This minimizes the cost of storing data and ensures the highest level of encryption.

Think of a data fabric like a self-driving car. When the driver is active then the autonomous mode takes a back seat. If the driver gets lazy and is not too alert then the semi-autonomous mode kicks in and makes course corrections, This is similar to how a data fabric works. It will monitor all data pipelines as a passive observer and once it understands, it will start to suggest more productive alternatives. For instance, a supply chain organization that uses data fabric can keep adding newly encountered data information that can improve decisions based on the newly integrated data.

To implement data fabric design, new technologies like semantic knowledge graphs, embedded machine learning etc. will be needed.

5. Growth of 5G technology

5G technology will ultimately be the backbone that drives all of the above technology trends. 5G networks have a much wider reach than 4G, offer faster internet speeds and even enable heavy code to download seamlessly. 2022 will most likely see 5G enter the mainstream market and become the most widespread form of mobile broadband connectivity. This presents a huge opportunity for businesses. As the new year quickly approaches, CIOs should brainstorm new ways to present their business digitally and new channels that they can open up, powered by 5G technology.

Technology is constantly evolving and these 5 tech trends are just a few of the many we expect to see develop over the next year. As the very form of the internet evolves with the introduction of web 3.0, there’s only one thing that’s for certain: businesses need to respond positively to change and leverage the latest technological advances if they want to maintain their competitive edge.

Web 2 was boom - lots of new companies emerged, Web 3 will go into the same direction with a possibility to be a much bigger change to our way of living!

AI/ML issue

2020-10-07T06:53:00.006-07:00

When talking about AI, there is often seen the concept of "one-shot-learning". This is quite a basic understanding difference between small baby learning and machine learning ;-).

In machine learning to create a good quality model, one must provide a large number of sample (train) data.

However, in real life, a small baby quite quickly learns what a cat and a dog look like.

But the most visible example is learning that something is hot:

- on just one sample when the kid burns his hand, he will learn what hot is and will never touch anything hot again - at least not on purpose

- if we give this to existing machine learning solutions - it will need thousands of samples, preferably of hot and cold, to learn and give the prediction "don't touch is hot" ;-)

The answer to this dilemma is "one-shot-learning", where we try to build models that could learn from a single sample.

However, my issue is completely different: my baby was observing me, and the household. She noticed that we put lots of stuff into the trash can. She is interested. And since she is learning by repetition/duplication of what she can see ... currently she puts everything she finds into the bin.

By not breaking her confidence, so she will continue development at high speed;-) How to teach her that whatever she is doing is wrong? Basically: stop.

The obvious note is: that on multiple occasions, during the day she constantly sees new examples (train data) that this is standard/expected behavior.

Good design instead of big architecture

2019-09-23T05:03:00.000-07:00

Architecture: instead of multiple fancy libraries, and overcomplicated pipelines/flows/services/microservices - just write good code.

Just want to share my support to the idea put onto the article from Uber guys :
"Software Architecture is Overrated, Clear and Simple Design is Underrated"

There are few points why I like it, especially since for some time I worked in an environment where managers were always thinking that using fancy jargon, and adding as much as possible different libraries make the solution "better" / "cooler".

Yes, I admit, I'd like to use nice new fancy libraries, but my excuse is I want to be in business and want to learn new stuff, from project/management perspective there is however one issue: at the end of the day - who will maintain that and why?

The second output of this article - if you build good software, then the architect position is .. obsolete. I know it's hypocrisy, on linkedin I use that title - and again my excuse is careerwise perspectives - "architect" is understandable by everyone in the programming world and by the head hunters.

Design patterns - no more

2019-09-23T04:51:00.001-07:00

Design patterns - choose good programming language without internal "philosophical" gaps, then you don't need to learn patterns!

Yes, I admit I have aversion to patterns, for a simple reason - if programming language that you choose is good ... you don't need patterns, you don't need workarounds to cover language issues.

In my opinion patterns are just workarounds/cover-ups for language gaps/inconsistency.
It looks like: you build the house on the tree and you forgot to put a ladder, so you cannot reach the doors. Then comes patterns: use ropes, build concrete staircases, etc...
If you spend time on planing tree-house in the first place - no need for workarounds.

Just to be clear yes I'm talking about java - if you want to write a good program you need to know the whole catalog of patterns (starting with GOF). If you compare it to python, scala (tobe checked: kotlin, swift) - no more patterns. There are some suggested good behaviors like use functional instead of OO, but that is your choice/ your style.

I think this is an advantage of new languages - if you want to promote a new language you need to have it clean and easy to learn and use.

Ubuntu graphics resolution 640x480 fix

2018-05-29T02:25:00.001-07:00

I stuck with 640x480 for some time ... so here is 3 easy steps to fix it:

Note: Tested on ubuntu 16.04 / 17.10 / 18.04 (bionic)

1) go to native terminal

ctrl+alt+F1

2) disable windows manager (step 1)

sudo lightdm stop

3) create new xorg config file (step 2)

sudo Xorg -configure

4) backup old xorg file ... (optional step for peace of mind)

sudo mv /etc/X11/xorg.conf /etc/X11/xorg.conf.bak

5) copy new configuration (step 3)

sudo mv /root/xorg.conf.new /etc/X11/xorg.conf

6) start lightdm (didn't work for me)

sudo lightdm start

7) reboot - this actually works!

reboot

and that's it ;-)

Upgrade all pip packages

2017-10-14T05:07:00.000-07:00

How to upgrade all installed Python packages at the same time?

pip freeze — local | grep -v ‘^\-e’ | cut -d = -f 1 | xargs -n1 pip install -U

Or pip3:

pip3 freeze — local | grep -v ‘^\-e’ | cut -d = -f 1 | xargs -n1 pip3 install -U

Solution: Ubuntu/Linux vs JDK => Maximum number of clients reached / unable to open display

2017-07-04T05:04:00.000-07:00

For work I'm using Ubuntu, and as developer, I constantly use some JDK tools/IDEs, usually it is IntelliJ IDEA, sometimes eclipse, some other specific, like nVidia studios..
I'm typical linux user - so "try it switch off and on" is not the standard way to deal with problems.
From some time when I tried to start some graphical tool I got this error:

Maximum number of clients reached unable to open display ":0"

I hate it and spend lots of time for investigation, went throug all standard:

xlsclients/xwininfo/xhost - which didn't work as they all need access to X (they are clients) - so typical catch 22

I spend some time on different ubuntu settings for X-org/ file limits etc - everything set properly, all almost unlimited...
then finally by just some good luck I shut down my IntelliJ and ... checked that thousand of resources suddenly disappeared from usage list..

Final solution: you need suckless-tools ;-)

apt install suckless-tools

BREW - clean update / upgrade

2016-11-15T08:00:00.000-08:00

Another one for future reference: brew
when I did brew update I've used to get:

>brew update
error: Your local changes to the following files would be overwritten by merge:

share/doc/homebrew/Acceptable-Formulae.md
share/doc/homebrew/Bottles.md
share/doc/homebrew/Brew-Test-Bot-For-Core-Contributors.md
share/doc/homebrew/Brew-Test-Bot.md
share/doc/homebrew/C++-Standard-Libraries.md
share/doc/homebrew/Common-Issues-for-Core-Contributors.md
share/doc/homebrew/Common-Issues.md
share/doc/homebrew/Custom-GCC-and-cross-compilers.md
share/doc/homebrew/External-Commands.md
share/doc/homebrew/FAQ.md
[......]

fix for that is:

cd `brew --prefix`
git fetch origin 
git reset --hard origin/master

if reset didn't work sometimes you need to do sudo, just make sure you change owner back to your user...

sudo chown -R user .

then :

brew update
brew upgrade

and all is nice and working cool ... again ;-)

R - upgrade all packages

2016-11-14T03:47:00.002-08:00

How to upgrade R?

There is this package, which you can use .... but you must have it installed before new version of R come ;-) - otherwise it is typical "catch 22":

Upgrade from 3.3.1 to 3.3.2 :

install.packages('installr', repos = 'http://cran.us.r-project.org')

Warning message:

package ‘installr’ is not available (for R version 3.3.1)

So basically - instead this go to https://cran.r-project.org/ and update using new R.

How to upgrade all your existing R packages (libraries)?

Use this command to upgrade all existing packages:

install.packages(
.libPaths()[1],
pkgs = as.data.frame(installed.packages(.libPaths()[1]), stringsAsFactors=FALSE)$Package,
type = 'source'
)

And quick update - to the article ;-) - use this instead:

update.packages()

update.packages(ask = FALSE, dependencies = c('Suggests'))

Have a fun!

Mac OS X - switch off discrete GPU in apps

2016-04-06T05:18:00.000-07:00

Hi - just quick one.
I have this program (WriteApp) which is quite cool, but every time I'm using it, it is automatically switching my mac into discrete graphics card mode. This app is not really into anything fancy/ high graphics dependent.
But when I launch Activity Monitor it is displayed as: "Requires High Perf GPU" - which means that my app spur the system to keep the advanced graphics processor active, and thereby drain battery life.

After some research ... I didn't find any solution apart using gfxCardStatus to permanently switch into "Integrated only" mode.

So ... there is a hack:
Open your app contents and inside them should be Info.plist file. Open it and add those 4 lines:

       <key>NSHighResolutionCapable</key>

       <true/>

<key>NSSupportsAutomaticGraphicsSwitching</key>

       <true/>

restart your app... and now - no more forcing into discrete mode.

10 Useful, Yet Paranoid, Java Programmer Techniques

2015-10-29T06:40:00.000-07:00

Defensive java programming is a great way to prepare for the inevitable to go wrong -- but it can be just as paranoid as it is useful.

After coding for a while (eek, almost 20 years or so in my case, time flies when you’re having fun), one starts to embrace those habits. Because, you know…

Anything that Can Possibly Go Wrong, Does.

This is why people embrace “defensive programming”, i.e. paranoid habits that sometimes make total sense, and sometimes are rather obscure and/or clever and perhaps a bit eerie when you think of the person who wrote it. Here’s my personal list of top 10 useful, yet paranoid Java programming techniques. Let’s go:

1. Put the String Literal First

It’s just never a bad idea to prevent the occasional NullPointerException by putting the String literal on the left side of an equals() comparison as such:

// Bad
if (variable.equals("literal")) { ... }
// Good
if ("literal".equals(variable)) { ... }

This is a no-brainer. Nothing is lost from rephrasing the expression from the less good version to the better one. If only we had true Options though, right? Different discussion…

2. Don’t Trust the Early JDK APIs

In the early days of Java, programming must’ve been a big pain. The APIs were still very immature and you might’ve walked across a piece of code like this:

String[] files = file.list();
// Watch out
if (files != null) {
    for (int i = 0; i < files.length; i++) {
        ...
    }
}

Looking paranoid? Perhaps, but read the Javadoc:

If this abstract pathname does not denote a directory, then this method returns null. Otherwise an array of strings is returned, one for each file or directory in the directory.

Yeah right. Better add another check, though, just to be sure:

if (file.isDirectory()) {
    String[] files = file.list();
    // Watch out
    if (files != null) {
        for (int i = 0; i < files.length; i++) {
            ...
        }
    }
}

Bummer! Violation of rule #5 and #6 of our 10 Subtle Best Practices when Coding Java list. So be prepared and add that null check!

3. Don’t Trust That “-1″

This is paranoid, I know. The Javadoc of String.indexOf() clearly states that…

the index of the first occurrence of the character in the character sequence represented by this object [is returned], or -1 if the character does not occur.

So, -1 can be taken for granted, right? I say nay. Consider this:

// Bad
if (string.indexOf(character) != -1) { ... }
// Good
if (string.indexOf(character) >= 0) { ... }

Who knows. Perhaps they’ll need ANOTHER encoding at some point in time to say, the otherStringwould have been contained if checked case-insensitively… Perhaps a good case for returning -2? Who knows.

After all, we’ve had billions of discussions about the billion dollar mistake, which is NULL. Why shouldn’t we start discussions about -1, which is – in a way – an alternative null for primitive typeint?

4. Avoid the Accidental Assignment

Yep. It happens to the best (although, not to me. See #7).

(Assume this is JavaScript, but let’s be paranoid about the language as well)

// Ooops
if (variable = 5) { ... }
// Better (because causes an error)
if (5 = variable) { ... }
// Intent (remember. Paranoid JavaScript: ===)
if (5 === variable) { ... }

Again. If you have a literal in your expression, put it to the left side. You can’t accidentally go wrong here, when you meant to add another = sign.

5. Check for Null AND Length

Whenever you have a collection, array, etc., make sure it’s present AND not empty.

// Bad
if (array.length > 0) { ... }
// Good
if (array != null && array.length > 0) { ... }

You never know where those arrays come from. Perhaps from early JDK API?

6. All Methods Are Final

You can tell me all you want about your open/closed principles, that’s all bollocks. I don’t trust you (to correctly extend my classes) and I don’t trust myself (to not accidentally extend my classes). Which is why everything that is not explicitly intended for subtyping (i.e. only interfaces) is strictly final. See also item #9 of our 10 Subtle Best Practices when Coding Java list.

// Bad
public void boom() { ... }
// Good. Don't touch.
public final void dontTouch() { ... }

Yes. It’s final. If that doesn’t work for you, patch it, or instrument it, or rewrite the byte code. Or send a feature request. I’m sure that your intent of overriding the above isn’t a good idea anyway.

7. All Variables and Parameters Are Final

As I said. I don’t trust myself (to not accidentally overwrite my values). Having said so, I don’t trust myself at all. Because…

… which is why all variables and parameters are made final, too.

// Bad
void input(String importantMessage) {
    String answer = "...";
    answer = importantMessage = "LOL accident";
}
// Good
final void input(final String importantMessage) {
    final String answer = "...";
}

OK, I admit. This one, I don’t apply very often, really, although I should. I wish Java got it right like Scala, where people just type val all over the place, without even thinking about mutability – except when they need it explicitly (rarely!), via var.

8. Don’t Trust Generics When Overloading

Yes. It can happen. You believe you wrote that super nice API which totally rocks and is totally intuitive, and along comes some user who just raw-casts everything up to Object until the darn compiler stops bitching, and suddently they’ll link the wrong method, thinking it’s your fault (it always is).

Consider this:

// Bad
<T> void bad(T value) {
    bad(Collections.singletonList(value));
}
<T> void bad(List<T> values) {
    ...
}
// Good
final <T> void good(final T value) {
    if (value instanceof List)
        good((List<?>) value);
    else
        good(Collections.singletonList(value));
}
final <T> void good(final List<T> values) {
    ...
}

Because, you know… Your users, they’re like

// This library sucks
@SuppressWarnings("all")
Object t = (Object) (List) Arrays.asList("abc");
bad(t);

Trust me. I’ve seen everything. Including things like

It’s good to be paranoid.

9. Always Throw on Switch Default

Switch… One of those funny statements where I don’t know whether to petrify with awe or to just cry. Anyway, we’re stuck with switch, so we may as well get it right when we have to. I.e.

// Bad
switch (value) {
    case 1: foo(); break;
    case 2: bar(); break;
}
// Good
switch (value) {
    case 1: foo(); break;
    case 2: bar(); break;
    default:
        throw new ThreadDeath("That'll teach them");
}

Because that moment where value == 3 is introduced into the software, it’ll come for sure! And don’t say enum, because it’ll happen to enums as well!

10. Switch With Curly Braces

In fact, switch is the most wicked statement anyone has every allowed to get into a language while they were either drunk or lost a bet. Consider the following example:

// Bad, doesn't compile
switch (value) {
    case 1: int j = 1; break;
    case 2: int j = 2; break;
}
// Good
switch (value) {
    case 1: {
        final int j = 1;
        break;
    }
    case 2: {
        final int j = 2;
        break;
    }
    // Remember:
    default: 
        throw new ThreadDeath("That'll teach them");
}

Within the switch statement, there is only one scope defined among all the case statements. In fact, these case statements aren’t even really statements, they’re like labels and the switch is a goto call. In fact, you could even compare case statements with the astonishing FORTRAN 77 ENTRY statement, a device whose mystery is only exceeded by its power.

This means that the variable final int j is defined for all the different cases, regardless if we issue abreak or not. Not very intuitive. Which is why it’s always a good idea to create a new, nested scope percase statement via a simple block. (but don’t forget the break within the block!)

Conclusion

Paranoid programming may seem weird at times, as code often turns out to be a bit more verbose than really needed. You might think, “oh this is never gonna happen”, but as I said. After 20 years or so programming, you just don’t want to fix those stupid little unnecessary bugs anymore that exist only because the language is so old and flawed. Because you know…

Re-print from: Lukas Eder

Fast data processing with spark - review

2015-04-24T02:40:00.002-07:00

Hi,
I've got that book "Fast data processing with spark" and yesterday I spend on reviewing it..
I will not go into deep details, just quick one: it was quite disappointing book:

If you interested in one language (aka scala) then 1/3 of book is useless, as author is providing every example in 3 languages: scala/java/python.

Together there are ... 2 examples, so not much to share/learn.

Second example uses library that you cannot find anymore :-(, I mean - maybe if you spent more time to dig particular problem on geolocation - but do not use that book as geolocation solution. For general publicity not worth.

Explanations what's going on on examples are really light - and this is me being polite.

Book: Not recommended.

If you want to check "working solution" updated to 2015 code base with existing libraries, check my project on github:

https://github.com/yarenty/spark_data_processing

LWJGL3 - use GPU [OpenCL] for multithreaded calculations

2015-03-25T06:47:00.001-07:00

Lastly I spent some time investigating usage of GPU in computations, particularly nvidia CUDA. But during research I spent some time looking for other options, like OpenCL. So that brings me to this Lightweight Java Game Library.

Short intro:

LWJGL is a Java library that enables cross-platform access to popular native APIs useful in the development of graphics (OpenGL), audio (OpenAL) and parallel computing (OpenCL) applications. This access is direct and high-performance, yet also wrapped in a type-safe and user-friendly layer, appropriate for the Java ecosystem. LWJGL is an enabling technology and provides low-level access. It is not a framework and does not provide higher-level utilities than what the native libraries expose. As such, novice programmers are encouraged to try one of the frameworks or game engines that make use of LWJGL, before working directly with the library.

And after quick look - you can get the same amount of access to all part of GPU as you can on JCUDA ( using CUDA of course ;-) ).

Here is simple OpenCL example updated to use LWJGL3:

package com.yarenty.lwjgl3;

import org.lwjgl.opencl.CLUtil;
import org.lwjgl.BufferUtils;
import org.lwjgl.PointerBuffer;

import java.nio.FloatBuffer;
import java.nio.IntBuffer;
import java.util.List;

import org.lwjgl.opencl.CL;
import org.lwjgl.opencl.CLCreateContextCallback;
import org.lwjgl.opencl.CLDevice;
import org.lwjgl.opencl.CLPlatform;

import static org.lwjgl.opencl.CL10.*;
import static org.lwjgl.opencl.CLUtil.checkCLError;
import static org.lwjgl.system.MemoryUtil.NULL;
import static org.lwjgl.system.MemoryUtil.memDecodeUTF8;
 
public class OpenCLSum {
    // The OpenCL kernel
    static final String source =
        "kernel void sum(global const float *a, global const float *b, global float *answer) { "
        + "  unsigned int xid = get_global_id(0); "
        + "  answer[xid] = a[xid] + b[xid];"
        + "}";
 
    // Data buffers to store the input and result data in
    static final FloatBuffer a = toFloatBuffer(new float[]{1, 2, 3, 4, 5, 6, 7, 8, 9, 10});
    static final FloatBuffer b = toFloatBuffer(new float[]{9, 8, 7, 6, 5, 4, 3, 2, 1, 0});
    static final FloatBuffer answer = BufferUtils.createFloatBuffer(a.capacity());
 
    
 private static final CLCreateContextCallback CREATE_CONTEXT_CALLBACK = new CLCreateContextCallback() {
  @Override
  public void invoke(long errinfo, long private_info, long cb, long user_data) {
   System.err.println("[LWJGL] cl_create_context_callback");
   System.err.println("\tInfo: " + memDecodeUTF8(errinfo));
  }
 };
    
    public static void main(String[] args) throws Exception {
     
     System.setProperty("org.lwjgl.opencl.explicitInit","true");
     
     
        // Initialize OpenCL and create a context and command queue
        CL.create();
        System.out.println("CL created");
        
        CLPlatform platform = CLPlatform.getPlatforms().get(0);
        System.out.println("Platform created");

        PointerBuffer ctxProps = BufferUtils.createPointerBuffer(3);
  ctxProps.put(CL_CONTEXT_PLATFORM).put(platform).put(0).flip();
  System.out.println("CTX created");
  
  
        IntBuffer errcode_ret = BufferUtils.createIntBuffer(1);
  System.out.println("ERRCODE created");

        
        List devices = platform.getDevices(CL_DEVICE_TYPE_GPU);
       // long context = clCreateContext(platform, devices, null, null, null);
  long context = clCreateContext(ctxProps, devices.get(0).getPointer(), CREATE_CONTEXT_CALLBACK, NULL, errcode_ret);
  System.out.println("CONTEXT created");

  
  checkCLError(errcode_ret);
        //CLCommandQueue queue = clCreateCommandQueue(context, devices.get(0), CL_QUEUE_PROFILING_ENABLE, null);
        long queue = clCreateCommandQueue(context, devices.get(0).getPointer(), CL_QUEUE_PROFILING_ENABLE, errcode_ret);
  System.out.println("Command Q created");

        // Allocate memory for our two input buffers and our result buffer
        long aMem = clCreateBuffer(context, CL_MEM_READ_ONLY | CL_MEM_COPY_HOST_PTR, a, null);
        //long buffer = clCreateBuffer(context, CL_MEM_READ_ONLY, 128, errcode_ret);
  System.out.println("A Buffer created");
        clEnqueueWriteBuffer(queue, aMem, 1, 0, a, null, null);
        long bMem = clCreateBuffer(context, CL_MEM_READ_ONLY | CL_MEM_COPY_HOST_PTR, b, null);
  System.out.println("B Buffer created");
        clEnqueueWriteBuffer(queue, bMem, 1, 0, b, null, null);
        long answerMem = clCreateBuffer(context, CL_MEM_WRITE_ONLY | CL_MEM_COPY_HOST_PTR, answer, null);
  System.out.println("OUTPUT Buffer created");
        clFinish(queue);
 
        // Create our program and kernel
        long program = clCreateProgramWithSource(context, source, null);
  System.out.println("PROGRAM created");

     //public static int clBuildProgram(long program, long device, CharSequence options, CLProgramCallback pfn_notify, long user_data) {

      
        CLUtil.checkCLError(clBuildProgram(program, devices.get(0).getPointer(), "", null, 0L));
        // sum has to match a kernel method name in the OpenCL source
        long kernel = clCreateKernel(program, "sum", null);
  System.out.println("KERNEL created");

        // Execution our kernel
        PointerBuffer kernel1DGlobalWorkSize = BufferUtils.createPointerBuffer(1);
        System.out.println("KERNEL work size created");
        kernel1DGlobalWorkSize.put(0, a.capacity());
        System.out.println("KERNEL work size copied");
        
        clSetKernelArg1p(kernel, 0, aMem);
        clSetKernelArg1p(kernel, 1, bMem);
        clSetKernelArg1p(kernel, 2, answerMem);
        
  System.out.println("Args send to kernel");

        clEnqueueNDRangeKernel(queue, kernel, 1, null, kernel1DGlobalWorkSize, null, null, null);
        System.out.println("KERNEL queued created");
        
        
        // Read the results memory back into our result buffer
        clEnqueueReadBuffer(queue, answerMem, 1, 0, answer, null, null);
        System.out.println("and output ... created");
        
        clFinish(queue);
        // Print the result memory
        print(a);
        System.out.println("+");
        print(b);
        System.out.println("=");
        print(answer);
 
        // Clean up OpenCL resources
        clReleaseKernel(kernel);
        clReleaseProgram(program);
        clReleaseMemObject(aMem);
        clReleaseMemObject(bMem);
        clReleaseMemObject(answerMem);
        clReleaseCommandQueue(queue);
        clReleaseContext(context);
        CL.destroy();
    }
 
 
    /** Utility method to convert float array to float buffer
     * @param floats - the float array to convert
     * @return a float buffer containing the input float array
     */
    static FloatBuffer toFloatBuffer(float[] floats) {
        FloatBuffer buf = BufferUtils.createFloatBuffer(floats.length).put(floats);
        buf.rewind();
        return buf;
    }
 
 
    /** Utility method to print a float buffer
     * @param buffer - the float buffer to print to System.out
     */
    static void print(FloatBuffer buffer) {
        for (int i = 0; i < buffer.capacity(); i++) {
            System.out.print(buffer.get(i)+" ");
        }
        System.out.println("");
    }
 
}

and you can find full scala project here: https://github.com/yarenty/scala_lwjgl3

Defaults - base start configurations for new projects.

2015-03-20T03:43:00.002-07:00

Every single time I'm starting new projects there is time which I spent on "organising" stuff:
- for maven create initial pom.xml
- for scala - build.sbt
- or define gitignore for different type of projects.
I know that there is some handy tools like gitignore generators, etc...
Tell you truth every single time I finish with going through all my previous projects and trying to find something close to the one that I'm doing at the moment. Then look into libraries to find if there are some new ones worth upgrading ... Yes, i know it could be fun, but ...

Finally I decided to do some life hack and have my BASE DEFAULTS - I created this project: https://github.com/yarenty/DEFAULTS

And hope I will keep them updated.

PS: Apart standard one I'm keeping stuff like "must install in new eclipse"

Findbugs:
http://findbugs.cs.umd.edu/eclipse
PMD:
http://sourceforge.net/projects/pmd/files/pmd-eclipse/update-site/
Checkstyle:
http://eclipse-cs.sourceforge.net/update/
or new marketplace version:
http://eclipse-cs.sourceforge.net/

Quick PX to EM conversion table for responsive web layouts

2015-02-04T05:42:00.001-08:00

This is simple table to remember or to do automatic replace when you'll go from old non-responsive design to responsive one.

Just one tip: when you'll replace automatically start from the biggest number ;-)

px	em
1 px	0.0625
2 px	0.125
3 px	0.1875
4 px	0.25
5 px	0.3125
6 px	0.375
7 px	0.4375
8 px	0.5
9 px	0.5625
10 px	0.625
11 px	0.6875
12 px	0.75
13 px	0.8125
14 px	0.875
15 px	0.9375
16 px	1.0
17 px	1.0625
18 px	1.125
19 px	1.1875
20 px	1.25
21 px	1.3125
22 px	1.375
23 px	1.4375
24 px	1.5
25 px	1.5625
26 px	1.625
27 px	1.6875
28 px	1.75
29 px	1.8125
30 px	1.875
31 px	1.9375
32 px	2.0
33 px	2.0625
34 px	2.125
35 px	2.1875
36 px	2.25
37 px	2.3125
38 px	2.375
39 px	2.4375
40 px	2.5
50 px	3.125
80 px	5.0
100 px	6.25
120 px	7.5
640 px	40.0
920 px	57.5
960 px	60.0

Ubuntu 14.04 + apple time capsule

2014-12-16T11:22:00.001-08:00

Finally got access to time capsule.

1) install libraries:

sudo apt-get install libgcrypt11 libgcrypt11-dev libgmp3-dev readline-common libreadline6 libreadline6-dev libfuse2 libfuse-dev libncurses5-dev

2) download afpfs (link here)
3) unarchive afpfs ( tar -xzvf )
4) navigate to afpfs (cd afpfs*)
5) you will have to patch the afpfs because it has a bug in ubuntu 12.04 /13.04 and still in 14.04:

wget -O offset.diff https://bugzilla.redhat.com/attachment.cgi?id=397155 && patch -p1 < offset.diff

wget -O afpfs-ng-0.8.1-pointer.patchhttps://bugzilla.redhat.com/attachment.cgi?id=505576 && patch -p1 < afpfs-ng-0.8.1-pointer.patch

6) supposing you are in afpfs folder you archived:

sudo configure --prefix=/usr

sudo make install

7) now you should be able to connect to your timecapsule using this:

mkdir ~/capsule

8) Command for MOUNT:

mount_afp afp://user:password@server_host_or_ip/Data ~/capsule

9)Command for UNMOUNT:

afp_client unmount ~/capsule

You should see a “capsule” folder in your home folder.
In my case Time Capsule stands on 10.0.1.1

Thanks to alex!

Scala1: Scala is craaazy

2014-12-10T20:00:00.000-08:00

I finished scala course provided by Martin Odersky. And just want to test them in some real case scenario. I had quite easy task: there is csv file, with headers, which need to be imported to database. Normal times when I used java I prepared myself for few days of coding. Typical steps are:

csv: find/list java csv processing libraries
csv: read / investigate few of them
csv: start playing, when one do not work as expected - check another one
db: fight with proper driver
db: find how to connect
db: prepare statements/sqls/ etc..
finally connect csv+db into one solution, and make it work smoothly

And after few days I would have working solution with probably few hundred lines of code...(not to mention tests/documentation).

In scala (and I am novice):
1. created my single object "importer.scala"
2. whole csv solution finished with adding to my build.sbt file single line:

libraryDependencies += "com.github.tototoshi" %% "scala-csv" % "1.1.2"

as my csv file using ";" as separator I put :

implicit object MyFormat extends DefaultCSVFormat {
override val delimiter: Char = ';' }

then to main function I added 2 lines of code:

val reader = CSVReader.open("myfile.csv")(MyFormat)
val full = reader.allWithHeaders

and ... that is it - whole csv solution done in less than 15 mins! (and yes in java I also used sbt like solution - i.e.:maven)

3. DB part:

Connection - again to build.sbt, added just one line (I'm using MySQL):

libraryDependencies += "mysql" % "mysql-connector-java" % "5.1.34"

.. and again to main function just added database connection

val dbc = "jdbc:mysql://localhost:3306/mydb?user=me&password=123"
Class.forName("com.mysql.jdbc.Driver")

val conn = DriverManager.getConnection(dbc)

and coding ... lets go ;-) :

try {
full.foreach(fields => processFields(fields, conn))
} finally {
conn.close
}

I know that I could do everything inside "full.foreach" but I had some old java habits to create methods for different actions... so I just added processFields function.

def processFields(in: Map[String, String], conn: java.sql.Connection) {
val l: MutableList[String] = new MutableList[String]()
fieldTypes.foreach(f =>
l += (in.get(f).get).replace("\"", "\\\""))
val sql = "INSERT INTO temp_products (" + myTypes.mkString(",") + ") VALUES ( " + "\"" + l.mkString("\",\"") + "\")"
val prep = conn.prepareStatement(sql)
prep.executeUpdate
}

As you see I had issue with ["] sign - which sometimes appeared inside csv fields.
This is plain/native sql - and whole solution takes me less than 2 hours.

Must say: I was little disappointed with ... so little effort to make things work.
And all I had is 60 lines of code - I really don't see reason to write comments/documentation as this is so obvious (if you know scala) that basically anything more will be overhead.

In next post I will show you about slick - proper DB/ORM mapping in scala and .. how easy is to do reverse engineering - generate objects from DB tables and do some crazy sqls without sql ;-)

Websites - Mobile Development

2014-10-09T05:37:00.000-07:00

Few tools and tricks for mobile website development - very RAW info.

Simulators

Android AVD Manager
Blackberry Ripple Emulator
Opera Mobile
Windows Phone Emulator

IDE

Aptana

Validation tools

http://validator.w3.org

W3C HTML5 validator
W3C CSS Validation Service

Debugging tools

Most browsers (F12 ;-) ) + Chrome, Firefox and OperaMobile support remote debugging
To stay sane: get your site working with a desktop browser and emulator then test on real devices.

Responsive Web Design

Fluid layout width - layouts that resize to fit in the available screen space
Media queries - load different CSS depend on screen size
flexible images and media - to suit screen size and bandwidth
typesetting: use em or % instead px

Box Problem

.borderBox *
{
-moz-box-sizing:border-box;
-webkit-box-sizing:border-box;
box-sizing:border-box;
}

RWD Formula

result = target/context
25px (target) / 16px (context) = 150% (result) => 1.5 em

CSS reset

Reset style sheets between different browsers (for consistency):

Eric Meyer's Reset
HTML5 Doctor CSS Reset
Yahoo! Reset CSS
Normalize.css

http://cssreset.com

Media queries

@media screen and (max-width: 479px) {
# header-logo {
width:75%
margin: 0 auto 0 } }
@media screen and (max-width: 737px) {
# header-logo {
width:100%
margin: 0 0 0 0 } }

Support good across browsers, except IE ( from IE9+) - use respond.js to emulate:

https://github.com/scottjehl/Respond

Screen widths

ICON

http://realfavicongenerator.net/

Responsive Navigation Menus

http://speckyboy.com/2012/08/29/10-responsive-navigation-solutions-and-tutorials/

Images

Prevent media from overflowing it's container:

img,embed,object, video {
max.width: 100%
}

for IE:

img.rwd { width:100%; height:auto; }

Image Adaption

Server-Side

WURFL
Adaptive images ( http://adaptive-images.com)

Client-Side

HiSRC jQuery plugin https://github.com/teleject/hisrc
Picturefill https://github.com/scottjehl/picturefill

HTML5 <picture> element

Supported in older browsers by Picturefill script

<picture>
<source media="(min-width: 45em)" srcset="large.jpg">
<source media="(min-width: 18em)" srcset="med.jpg, med-hd.jpg 2x">
<source srcset="small.jpg">
<img src="small.jpg" alt="some text">
<p>some text</p>
</picture>

JQuery Mobile - jQM

Gallery: http://www.jqmgallery.com

data-role:

page
header
content
footer

ThemeRoller: http://jquerymobile.com/themeroller

Modernizr

Open-source JavaScript feature-detection library http://modernizr.com

if (Modernizr.touch) { // code for toechscreen
} else { // no touchscreen
}

Modernizr.localStorage (sessionStorage)

Modernizr.geolocation

data.setCurrentLocation = function () {
   if (Modernizr.geolocation) {
navigator.geolocation.getCurrentPosition(data.geoSuccess, data.geoError);
} else {
data.geoError('not supported');
}
};

data.geoSuccess = function (position) {
var s = document.querySelector('#status');

if (s.className == 'success') { //Firefox fix
   return;
}

   s.className = 'success';

   data.currentLatLng = new google.maps.LatLng(position.coords.latitude, position.coords.longitude);

var myOptions = {
zoom: 15,
center: data.currentLatLng,
mapTypeControl: true,
navigationControlOptions: {style: google.maps.NavigationControlStyle.SMALL},
mapTypeId: google.maps.MapTypeId.ROADMAP
};

data.currentLocationMap = new google.maps.Map(document.getElementById("map_canvas"), myOptions);
var currentPositionMarker = new google.maps.Marker({
map: data.currentLocationMap,
position: data.currentLatLng
});
};

CUDA - Java integration

2014-09-08T07:19:00.000-07:00

The last time I worked with CUDA was 3 years ago, and the only option to integrate was by using a bridge through C -> JNI (or JNI-like). Now I did some quick research on possibilities that are there and ... there are lots of them.

There are existing 3 types of integration:

(Byte)code translation and OpenCL code generation
Language extensions
Java OpenCL/CUDA binding libraries

(Byte)code translation and OpenCL code generation

http://code.google.com/p/aparapi/ : An open-source library that is created and actively maintained by AMD. In a special "Kernel" class, one can override a specific method that should be executed in parallel. The byte code of this method is loaded at runtime using its own bytecode reader. The code is translated into OpenCL code, which is then compiled using the OpenCL compiler. The result can then be executed on the OpenCL device, which may be a GPU or a CPU. If the compilation into OpenCL is not possible (or no OpenCL is available), the code will still be executed in parallel, using a Thread Pool.
https://github.com/pcpratts/rootbeer1 : An open-source library for converting parts of Java into CUDA programs. It offers dedicated interfaces that may be implemented to indicate that a certain class should be executed on the GPU. In contrast to Aparapi, it tries to automatically serialize the "relevant" data (that is, the complete relevant part of the object graph!) into a representation that is suitable for the GPU.
http://code.google.com/p/java-gpu/ : A library for translating annotated Java code (with some limitations) into CUDA code, which is then compiled into a library that executes the code on the GPU. The Library was developed in the context of a PhD thesis, which contains profound background information about the translation process.
http://code.google.com/p/scalacl/ : Scala bindings for OpenCL. Allows special Scala collections to be processed in parallel with OpenCL. The functions that are called on the elements of the collections can be usual Scala functions (with some limitations) which are then translated into OpenCL kernels.

Language extensions

http://www.ateji.com/px/index.html : A language extension for Java that allows parallel constructs (e.g. parallel for loops, OpenMP style) which are then executed on the GPU with OpenCL. Unfortunately, this very promising project is no longer maintained.
http://www.habanero.rice.edu/Publications.html (JCUDA) : A library that can translate special Java Code (called JCUDA code) into Java- and CUDA-C code, which can then be compiled and executed on the GPU. However, the library does not seem to be publicly available.
https://www2.informatik.uni-erlangen.de/EN/research/JavaOpenMP/index.html : Java language extension for for OpenMP constructs, with a CUDA backend

Java OpenCL/CUDA binding libraries

http://jcuda.org/ : Java bindings for CUDA: Low-level bindings that are a 1:1 mapping of the original CUDA API
http://jocl.org/ : Java bindings for OpenCL: Low-level bindings that are a 1:1 mapping of the original OpenCL API
http://code.google.com/p/javacl/ : Java bindings for OpenCL: An object-oriented OpenCL library, based on auto-generated low-level bindings
http://jogamp.org/jocl/www/ : Java bindings for OpenCL: An object-oriented OpenCL library, based on auto-generated low-level bindings
http://www.lwjgl.org/ : Java bindings for OpenCL: Auto-generated low-level bindings and object-oriented convenience classes