Rajan's View

Clawing Now to Speed Up Later

2026-02-22T12:03:00.009+05:30

OpenClaw

OpenClaw has captured people's imagination in a big way in the last few weeks. The ability to turn an idea into something concrete like an app has always been aspiration for a business user or a technology developer.

OpenClaw simply removes the barriers for anyone to be able to build an app or an AI agent.

Taking it for a spin

My last post was a geeky take on using Gemini CLI as an Investment Advisor. That was perhaps best appreciated by a developer and not a non-tech person. I am taking the same use case from financial industry and turning it into a tool that anyone could use with OpenClaw.

OpenClaw The Builder

I built 2 skills for stock trading. The first one is an investment council which studies the market and serves as a stock screener. The second is a detailed stock analyzer.

Investment Council

This is a skill that contains a series of python scripts to perform the above capabilities. In total, it is about 15,000 lines of python code, all written by AI.

Key Features

Multi-Source Data Integration: Pulls market data, fundamentals, and sentiment from Yahoo Finance, Composite ETFs, SearXNG Premium, and the VITALS Framework.
9 Diverse Investment Strategies: Utilizes a broad spectrum of quantitative and qualitative models, including:
- Magic Formula, GARP, Value, Quality, and Momentum.
- Buffett Moat, Low Volatility, Dogs of the Dow, and QVM Combo.
Ensemble Scoring Engine: Combines strategies using a "Regime-Based Weight Adjustment" that shifts focus depending on whether the market is in a risk-on or risk-off state.
Sentiment & Regime Detection: Features a specialized VITALS Framework to detect market regimes and a Sentiment Adjuster that weights news based on source credibility (Tier 1 vs. Tier 3).
Specialized Recommendation Engines:
- Thematic: Focuses on sectors like AI, Healthcare, and Energy Transition.
- Risk-Adjusted: Tailors portfolios to Conservative, Moderate, or Aggressive profiles.
- Diversified: Ensures sector-balanced portfolios and correlation awareness.
Automated Reporting & Delivery: Generates 10-page PDF reports and interactive charts, delivered via Telegram or Web UI.

Key Benefits

Explainability-First Logic: Every investment recommendation is traceable back to its source data and specific decision rationale, reducing "black box" AI uncertainty.
Dynamic Market Adaptation: The tool doesn't just pick stocks; it adjusts its strategy weights (e.g., boosting Low Volatility or Quality during downturns) to match the current market regime.
Reduced Data Bias: By using Provenance Tracking and a three-tier fallback chain for data ingestion, the tool ensures high data reliability and lineage.
Zero-Cost Infrastructure: Designed to operate using Zero API Keys by leveraging free/scraped data sources and web searches with graceful degradation.
Comprehensive Risk Management: Features built-in "Sentiment Integration" to act as a contrarian or trend-following filter, helping to avoid "value traps" or overhyped assets.
Retail-Friendly Output: Translates complex financial metrics into plain English explanations and visual charts, making professional-grade analysis accessible.

Stock Analysis

This is a skill that contains a series of python scripts to perform the above capabilities. In total, it is about 6,000 lines of python code, all written by AI.

📊 Comprehensive Multi-Dimensional Analysis

Feature: Integration of Technical, Fundamental, Risk, and Valuation modules (including DCF and peer comparables).
Benefit: Provides a 360-degree view of a stock, ensuring investment decisions aren't based on a single metric but on a holistic data set.

🔍 Deep Research Engine

Feature: Automated fetching from 6 high-value sources (Wikipedia, Analyst Consensus, Ownership Data, SWOT, Porter’s Five Forces, and Recent News).
Benefit: Saves hours of manual labor by synthesizing qualitative business context alongside quantitative financial data.

🎨 Advanced Visualization Suite

Feature: Specialized modules for visual reporting, such as ROIC gauges, DCF sensitivity heatmaps, and peer radar charts.
Benefit: Converts complex numerical data into intuitive, scannable graphics, making it easier to spot outliers and trends at a glance.

🛠️ Production-Grade Reliability

Feature: Graceful degradation, deterministic caching, and "No Auto-Fix" policies.
Benefit: Ensures the system remains stable and predictable. If a specific data source is down, the system generates a partial report rather than crashing, while caching prevents redundant API calls and speed throttles.

📄 Retail-Friendly Report Generation

Feature: Automated PDF generation with "Plain English" explanations and visual traffic lights (Buy/Hold/Sell).
Benefit: Bridges the gap between professional-grade data and retail investor accessibility, making sophisticated institutional analysis easy to interpret.

⚙️ Modular & Scalable Architecture

Feature: 16+ specialized Python scripts orchestrated via a CLI (Command Line Interface).
Benefit: The "plug-and-play" nature allows developers to easily update or swap individual modules (like changing a pricing API) without breaking the entire ecosystem.

Tips for Builders

Both of the above were built over a few hours by instructing and testing the skills over a Telegram chat. It didn't matter if I had some spare time at home or stand in line on a bus stand, I could continue building and testing these skills over Telegram.

Here are a few high level tips:

Always plan for a feature before asking the agent to build it.
As part of planning process, ask the agent to research from the web and challenge, extend and solify the features.
Make the agent save the features to a ROADMAP.md file.
Then, break down the implementation into multiple phases and save it into a PLAN.md.
Prod the agent to implement 1 phase at a time and keep reviewing the output (both the agent and yourself).
Ensure that non-functional requirements such as handling errors, graceful degradation, runtime parameters, cleaning up are part of the skill.
Ensure that the SKILL.MD and CHANGELOG.MD capture all the implementation details iteratively. This way, if the context is full or memory is lost, the agent can always pick up from where it left off.
Customize the SOUL.md, IDENTITY.md and AGENTS.md to adjust the behaviour of the agents to your working style for both, you as the builder and the user.

OpenClaw The Worker

Here is a short demo of both the skills in action.

The current reports from the investment council and stock analysis skills are attached below.

Investment Council Report (with dynamic market regime detection and weighted and ensemble strategies)
Top 5 Stock Recommendations (Buy/Hold/Sell)

OpenClaw The Assistant

Now that these skills are ready, they have been setup to run on a weekly basis to detect the market regime and send the recommended stock along with the actions - Buy / Hold / Sell.

What works well

The initial impressions are:

The ability to interact with OpenClaw over existing messaging apps like Telegram, Whatsapp are perhaps the biggest draw. It immediately lowers the cognitive barrier for anyone. This also shows how the "simple chat" interface is getting more powerful as compared to custom designed UI for a human-computer-interaction. For all the OCB's (obsessive compulsive builders) out there, this may be a boon.
The "developer loop" enabled by the Pi coding agent within OpenClaw self reflects and is able to complete complex tasks or create apps with minimal supervision. This enables a user to "not spend time" in front of the screen. It feels a bit unnatural at first to "lob something" over to AI and expect it to be done without supervision. I suppose we all need to let go of the "control freak" in all of us.
Using the simple cron mechanism to get OpenClaw to proactively or regularly perform tasks is another aspect of user experience that feels refreshing. In a world where we are sometimes starved of focus, if all of us can have a personalized assistant, that's probably a welcome thing.

How to apply & scale it in Enterprises

I am going to lean into the financial industry again. Some of the teams in the financial industry like Quantitative Analysts, Data Analysts, Risk Managers have always wanted the ability to do data wrangling as well running calculations for various purposes like - Value at Risk, Sensitivities, Scenario Planning, Simulations etc. An enterprise version of a tool like OpenClaw with all the guardrails, data provenance, governance, security and traceability could help them tremendously.

A path forward could be along the lines of:

Identify use cases, where we can do a "hard" shift-left, meaning empowering business users to create and execute their workflows and agents.
This would of course necessiate change management to redefine the business workflows.
Either OpenClaw or it's many derivatives such as PicoClaw, NanoBot etc. could be adapted to comply with the entrprise standards and safeguards This can be started in parallel to 1 and 2. There are already many approaches to hardening Openclaw.
Integrate these tools into messaging apps such as Google Chat, Teams, Slack etc. for widespread adoption.
Initial pilots can be done in a "sandbox" envioronment before scaling it further.

Another potential use case could be for SMB’s, where the business owners could create custom workflows.

Gemini CLI - Your Personal AI Investment Advisor

2025-07-16T18:44:00.022+05:30

Unlocking Financial Insights with Gemini CLI and MCP Servers

The world of finance is complex, volatile, and often intimidating. From deciphering market trends to evaluating individual stocks, making informed investment decisions can feel like a full-time job. What if you could have a tireless, intelligent assistant by your side, sifting through data, analyzing performance, and even offering strategic insights?

Good news: Google recently launched the Gemini CLI open source agent. There is a generous usage tier available for free - "60 model requests per minute and 1,000 requests per day at no charge". With the power of Google's Gemini CLI and a custom setup of Model Context Protocol (MCP) servers, that future is not only possible, it's within your reach right now. Forget expensive subscriptions or black-box algorithms – we're talking about building your own sophisticated AI investment advisor.

The Brains Behind the Brawn: Gemini CLI Meets MCP

At the heart of this AI investment advisor lies the Gemini Command Line Interface (CLI), a powerful tool that allows you to interact directly with Google's Gemini models. But Gemini, while incredibly capable, is a generalist. To turn it into a financial wizard, we need to equip it with specialized knowledge and tools. This is where MCP (Model Context Protocol) servers come into play.

Think of MCP servers as specialized "plugins" or "skills" for your AI. Each server is designed to perform a specific task or access a particular type of information. By orchestrating these servers, we can empower Gemini to perform incredibly complex, multi-faceted financial analysis. In most cases, these MCP servers act as a scaffolding over API's or Databases.

My Setup: A Glimpse Under the Hood

I've been experimenting with this exact setup, running the Gemini CLI within WSL (Windows Subsystem for Linux) on my Windows machine. This provides a robust Linux environment while still leveraging the convenience of Windows. Here's a peek at the MCP servers I'm currently running, transforming my Gemini CLI into a useful financial analyst. Install the below MCP server and configure them in the settings.json for the Gemini CLI.

"mcpServers": {
  "context7": {
    "command": "npx",
    "args": ["-y", "@upstash/context7-mcp"]
  },
  "sequential-thinking": {
    "command": "npx",
    "args": ["-y", "@modelcontextprotocol/server-sequential-thinking"]
  },
  "basic-memory": {
    "command": "uvx",
    "args": ["basic-memory", "mcp"]
  },
  "stock-analyzer": {
    "command": "uv",
    "args": [
      "--directory",
      "<absolute path>/mcp-trader",
      "run",
      "mcp-trader"
    ],
    "env": {
      "TIINGO_API_KEY": "YOUR_TIINGO_API_KEY_HERE" 
    }
  },
  "financial-hub": {
    "command": "uv",
    "args": [
      "--directory",
      "<absolute path>/mcp-finnhub",
      "run",
      "fastmcp",
      "run",
      "server.py"
    ]
  },
  "financial-modelling-prep": {
    "command": "<absolute path>/fmp-mcp-server/.venv/bin/python",
    "args": ["<absolute path>/fmp-mcp-server/src/server.py"]
  },
  "quickchart": {
    "command": "npx",
    "args": ["-y", "@gongrzhe/quickchart-mcp-server"]
  },
  "code-runner": {
    "command": "npx",
    "args": ["-y", "mcp-server-code-runner@latest"]
  },
  "powerpoint": {
    "command": "uvx",
    "args": ["--from", "office-powerpoint-mcp-server", "ppt_mcp_server"],
    "env": {}
  },
  "markdown2pdf": {
    "command": "node",
    "args": ["<absolute path>/markdown2pdf-mcp/build/index.js"],
    "env": {
      "M2P_OUTPUT_DIR": "<absolute path>/workspace/stocks"
    }
  }
}

Let's break down what each of these powerful MCP servers brings to the table:

sequential-thinking: This is crucial for enhancing Gemini's ability to maintain context over longer conversations and engage in multi-step logical reasoning – essential for complex financial inquiries.
basic-memory: Allows Gemini to remember past interactions and information, building a more personalized and consistent advisory experience.
stock-analyzer (powered by Tiingo): This is where the real-time stock data comes in. It enables Gemini to fetch up-to-the-minute stock prices, historical data, and other critical metrics.
financial-hub (likely Finnhub integration) & financial-modelling-prep: These MCP servers provide access to a vast array of fundamental financial data, including company financials, earnings reports, analyst ratings, and more. This is key for in-depth company analysis.
quickchart: Imagine asking Gemini to "show me a chart of Apple's stock performance over the last year" and instantly getting a visual representation. This MCP server makes that a reality.
code-runner: For more advanced statistical analysis or custom data processing, Gemini can leverage this MCP server to execute code, providing unparalleled flexibility.
powerpoint & markdown2pdf: These MCP servers are nifty tools for presentation and reporting. Gemini can potentially generate a quick summary report of your portfolio or a detailed analysis of a target stock, ready for sharing or printing.

context7: This fetches the latest documentation of the software libraries. It ensures that any code written by Gemini is accurate and minimzes errors.
`usolver MCP server`: Apply various solvers such as Google's OR tools and Microsoft's Z3 theorem prover. These help add a "logic" layer on top of the stock recommendations to ensure that they meet the portfolio objectives.
`vectorbt`: This popular framework can be used to backtest for efficiently backtesting trading strategies and performing quantitative analysis.

What Can Your AI Investment Advisor Do?

With this powerful combination, your Gemini-powered AI investgment advisor also needs guidelines on how to leverage these tools. Here is a sample GEMINI.md template.

Project Guidelines
This document contains critical information about working with this project. 
Follow these guidelines precisely.

You have the following tools:
sequential-thinking:
A detailed tool for dynamic and reflective problem-solving through thoughts.
This tool helps analyze problems through a flexible thinking process that can adapt 
and evolve.
Each thought can build on, question, or revise previous insights as understanding 
deepens.
When the user asks an open ended or vague question, use the sequential thinking tool 
always.

basic-memory:
Use this tool to search, store, retrieve, update notes.
Whenever a chart is downloaded as an image, always update the relevant note with 
the link to the image.

stock-analyzer:
Analyse a stock from multiple perspectives.

financial-hub:
Use this tool to get all the stock fundamentals, news and market trends.

financial-modelling-prep:
Get advanced market and stock details.

quickchart:
Use this tool to generate charts. 
Once the chart is created, run the wget command in shell to download the chart as 
an image.

powerpoint:
Use this tool to create powerpoint slides. If no template is provided, 
use the professional or minimal templates.
Ensure that all the powerpoints created are formatted with the optimal font and 
font size. The presentation type is always business.

code-runner:
When ad-hoc calculations are required, use python to write the calculations execute 
them using this tool.
Use the below Code Quality guidelines -
Type hints required for all code
Public APIs must have docstrings
Functions must be focused and small
Follow existing patterns exactly
Line length: 88 chars maximum
Testing Requirements

Code Style should be -
PEP 8 naming (snake_case for functions/variables)
Class names in PascalCase
Constants in UPPER_SNAKE_CASE
Document with docstrings
Use f-strings for formatting
For commits fixing bugs or adding features based on user reports add:

markdown2pdf:
Convert the notes to pdf using this tool.

context7:
Use this tool to get the latest documentation on a python library.

Getting started

Follow the installation instructions from github. Once you have configured the settings.json and GEMINI.md, you can run the CLI. You will notice the message that Gemini is now ready to go with the instructions provided and the 10 MCP servers.

Now, use Ctrl+T to view all the MCP server, their status and the tools available. For sake of brevity a few of the MCP servers are show below.

Provide real-time stock quotes and historical data.
Let's use Google as a test case for all our analysis. You can type in a prompt - "
get basic fundamentals of GOOG".

After confirming the tool execution, the results are displayed in the Gemini CLI. One can see the "raw data" as well as the summary from Gemini.

Analyze key performance indicators.
Next, type in the prompt - "analyze the technical setup of GOOG". Again, a different tool asks for permission.

Once, the tool runs, the technical analysis results are shown. Not bad.

Summarize news and sentiment around specific stocks or sectors.

Now, let's ask for "get the latest market news on tech stocks".

Once the tool confirmation is provided, the news is retrieved from the MCP Servers/API's.

In addition, Gemini also recommends and alternate stock for analysis. We can accept Gemini's suggestion to compare Google with another technology stock such as Microsoft (MSFT).

Again, the same fundamental and technical analysis is done on MSFT.

As a bonus, we can also get the broader market moves for the day.

Generate charts and visualizations of financial data.
We can now try to visualize the data a bit better by asking - "what time series data do we have available for GOOG and MSFT".

Based on the data already retrieved, charts can also now be created.

Even offer preliminary insights into potential investment strategies

Now, let's ask for the "based on the data, what is the investment recommendation between these stocks"

Gemini CLI is able to give recommendations grounded on the stock fundamentals and technical analysis already performed in earlier steps.

Going a little deeper, let's ask for - "calculate the position size for Google".

Gemini CLI asks for inputs to calculate the position size.

It uses the relevant tools to provide the "position size" recommendations.

Create reports and presentations on demand

Now that we have done all the analysis, let's create a presentation - "create a presentation with all the analysis done so far and add the charts as well". The relevant tool now proceeds to create the presentation with all the details and charts.

Three presentations for further analysis are created using Gemini CLI and the MCP servers.

Google stock recommendation - An initial presentation on recommendations for the Google stock can be accessed here.
Google and Microsoft Trading Strategy - A detailed trading strategy for Google and Microsoft is here. This contains portfolio optimization, Value at Risk analysis, HiGHS solver, Monte Carlo simulation, Entry/Exit calculations, Performance Attribution, Sensitivity Analysis and Phase-wise trading strategy.
Backtesting of the Trading Strategy - Now, that we have the trading strategy, it is prudent to "backtest" it. The final presentation about backtesting the trading strategy is here.

Roadmap

As an extension to the above setup, here are some useful additions I am currently testing.

skfolio: This is a portfolio optimization framework build on top of scikit-learn to perform model selection, validation and parameter tuning.
n8n MCP Server: Once the trading strategies are created, one can execute and monitor the strategy with n8n workflows.

The Future of Personal Finance is Agentic

This isn't about replacing human financial advisors entirely, but about democratizing access to powerful analytical tools for retail and institutional investors alike. Imagine being able to quickly validate an investment idea, get a second opinion on a stock, or simply stay better informed about your portfolio – all through a natural language interface.

The setup might seem a bit technical at first, especially for those new to command-line tools or development environments. However, the open-source nature of many MCP servers and the clear documentation available for Gemini CLI make this an incredibly rewarding journey for anyone willing to dive in.

Also, the cost to setup the Gemini CLI and the associated MCP Servers and API's is zero, so any returns you make on your investments is all profit!

Are you ready to take control of your financial future with the power of AI? The tools are available, the potential is immense, and the learning curve is an investment in itself. Start exploring the Gemini CLI and the exciting world of MCP servers today – your personal AI investment advisor awaits.

Enjoy learning the world of agents and happy investing!

Disclaimer: This content does NOT constitute financial advice, investment recommendations, or a solicitation to buy or sell any securities or financial products. Key Considerations and Risks:

Not Personalized Financial Advice: The examples and discussions herein do not consider your individual financial situation, investment objectives, risk tolerance, or specific needs. Investing decisions should always be tailored to your unique circumstances.
Investment Risks: All investments involve inherent risks, including the potential for loss of principal. The value of investments can fluctuate significantly due to market volatility, economic conditions, company-specific factors, and other unforeseen events. There is no guarantee that any investment strategy, even one informed by AI, will be profitable.
Past Performance Not Indicative of Future Results: Any historical data, analysis, or simulated performance presented or generated by the described AI setup is for illustrative purposes only and should not be considered a reliable indicator of future investment results.
AI Limitations and Errors: While powerful, AI models, including Gemini and the MCP servers, are based on algorithms and data. They can make errors, produce incomplete or biased information, or misinterpret complex financial nuances. The data sources used by the MCP servers (e.g., Tiingo, Finnhub) may also have their own limitations or delays. Always exercise your own independent judgment and critical thinking when using AI-generated insights.
No Guarantee of Accuracy or Completeness: We strive for accuracy in our content, but we make no warranties or representations, express or implied, regarding the completeness, accuracy, reliability, or suitability of any information presented. Financial markets are dynamic, and information can become outdated rapidly.
Human Oversight is Crucial: The AI setup described is a tool to assist with analysis, not a replacement for human expertise, due diligence, or professional financial advice. You are solely responsible for all investment decisions you make.
Tax and Legal Implications: Investment decisions can have tax and legal implications. You should consult with qualified tax and legal professionals for advice relevant to your specific situation.
Third-Party Services and APIs: The MCP servers leverage various third-party services and APIs (e.g., Tiingo). Your use of these services may be subject to their respective terms of service, privacy policies, and potential costs. We are not responsible for the performance or policies of these third-party providers.
Your Responsibility: By reading this blog post and considering the concepts presented, you acknowledge and agree that you are solely responsible for any investment decisions you make. You assume all risks associated with your financial actions and will not hold the author, blog, or any associated entities liable for any losses or damages incurred.
Always consult with a qualified and licensed financial advisor, investment professional, or other relevant expert before making any investment decisions.

Vibe coding your way to Agentic AI

2025-04-19T16:51:00.017+05:30

If someone told me that “vibe coding” would be a thing someday, I would have laughed it away. But, here we are. Agentic AI is already starting to change the way we interact with AI. One can think of vibe coding as an “agentic” way of developing software. Let’s see how one can “vibe code” an Agentic AI application.

Tools of the Trade

No pun intended, here are the developer tools one could use.

The Cloud Editor in Google Cloud Shell. This editor is similar in look and feel to VS Code. So, the application development can now happen entirely in the cloud, without installing anything on the laptop.
Install the Roo Code extension in the Cloud Shell Editor and configure it to use Gemini 2.0 Flash. You will have to create an API Key and set it up, as shown below.
Lastly, we will use Google Agent Development Kit (ADK) framework to create an agentic application.

Once the Roo extension is installed, developer can use it in different modes such as - Architect, Code, Ask, Debug etc.

For example, in the example below, I "asked" Roo/Gemini 2.0 Flash to analyze unused imports in the code.

Beyond suggestions, I also used Roo/Gemini 2.0 to write the code and update the README.md. You can see the generated agent code at GitHub. The steps to install, test and deploy the app are in the same GitHub repository.

The Agentic AI application was built in the following way

Hand Coded by Me

Overall agent skeleton code
call_code_agent
call_web_search
Dockerfile
Flask App

Vibe Coded by AI

get_exchange_rate
get_historical_exchange rate
get_exchange_rate_trend
setenv.sh
run.sh
deploy.sh

Agentic AI using Google ADK

In this example, we have created a “currency agent” application. This application contains multiple agents and tools. All these agents and tools are leveraging Google ADK. There is no MCP or A2A in this example, that is for another day.

This Agentic AI app can - get current and historical currency exchange rates. In addition, it can perform ad hoc calculations and also search the web to analyse current events.

We can use the Google ADK Web Developer UI to test the Agentic AI application. As you can see in the screenshots, all of the below steps are from a single session. If we want to do this across sessions, we will need a stateful session service. This example uses an in-memory session service provided by Google ADK framework.

1. First, a user starts by asking for the current exchange rate. Due to the free API’s being used, the base currency is always US Dollar (USD).

2. Next, the user asks about the causes of currency fluctuations. This information is retrieved from the web.

3. Now, user is following up with historical exchange rates. Current and historical exchange rates are retrieved using tools, which in turn call APIs.

4. In addition, user can now ask for predictions. This is done by the agent writing python code to arrive at the prediction as well as show visualizations (which was not asked by the user at all). This is the first example of a “agentic ai” where an application understands the context and provides more intelligent answers to the user.

5. Next, is the second example of an “agentic ai” capability, formatting and summarizing data. In this case, it is a synthesis of data that was retrieved from multiple sources -

Historical data from Currency APIs
Future predicted exchange rate
Analysis from the web

6. Here is the third example of “agentic ai” capability, perhaps the most comprehensive. User provides the high level instruction - to create a summary for USD to GBP currency pair. Agent understands the instruction and automatically performs all the steps that were done earlier for the USD to SGD currency pair, without explicit directions from the user.

7. These steps include the prediction, creation of visualization and analysis from the web.

8. Since the currency-agent application have access to the web, users can also perform adjacent analysis such as impact on stock market because of currency fluctuations.

9. And, combine the currency analysis with the stock market analysis. This is the fourth example of “agentic ai” capability, maintaining context and smartly responding to user requests.

10. Last step is to request for trading recommendations. The Gemini 2.0 Flash model is able to go through the context and provide some insights.

11. Google ADK does a great job at interpreting the multi-modal outputs provided by the various tools and agents. These outputs such as PDF and Images show up as artifacts. These artifacts can be presented to the user as well as used by the ADK framework to pass it as a context to various tools and agents for subsequent processing. In addition, the features like events, memory, session and context ensure that all user interactions are passed to the AI model.

12. The Google ADK Dev UI also enables the developer to visually check the flows and debug each step/interaction.

Design Pattern

Various design patterns will probably emerge on how we build the Agentic AI applications. Based on this example agent, an emerging design pattern could have the following attributes -

An Agentic AI application is bound by a domain context, in this example it is currency exchange rates.
The application can have multiple agents and/or tools. Typically, these tools perform tasks that are inter-related within the domain context.
The tools can be a mix of -

Ability to query structured data, typically present in a database
A code interpreter that can perform ad-hoc tasks that do not have to be “hand coded”
A search capability that is grounded on internal and/or external or unstructured data
Tools that integrate with internal / external APIs

Out of box capability for multimodal input and output
Users start using the application by mainly providing the goals and letting the agent "think" and execute the workflows.

What’s Next

One can use the existing source code of the Currency Agent as a “seed” and let Roo/Gemini 2.0 Flash “vibe code” new agents automatically. More on that, later.

Meta Prompting for Success

2024-02-17T08:50:00.022+05:30

Meta Prompt

Regardless of the Generative AI model used, sometimes the model may refuse to provide a response for a valid prompt. Often the reasons cited are - "I don't know", "I am not trained for the task", "violation of safety/responsible/ethical guardrails". This is sometimes more acutely felt when using Generative AI models for creating images. Overcoming this behavior can be frustrating and time consuming.

An approach is to ask the AI model itself to generate prompts so that the prose and nuances of prompting are aligned with how the AI model has been built. This is called meta-prompting. The theory is that the prompts created by the AI model would have better chance of working.

Let's dive in to see if and how this meta-prompting works in Gemini.

How to Create a Meta Prompt

As with all good prompts, start with a persona/situation that describes the overall context. Next, describe the task in detail. In this case, it is about creating some sample prompts/templates. Depending on the context, provide detailed steps to create the prompts. Goal describes how these prompts are going to be used next. Output is to reiterate the goal in terms of formats and structure.

A Sample Meta Prompt

I am an entrepreneur running an executive coaching organization.

Task - I want you to create 3-5 sample prompts that I can use with you to create marketing images.

Steps - To complete this task, please give me example prompts that are safe and responsible. The prompts should be reusable for various scenarios.

Goal - Prompts that will successfully create images

Output - Clear prompts that will work

Gemini Draft Response 1

5 Safe and Reusable Prompts for Your Executive Coaching Marketing Images:

1. Focus on the Impact:

Prompt: A diverse group of confident executives standing tall in a modern, sunlit office, radiating success and achievement. Include subtle coaching elements like books or diagrams in the background. (Style: Photorealistic)
Variations: Change the setting, ethnicity of the executives, or add specific coaching tools relevant to your services.

2. Highlight the Transformation:

Prompt: Two executives, before and after coaching, standing side-by-side. The "before" version appears stressed and overwhelmed, while the "after" version exudes confidence and calm. (Style: Minimalist illustration)
Variations: Show a single executive transforming, or use different visual metaphors for transformation, like climbing a mountain or emerging from a chrysalis.

3. Capture the Human Connection:

Prompt: A coach and client in a warm, inviting setting, engaged in a deep and supportive conversation. Focus on the trust and connection between them. (Style: Soft, painterly)
Variations: Change the setting, gender, or age of the coach and client. Add props like a whiteboard or coaching cards to enhance the context.

4. Showcase Expertise and Authority:

Prompt: A coach standing on stage, confidently delivering a keynote presentation to a captivated audience. Project a slide with your company logo and tagline in the background. (Style: Dynamic, high-contrast)
Variations: Change the presentation setting, focus on the coach writing a book, or feature them holding an award.

5. Evoke Emotions and Aspirations:

Prompt: A panoramic view of a cityscape at sunrise, symbolizing the potential and growth unlocked through executive coaching. Include subtle coaching metaphors like a compass or a ladder. (Style: Dreamlike, evocative)
Variations: Use different landscapes, motivational quotes, or symbolic objects that resonate with your target audience.

Bonus Tip: Include your company logo, tagline, or website URL discreetly within the image for branding purposes.

Remember, these are just starting points. Feel free to experiment and personalize the prompts further based on your specific brand identity and target audience.

I hope these prompts help you create impactful images for your executive coaching marketing!

Executing Prompts

Prompt 1

Prompt 2

Prompt 3

Prompt 4

Prompt 5

Gemini Draft Response 2

3 Sample Prompts for Marketing Images (Executive Coaching):

1. Concept-Driven:

A diverse group of executives gathered around a table in a bright, modern office, engaged in a collaborative brainstorming session. Sunlight streams through the windows, highlighting their focused expressions and positive body language. The image conveys the power of coaching in fostering teamwork and leadership growth. (Style: Photorealistic, Mood: Uplifting, Colors: Light and airy)

2. Specific Service Highlight:

A close-up shot of hands holding a notebook with a handwritten list of goals and achievements. A coach's hand rests gently on top, offering guidance and support. The image focuses on the personalized approach of your coaching and the tangible results it brings. (Style: Photorealistic, Focus: Hands and notebook, Colors: Neutral tones)

3. Problem-Solution:

A split image with one side showing a stressed executive overwhelmed by work and challenges. The other side shows the same executive relaxed and focused, exuding confidence. The image emphasizes the transformative power of coaching in overcoming obstacles and achieving success. (Style: Minimalistic, Layout: Split image, Colors: Contrasting tones)

4. Bonus Tip:

Remember to adjust the prompts based on your specific target audience and branding.
Provide additional details to the prompts for more specific results, like clothing styles, office settings, or specific goals being discussed.
Experiment with different styles and moods to find what resonates best with your audience.

By using these prompts as a starting point, you can create high-quality marketing images that effectively capture the essence of your executive coaching services and attract your ideal clients.

Executing Prompts

Prompt 1

Prompt 2

Prompt 3

Conclusion

Not only is the primary purpose of generating images met, in addition there interesting cues that are provided in the Gemini response -

There is a 100% success rate of all the prompts successfully creating the images.
Potential objectives that the images/content could help with. These are things like - impact, human connection, highlighting service, transformation and so on.
Lots of hints on how to further "style" the image in terms of illustration, setting, scenes, color, contrast and tone.
Additional tips to further customize the prompts to generate more contextual images.
The prompts generated act as a diverse set of templates that can further be fine tuned by the users to get the more desired results.
While this post showed the example from Gemini, a similar approach can be used when building enterprise apps using Gemini models in Google Cloud Vertex AI platform.

Forget prompt engineering, now start meta-prompting!

The Many Faces of Val Kilmer

2023-11-12T07:02:00.007+05:30

Tribute to The Saint

This is a tribute long overdue. To the movie and one of my favorite actors - Val Kilmer. He’s had many successful movies, but perhaps a favorite of mine is “The Saint” from 1997. And, the luminescent Elizabeth Shue provides an apt foil for Val Kilmer. In keeping with the namesake in the movie, the original soundtrack for the movie by Graeme Revell is medieval and atmospheric.

In a recent interview, the “Past Lives” actress Greta Lee describes Val Kilmer in this movie being one of her early inspirations. While the movie “The Saint” is not seen as a critical or commercial success, if it inspired a brilliant actress, who can complain.

In the movie, Val disguises himself after Catholic saints. As a "Saint Head", something I always wanted to understand a bit more, a background of all the Catholic saints that crop up in the movie.

Simon Magus

Val Kilmer as the character “Simon, The Magician”

Simon Magus is a controversial figure in Christian history. He is mentioned in the Acts of the Apostles as a Samaritan magician who converted to Christianity but later tried to buy the power of the Holy Spirit from the apostles Peter and John. This incident is the origin of the term "simony," which refers to the buying or selling of ecclesiastical offices or sacred things.

John Rossi

Val Kilmer as the character “Ivanovanovitch”

Saint John Rossi (1622-1685) was an Italian priest and canon lawyer who was martyred for his defense of the poor and oppressed. He was born in Piacenza, Italy, to a wealthy family. After studying law at the University of Bologna, he was ordained a priest in 1649. He then devoted his life to serving the poor and disadvantaged, and to defending the rights of the oppressed.

Martin de Porres

Val Kilmer as the character “Martin de Porres”

Saint Martin de Porres (1579-1639) was a Peruvian lay brother of the Dominican Order who is known for his charitable work and his devotion to the Eucharist and constant prayer. He is the patron saint of mixed-race people, barbers, innkeepers, public health workers, and all those seeking racial harmony and animals.

Bruno Hautenfaust

Val Kilmer as the character “Bruno Hautenfaust”

Saint Bruno Hautenfaust, also known as Saint Bruno of Cologne, was a 11th-century German canon and theologian who founded the Carthusian Order, a monastic order dedicated to a life of solitude and contemplation. He was born into a wealthy family in Cologne, Germany, around 1030. After studying at the cathedral school in Reims, France, he became a canon of the cathedral. He was a brilliant scholar and teacher, and he quickly rose through the ranks of the clergy.

Thomas More

Val Kilmer as the character “Thomas More”

Saint Thomas More (7 February 1478 – 6 July 1535) was an English lawyer, author, statesman, and noted Renaissance humanist. He also served King Henry VIII as Lord Chancellor of England from October 1529 to May 1532. He is best known for his book Utopia, published in 1516, which describes the political system of an imaginary island state.

Charles Borroneo

Charles Borromeo (1538-1584) was an Italian cardinal and archbishop who was one of the most important figures of the Counter-Reformation. He was born into a wealthy and aristocratic family in Milan, and was groomed for a career in government or the Church. However, he was drawn to a life of prayer and service, and decided to become a priest.

Peter Damian

Saint Peter Damian (1007-1072) was an Italian cardinal, bishop, and Doctor of the Church. He was a leading figure in the Gregorian Reform movement,

Luigi Guanella

Luigi Guanella was an Italian Catholic priest and the founder of the Servants of Charity, a religious order dedicated to caring for the poor and the sick. He was born in 1842 in Fraciscio, Italy, and ordained a priest in 1866. After his ordination, Guanella served as a parish priest in several parishes in northern Italy. He was deeply moved by the plight of the poor and the sick, and he began to establish schools, orphanages, and hospitals to care for them. In 1890, he founded the Servants of Charity, a religious order dedicated to this work.

Nicholas Owen

Saint Nicholas Owen (c. 1562 – 1/2 March 1606) was an English Jesuit lay brother who was the principal builder of priest holes during the reigns of Queen Elizabeth I and James I of England. Owen built many priest holes in the buildings of English Catholics from 1588 until his final arrest in 1606, when he was tortured to death by prison authorities in the Tower of London. Owen is honoured as a martyr by the Catholic Church and was canonized by Pope Paul VI in 1970.

Vincent Ferrer

Val Kilmer as the character “Vincent Ferrer”

Saint Vincent Ferrer (1350-1419) was a Dominican friar and preacher who is known for his missionary work and his role in helping to end the Great Western Schism. He was born in Valencia, Spain, and entered the Dominicans at the age of 19. He was a gifted student and became a professor of theology at Valencia. In 1394, he was appointed confessor and theologian to the antipope Benedict XIII at Avignon. However, he resigned from this position after five years in order to undertake missionary work. In 1394, Ferrer was preaching in Valencia, Spain, when he was betrayed by a group of his own followers. These followers were disaffected with Ferrer's support of the Avignon papacy and his attacks on heresy.

August Christopher

Val Kilmer as the character “August Christopher”

Saint Christopher is a venerated saint in the Catholic Church and Eastern Orthodoxy. He is the patron saint of travelers, motorists, ferrymen, and all who work in transportation. According to legend, Christopher was originally named Reprobus and was a Canaanite of immense size and strength. He vowed to serve only the greatest king, and so he set out to find the most powerful ruler on Earth. He served many kings, but none of them satisfied his desire for the greatest king.

And, here is a review, candid and in support of “The Saint”.

Notes

The photos are sourced from the internet
Google Gemini helped provide the quick summary of each of the catholic saints

Update

Val Kilmer passed away on 1st April 2025. May the saint rest in peace.

Now you can see me

2023-10-29T18:17:00.019+05:30

Overview

As customers, we shop with various brands (i.e. labels, retailers and ecommerce marketplaces). Once, we grow comfortable with certain bands, we tend to interact more frequently with them. In today's channel-less world, this interaction takes various forms - visiting stores, following the social media updates for inspiration, searching/browsing online in the apps and last but not the least talking to customer service.

However, despite all the "breadcrumbs" we leave behind with the brand, the experience often leaves us wanting more. On the other hand, brands are increasingly wanting to know and understand about their customers. Various attempts are made at "personalizing" the experience. This is mainly along three pillars below -

Elevating service in physical touchpoints such as stores
Loyalty programs to incentivize repeat purchases
Seeking out patterns in data from products, services, channels and customers

Out of the the above 3, #1 and #2 can be expensive, prohibitively so sometimes. Often the more reliable way is #3 - to get the insights from data. Historically, this has been easier said than done as it involves investment in creating large data science and analytics teams who are constantly monitoring customer behavior. The more detailed and granular the monitoring, it helps brands to take proactive and corrective actions.

Brands who are unable to fully execute the 3 pillars have a gap. Let's call this the "satisfaction gap". As you can see in the model below, these gaps exist at multiple levels. The gaps start with understanding the intent and need of customer to executing the relevant experience for the same customer.

Credits: Howard Tiersky @ CMSWire

Bridging the Satisfaction Gap

If we were to go down pillar #3, in the last couple of decades - technologies, frameworks, design and architecture patterns and engineering practices have evolved to a great extent.

Databases have evolved from having rigid data structure to flexible schemas
Cloud has been a huge enabler to store and process data at scale
Business Intelligence tools have evolved from being able to just "view" dashboards to "actioning" insights
Multiple tools have emerged to manage data lineage and governance
Data engineering has finally borrowed the best practices from software engineering around topics like - infrastructure as code, test automation and data meshes.

Art of the Possible, Today

This is where a combination of analytics, AI and Generative AI can help understand customers better. Before we get into the technology, let's look at the constraints that brands normally have. These generally tend to be -

Limited data, especially when it comes to the brick-and-mortar stores
Paucity of talent that is required to build and maintain data platforms

Let's face it, these are not challenges that are easy to solve. So, what can a brand do? Here's a quick guide -

Control the Controllable. Look at the above constraints and work with them for progressing further.
Small data, big data are just labels. More important question is - Is it "good enough" data? Is it available at the right time to the right users? Are decisions taken on "gut feeling" or based on data? We get more value on ensuring the business processes and IT systems capture and maintain the existing data better.
Build insights over the existing data, some of these can be deterministic (based on facts), some of these can be probabilistic (derived from facts).
Brands often obsess over insight that tell "WHAT" happened in the past. No doubt, this is required. Especially to measure and adjust business performance and operations.
Brands also need to obsess over "WHY" something happened and "HOW" do we improve it in future. This comes from insights that are actionable.

All these might sound very high level. So, let's look at some concrete examples of how actionable insights can be generated even from the most basic data that is available today.

Capturing Intent Signals

Brands already capture customer intent in various forms. The diagram below illustrates the typical customer journey. As the customer progresses down the journey, the strength of the signal gets stronger.

Credits: Rajan Manickavasagam

All of the above steps in the customer journey are already captured by a brand. All it needs is stitching them together in a data platform and extracting the insights. The most crucial insights would be customer attributes and segments.

Understanding the Intent

Now comes the interesting part. Broadly, we could divide the customer attributes into two types - deterministic and probabilistic. The basis for creating these attributes all come from a few forms of economics namely - microeconomics, behavioral economics and consumer theory. Of these, perhaps the most fascinating is behavioral economics which is a combination of economics and psychology.

The deterministic part of understanding intent could be captured by following attributes. These are calculated base on business rules, statistics and plain mathematics.

Frequency of purchase
Recency of purchase
Basket Mix
Channel Affinity
Loyalty
Transaction Value
Time between purchases
Locations of purchase

The probabilistic part of understanding intent is captured by the following attributes. And, this is where we can start leveraging Generative AI.

Demographics
Lifestyle
Behaviors and Tendencies
Habits
Interests

When we combine the deterministic and probabilistic attributes, we can create almost unlimited customer segments down to that elusive "Segment of 1". This is where each customer is identifiable individually. Something like how each viewer gets a very personalized homepage when they open Netflix. Here is a wonderfully concise article on the personalization journey at Netflix by Gibson Biddle, former VP/CPO of Netflix.

Coming back to retail brands, here is what a typical retail store transaction looks like. Most of the below data comes from the point of sale systems augmented by other sources like merchandising system as well external sources like weather.

Transaction
{ 
 time: "28-10-2023",
 weather: "30 celsius, humidity 60%",
 store: "flagship store, orchard road, singapore",
 amount: 1000,
 currency: "singapore dollar",
 loyalty_points_earned: 100,
 loyalty_points_redeemed: 300,
 items : {
   item_1: {
    product_name: “t-shirt long sleeve (black color)”,
    product_category: “womens - casual - t-shirt”,
    quantity: 3,
    price: 300 
   },
   item_2: {
    product_name: “womens mid rise jeans (medium size)”,
    product_category: “womens - casual - bottoms”,
    quantity: 2,
    price: 600 
   },
   item_3: {
    product_name: “cashmere scarves (black and white)”,
    product_category: “womens - accessories”,
    quantity: 1,
    price: 100 
   },
 }
}

Let's use the PaLM model from Google Cloud Vertex AI platform. We pass the above transaction to the text-bison large language model and prompt it with the below instruction.

Segments
Season - Unknown, Spring, Summer, Autumn, Winter
Loyalty - Unknown, Casual, Committed
Purpose of Shopping - Unknown, Travel, Weekend, Office, Mixed
Demographics - Male, Female, Transgender, Gender neutral, 
Non-binary, Agender, Pangender, Genderqueer, Two-spirit, 
Third gender, All, None, Unknown
Age - 20-30, 30-40, 40-50, 50-60, Unknown

For the above transaction, provide possible segments and values in 
the following json format -
{segment name: {[multiple segment values], [explanation of segment]}}

The response from the model is below. Of course, more prompt engineering can be done to further customize the response.

{
  "Season": ["Unknown", "Summer"],
  "Loyalty": ["Committed"],
  "Purpose of Shopping": ["Casual", "Office"],
  "Demographics": ["Female"],
  "Age": ["20-30"],
  "Segments": {
    "Season": "The transaction took place in October, 
    which is the summer season in Singapore. 
    However, the season cannot be determined 
    conclusively from the transaction data alone.",
    "Loyalty": "The customer redeemed 300 loyalty points, 
    which suggests that 
    they are a committed customer.",
    "Purpose of Shopping": "The customer purchased a mix of 
    casual and workwear items. The purchase of three 
    long-sleeved t-shirts suggests that the customer may be shopping 
    for the cooler autumn weather. Additionally, the purchase 
    of two pairs of mid-rise jeans and one cashmere scarf 
    suggests that the customer may also be shopping for workwear.",
    "Demographics": "The customer purchased women's clothing, 
    suggesting that they are female.",
    "Age": "The customer spent SGD 1000, which is a 
    relatively high amount for clothing. This suggests that 
    they are likely to be in the 20-30 age group, when people are 
    typically more likely to have disposable income."
  }
}

Let's look at another sample transaction, this time from the online channel.

Transaction
{ 
 time: "29-10-2023",
 weather: "autumn",
 channel: "mobile app",
 amount: 500,
 currency: "singapore dollar",
 loyalty_points_earned: 0,
 loyalty_points_redeemed: 0,
 items : {
   item_1: {
    product_name: “formal shirt short sleeve (white color)”,
    product_category: “mens - formal - shirt”,
    quantity: 1,
    price: 100 
   },
   item_2: {
    product_name: “mens linen trouser - black (small size)”,
    product_category: “mens - formal - bottoms”,
    quantity: 2,
    price: 150 
   },
   item_3: {
    product_name: “pin stripe tie - blue and red”,
    product_category: “mens - accessories”,
    quantity: 1,
    price: 100 
   },
 }
}

We pass the above online transaction to the text-bison large language model and prompt it with the same instruction as before.

Segments
Season - Unknown, Spring, Summer, Autumn, Winter
Loyalty - Unknown, Casual, Committed
Purpose of Shopping - Unknown, Travel, Weekend, Office, Mixed
Demographics - Male, Female, Transgender, Gender neutral, 
Non-binary, Agender, Pangender, Genderqueer, Two-spirit, 
Third gender, All, None, Unknown
For the above transaction, provide possible segments and values 
in the following json format -
{segment name: {[multiple segment values], [explanation of segment]}}

The response from the model is below. This time, the response is tailored to the transaction as well.

{
  "Season": ["Unknown", "Summer", "Autumn"],
  "Loyalty": ["Unknown"],
  "Purpose of Shopping": ["Office", "Rainy Day"],
  "Demographics": ["Male"],
  "Age": ["20-30", "30-40"],
  "Segments": {
    "Season": "The transaction occurred in October, 
	which is autumn in Singapore.",
    "Loyalty": "The customer did not redeem or earn 
    any loyalty points, which suggests that they are 
    either a new customer or a customer who does not 
    shop at the store often.",
    "Purpose of Shopping": "The customer purchased a 
    formal shirt, trousers, and tie, which suggests 
    that they are shopping for work or a special occasion. 
    The customer may also be purchasing these items for a 
    rainy day, as they are all made of materials that 
    are water-resistant or quick-drying.",
    "Demographics": "The customer's age is unknown, but 
    the average age of customers who purchase formal wear 
    is typically between 20 and 30 years old.",
    "Age": "The customer purchased items that are 
    typically popular with young adults and adults in 
    their early 30s, such as a formal shirt and linen 
    trousers. This suggests that they are in the 20-30 
    or 30-40 age range."
  }
}

Imagine if we could run this on all store and online transactions. Even with the most basic data available with a brand, we are able to derive customer insights into - demographic, lifestyle, interests and habits. And, all of this is done in a privacy friendly way without the need for personal information like - name, phone and email.

Now, brands can get more insight into not just what is selling well or not well, but also get a better understanding of who is buying the products and why. The prompts could be further optimized by using other prompt engineering strategies like - few shot, chain of thought, tree of through etc.

The prompt could be customized to the type of retail brand (convenience, grocery, department store, fashion, luxury, marketplace etc.) as well as the attributes that the brand is interested in. Of course, these attributes would have to be calculated/re-calculated on a weekly or fortnightly basis to keep capturing the latest customer "zeitgiest'.

Creating Experience from the Intent

Once the brand has derived the customer insights from the various data points like - purchase transactions, customer feedback, product reviews, this can be leveraged across the organization. The potential applications of the customer attributes and segments can be -

Optimize assortment by store / location
Additional signal for refining pricing and promotions (especially quantum and timing)
Personalize the digital experience (homepage, search ranking and product recommendations)
Decide format and location of stores
Contextual digital and store marketing campaigns

Summary

In the next post, we will see what are some of the best practices to close the "satisfaction gap".

Dear brands, now that you can see me clearer, you can serve me better. Mind the Gap!

Singapore Inspired Chilli Chicken

2023-09-30T17:02:00.002+05:30

Singapore Inspired Chilli Chicken

If you are living in Singapore, inevitably one gets to try the various hawker centers. The trademark style of many of the stalls is various cuisines, all cooked fresh. And, there is a hint of different cultures like - Chinese, Malay, Indian, Indonesian, Thai and Vietnamese in the food and the cooking styles.

Inspired by the cooking style, this is a simple recipe to make chilli chicken. It can be had as a starter/appetizer, finger food for drinks or even mixed to plain soba / rice noodles as a garnish.

The recipe is a bit of a shortcut, utilizing the ready made sauces and pastes that are available from the supermarkets.

Contributed by Rajan Manickavasagam.

Published 30. September 2023

Chilli Chicken garnished with Coriander

Ingredients

2 Chicken breast pieces
1 cup of Ris Red Chilli Sauce
1 cup of Riverina Garlic, Chilli, Ginger paste
1 bunch of chopped Coriander leaves
3 tablespoons of vegetable oil
Salt (to taste)

Instructions

Dice the chicken breasts into cubes.
Rub in 1 tablespoon of salt to diced chicken pieces.
Add in the garlic chilli ginger paste and red chilli sauce to the chicken. Toss the chicken pieces around and marinate them in the fridge for 10 minutes.
In a flat griddle pan, heat the vegetable oil on high heat.
As the oil begins to sizzle, add the marinated chicken to the flat pan. Sear all sides of the chicken until they are dark brown.
Reduce the heat to low / medium and sauté the chicken until done.

Finally, transfer the chicken pieces to a serving dish. Add the freshly chopped coriander and serve.

Happy eating.

Further details

Serves 4-6 people.

Preparation time is approximately 15 min.

Cooking time is approximately 15 min.

A vegetarian version can be made with Tofu or Indian cottage cheese or medley of vegetables like cauliflower, beans and carrots.

Light and Dreams

2023-08-27T17:45:00.005+05:30

Lights and Dreams

I drink in your sight,

Along with bursts of morning light.

As the shadows disappear,

Our dreams appear.

A short poem inspired by the haiku format.

Book Review - The Price of Being Fair

2023-07-31T19:13:00.007+05:30

Overview

As the FairPrice group in Singapore completes 50 years, "The Price of Being Fair" book is a timely story of it's birth and evolution. The book has been authored by Sue-Ann Chia and Shing Huei Peh. While being true to history, the book has nuggets of "behind the scenes" information, given the time the authors have spent with various leaders at FairPrice. The book is already a best seller according to Kinokuniya book store. At under 200 pages, it makes for a breezy read.

The Usual Suspects

Having worked in retail industry for over two decades, the travails of NTUC FairPrice seem familiar to me.

A retail CEO once remarked that retail is a pretty "public" industry. There is not much behind which a retail business can hide. As such, retailers often have to follow the aspirations and concerns of the communities and customers they serve. The journey of FairPrice from being an EDLP (everyday low price) retailer to one having multiple formats, businesses and channels has been catalogued well in the book.
FairPrice overseas business forays make for interesting reading. It is one that I was not aware of. However, FairPrice is not alone in this aspect, many global retailers have had similar experiences. These experiences often help reinforce the core vision that retailers originally start out with.
The advent of competition, both international and local has spurred FairPrice into focusing on the customer. Often, this is the most reliable way of success for retailers instead of following competition.
Also, the adage of "location, location, location" continues to hold true in retail even with rising competition from digital channels like - ecommerce.

The New Trajectories

Be it digital channels or loyalty, FairPrice continues to engage customers with value propositions beyond retail. It is starting to build a "consumer lifestyle platform" of sorts by offering financial services (through the tie up with Trust Bank), Core retail (FairPrice and Cheers) and F&B (in the form Kopitiam). It is an interesting play, not tried by many other retailers.
As FairPrice sources products from around 100 countries, it is also gaining confidence in it's house brands. In retail, it is often the holy grail to create brands and earn higher margins through them. It is also exploring the ideas of growing these brands further in local and overseas markets.

Summary

Among all retailers, grocery retailers have a unique distinction. They form an invisible bond and sometimes tenuous relationship with the communities that they serve.

The continued market leadership of FairPrice is good indicator of the trust that customers continue to have in the brand. As FairPrice completes it's 50th anniversary, here is wishing it many more!

Together

2023-07-31T18:27:00.003+05:30

Together

I see the sun set in you eyes.

A sky full of stars,

fills my sight.

A short poem inspired by the haiku format.

Roast Chicken for the Holiday Season

2021-12-26T09:12:00.001+05:30

Roast Chicken

In some parts of the world, December is the time of holidays and colder weather. During this time, it's nice to have a sumptuous and healthy meal. Here is a fairly simple recipe for roast chicken. The ingredients should be available everywhere.

Contributed by Rajan Manickavasagam.

Published 19. December 2021

Roast Chicken with Spiced Pomegranate Juice

Ingredients

Salt (to taste)
Crushed black pepper
1 Pomegranate
12-15 peeled baby potatoes
2 Onions
2 Carrots
2 Red Bell Peppers
6 Chicken thigh pieces
10 tablespoons of olive oil
1 tablespoon of garam masala
1 tablespoon of Kashmiri red chilli powder
1/2 teaspoon white wine vinegar
1/2 teaspoon baking soda

Instructions

Roast Chicken

Peel and blend 1 pomegranate in a blender without adding any water.
Divide the pomegranate juice into 2 cups/portions.
To on one of the portions, add 2 tablespoons olive oil, salt, pepper, 2 tablespoons garam masala and 1 tablespoon Kashmiri red chilli powder.
Coat the chicken pieces in this mixture and let it marinate for a few hours.
In a flat bottom pan, add some olive oil, garlic, red bell peppers, diced carrots and onions.
Once the onions start to brown, add the chicken to the pan and stir fry them until they are half cooked.
Transfer the garlic, onions and chicken to a baking tray. Drizzle some olive oil on the chicken and bake at 180C for 30 minutes.

Pomegranate Sauce

Remove some of the garlic and onions from the baking tray and blend them in a food processor along with the remaining portion of the pomegranate juice.
Add half a teaspoon of white wine vinegar and 1 tablespoon of olive oil to the sauce.
This now becomes the pomegranate based sauce for the meal.

Roast Potatoes

Peel the baby potatoes and boil them in a hot water again with some salt, pepper and baking soda.
Once the potatoes are cooked, drain the water and fry them in 2 tablespoons of olive oil with salt and pepper.

Finally, serve all of the above and drizzle the pomegranate sauce over the roast chicken and the vegetables.

Happy eating.

Further details

Serves 4-6 people.

Preparation time is approximately 30 min.

Cooking time is approximately 60 min.

Prime Candidate for Analysis

2021-07-04T11:37:00.013+05:30

Promotions in Retail

Promotions are part of the 4P's (Product, Price, Promotion, Place) of retail marketing. Promotions have always been a key part of the customer experience in retail. Promotional activities largely include a mix of marketing and advertising spend.

Evolution of Promotions

With the onset of digital/online retail, the nature of promotional activity have been changing.

Earlier in the world of physical retail, promotions were characterized by a sense of "theatre". It was about decking up the stores with banners, special merchandising and other ploys (such as the KMart Blue Lights).

In the digital/online world, the "theatre" element has been replaced by one of "urgency". Flash sales, auctions, product launches etc. give customers a sense of urgency to grab the deals as they are available for a small amount of time.

The One that Started it All

Typically, promotions had been linked to major festivals or events such as Christmas (in most parts of the world), Diwali (in India), Chinese New Year (in China) and so on. Supplementing these festival events were the other smaller events such as - Back to School, Mothers Day, Valentines Day, etc. But, none of these events happened in July which was an "empty" spot in the retail calendar.

So, Amazon picked up this time of the year for their "Prime Days". But, another aspect of Prime Days is that it is also aimed at shoring up customers for the "Prime" loyalty program. Of course, apart from quicker deliveries, Prime customers also enjoy other add-on services from Amazon such as Music, TV, Movies etc.

The recent acquisition of MGM studios by Amazon is probably going to make these add-on services more compelling. Also, the copyright from MGM could potentially be leveraged for themed products. For example, the James Bond franchise is owned by MGM. Maybe next time, you fire up a drone, it's not Alexa but Bond who would do your bidding. Before I digress further, let's get back to Prime!

Analysis

As he has been doing for the past few years, Jeff Barr (AWS Technical Evangelist) has been revealing a few tidbits about the technical infrastructure powering Amazon Prime Day. So, I took the opportunity to study Prime Day in a bit more detail both from a business and technical perspective.

The below table has some data on the different types of technical infrastructure used by Amazon. Of course, it may not be the full list, but still gives us an idea of scale (and therefore cost) of the infrastructure required to run an event as massive as Prime Day. It gives us some clues to the nature/style of the architecture as well.

Apart from the technology, some of the other analysis is around sales, number of items sold and the split between Amazon and 3rd Party Seller share of sales.

Insights

Of course, some of the above data is based on assumptions and estimates. But, it can still help draw some interesting inferences and insights. Let's split these into business and technology. Potentially, we can also draw some correlation between the two as well.

Business

In terms of percentage to overall sales, Prime Day will probably contribute around 3% to overall Amazon sales in 2021
Compared to the first edition in 2015, duration of Prime Day has increased by around 50% to 66 hours
The average sales per item also probably increased from $26 in 2015 to around $54 in 2020
Over the years, the share of Amazon (vs Marketplace sellers) in Prime Day sales has increased from around 50% to 70%

Technology

If we look at a couple of the infrastructure metrics like AWS Cloudfront and AWS DynamoDB, they are not that different between 2020 and 2021. However, the sales seem to have doubled. This probably indicates a few things -

We have already seen that mall footfall has been hit hard by the pandemic. More customers continue to shop online
More customers have shopped with an intent to purchase items and conversion has probably been higher

Compared to 2020, AWS EBS volumes have come down by 34% in 2021. It could indicate one or a combination of the below -

Optimization in the technical infrastructure or architecture
Leaner assortment of products
Better customer targeting
Lesser number of deals

Depending on the cloud that you are using, one can punch in the above technical metrics from the table into the AWS Pricing Calculator / Azure Pricing Calculator / GCP Pricing Calculator to get a sense of the cost required to run an event like Prime Day

Summary

Assuming that more efficiency can be squeezed out of operations, there is a possibility that Prime Day may one day morph into a Prime Week (like the longer sales period done by retailers such as JD.com)
The profitability from Prime Day is probably higher as the supply chain network (warehouses and delivery fleets) can be leveraged for faster inventory turns than during normal periods. And, this profitability will only increase further with investments such as -

renewable energy for the warehouses and deliver fleet
better robotics
improved data and AI for supply chain
competitive pricing

Continue to retain and acquire customers for the Prime program
A trend probably exacerbated by the pandemic has been the rise of marketplace aggregators that are buying up the Fulfilled By Amazon sellers and Direct-to-Consumer brands. Examples of these are companies such as - Thrasio, SellerX, RazorPay etc. For example, Thrasio has so far acquired over 100 brands with around 15,000 products. Over a period of time, we may see a kind of triopoly on Amazon consisting of - Amazon, CPG Brands and Marketplace Aggregators.

References

A Hyper Local, Crowd Sourced, Data Driven Chatbot for Covid

2020-07-18T22:24:00.015+05:30

Overview

The COVID-19 pandemic has captured everyone's attention and impacted our daily lives for the past few months. It is likely to remain that way for the foreseeable future too as we learn to navigate around it.

Several organizations and volunteers globally have create various kinds of dashboards, databases etc. so that information is available to everyone. Some of them are -

Covid 19 database by John Hopkins - https://github.com/CSSEGISandData/COVID-19
Covid 19 India dashboards - https://github.com/covid19india/covid19india-react
Covid 19 India clusters - https://github.com/someshkar/covid19india-cluster
WHO app - https://github.com/WorldHealthOrganization/app
Covid 19 time series data - https://github.com/pomber/covid19
And, many others

Approach

From the various open source applications already created for COVID, came my inspiration too, for a chat bot. The idea is to empower local communities with data and insight so that they are better informed as they go about their daily tasks and routine. This might help people assess the "risk" where -

some of them have to an office or public place for work
children going to schools/parks
people heading out for chores/outdoors
and similar scenarios

Concept

Governments, NGO's, businesses, health officials all over the world are taking several initiatives to keep people safe. However, it is imperative that local communities also take steps to safe guard themselves. The idea is to create a chat bot that is setup and maintained by each local community, kind of like a ham radio movement.

Such chat bots could provide both quantitative and qualitative data to the users. This chat bot could act as a digital sentinel for each local community. So, let's call them DISCO (Distributed Information Sentinels for COvid) chat bots. Each of these DISCO bots will be maintained by each community.

Technology

The first step is to gather the relevant data. This can be done in a google spreadsheet, as it allows multiple people to update the sheet at the same time. In the example here, data from Arogya Setu app is used to capture the covid positive cases and at-risk cases for a given location. A sample spreadsheet is here. This is the kind of "quantitative" data that the DISCO bot could provide.

Often, many people have queries regarding the pandemic. An FAQ conversation model can be built that can also be integrated into the DISCO bot. A sample is provided here. This is the kind of qualitative data the DISCO bot could provide.

Next, we need a host computer to run the bot. It could be a desktop, laptop, server, cloud or even a Raspberry Pi. This bot would download the above spreadsheet regularly and keep a local cache (for quicker responses to user queries). My copy of the DISCO bot for the local community is running on my Raspberry Pi 3 that has been idle for years.

Lastly, the community can use Telegram chat app on a compatible device to interact with the bot. Scroll below for the demos on how to setup and use the DISCO bot.

Drivers and Goals

The following drivers and goals have driven the design and implementation of this chat bot.

Local
Yet Global
Community/Volunteer Driven
Accessible
Simple and Cheap

Local

As we have seen so far, each community can setup their own little 'database' in the form of a google spreadsheet. The idea is not to have a global or even a national instance of the chat bot, but for each community/locality/suburb to maintain one for themselves.

With many countries and communities in various stages of lock down at some time or other, many people - professionals, students, elderly and children (basically everyone!) are home bound most of the time. Hopefully, this idea motivates people to reuse this chat bot and build something similar for their local communities. This way, people can learn/share some knowledge on technology and help their communities at the same time.

Yet Global

Largely, people are limiting their travel and daily routines to their immediate and near by areas. This approach can be adopted by communities and organizations around the world.

Use cases for this bot could be residential communities, educational institutions, workplaces (where remote working is not possible), etc.

Community/Volunteer Driven

The two key stakeholders for this chat bot are the volunteers -

Data volunteers - Their responsibility is not just to enter/maintain the data, but to also ensure that the data is "as true" as possible. The data volunteers are kind of playing the role of citizen journalists. One of the "journalistic ethics" is to have a story/lead/data verified by at least 2 independent sources. This is the reason for the google spreadsheet to have the 2 columns - "Data Source Verified By (1): and "Data Source Verified By (2)". It is also important to ensure that the data maintained is sufficiently protected and privacy maintained. For more details, please refer Reuters Handbook.
Technology Volunteers - Their responsibility is to setup and run the chat bot. It should take roughly 1-2 hours to set it all up from scratch. I will be happy if someone can set it up faster than that. The source code and installation instructions are available here at Github.

Accessible

Creating chat bots using Telegram API's is incredibly simple. Also, Telegram provides various free chat clients across all devices and form factors - web, desktop, mobile and tablet apps.

Unless someone is on a "feature phone", virtually anyone in the world should be able to access their respective local COVID chat bot over a data connection (mobile or broadband or wi-fi) from a device of their choosing.

Simple and Cheap

Simplicity here has 4 connotations -

simple to maintain the data
simple and light weight to setup and run the chat bot
simple and actionable insights
simple for anyone to use the chat bot

All the technologies used in this chat bot are either free or open source or cheap to buy.

No	Technology	Purpose	Cost	Availability	Remarks
1	Google Spreadsheet	Serves as UI and database for the volunteers to enter and maintain data	Free	Global
2	Python	All the programs and libraries for the chat bot are written using Python	Free	Global
3	Chat Bot Host	Run the chat bot and other programs	Raspberry Pi - 4,000 INR (approx. 50 USD)	Global	The chat bot created in this post runs on a Raspberry Pi 3. Alternatively, these programs can just run in any existing laptop/desktop/server that can run python 3.6.4 or above.

Chat Bot in Action

Step 1 - Updating data in a google spreadsheet

Step 2 - Training a sample FAQ for the chat bot

Step 3 - Downloading google spreadsheet to a local file cache and running the chat bot on a Raspberry Pi

Step 4 - Testing an alpha version of the chat bot in the Telegram Web app

Step 4 - Testing current version of the chat bot in Telegram app for iPhone

How to setup your own Bot

The source code and installation instructions are available here at Github. Feel free to customize to your requirements. You can setup and run a bot for your local community/organization.

Summary

In this example, the health and contact tracing application from the Indian central/federal government - Arogya Setu (rough English translation - Bridge to your Health) has been used to collect anonymous data for a specific location where I live.

A brief comparison between the two apps, key points highlighted in bold.

Arogya Setu	‘Local’ DISCO Bot
A health an contact tracing app maintained by the central/federal government of India	A community driven app built and maintained by each local community
Uses GPS, Data, Bluetooth	Uses only Data
Collects information based on the scenario	Doesn’t collect any information from end user
Provides data at a “point in time” for a location. Data available for entire country.	Provides data at a “point in time” and trends. Data available only for a location.
Contains a FAQ	Contains FAQ
Requires installation of the Arogya Setu App on Android or iOS	Requires installation of the Telegram messenger app on Desktop, Mobile and Tablet
No customization	Customization based on the needs of the community by the community volunteers themselves
Public and health agencies collate the data and feed into the app	Data volunteers need to look at the Arogya Setu app and/or other sources and enter data into a google spreadsheet for their location

Take care and stay safe.

Tech - Check your Salt - Part 3

2020-03-29T22:33:00.003+05:30

Overview

In the previous post, we saw how beacons were deployed in all machines using state files. Salt internally uses zero MQ messaging for communication between the master and minions. As with any messaging framework, there are publisher(s) and consumer(s). One can think of the beacons (on salt minions) as publishers and reactors (on salt master) as consumers.

Salt Concepts Covered

In this post, the following Salt concepts are covered -

Beacons - Monitor minions for state changes, system diagnostics etc.
Reactors - Trigger reactions when events occur in the managed systems running minions.

Beacons

A beacon on the minion can be used to monitor various system events such as - memory usage, CPU load average, overall statistics, user commands etc. The beacon deployed in the previous post contains the 'sh' module. Sh module monitors all commands run on a machine which has the salt minion running.

Note: Every time a beacon is deployed to the salt minions, the minions needs to be restarted.

Reactors

Now that we have the minion setup, they keep sending the events to the salt master. We now create a reactor that will respond to these events.

Create the /etc/salt/master.d/reactor.conf file on the master -

# contents of the reactor.conf file
reactor:
- 'salt/beacon/*/sh/*'
- '/srv/reactor/sh.sls'

Create a Slack account and add an application in it. Use the API key and the channel name from Slack in the below reactor - sh.sls.

Next, create the /srv/reactor/sh.sls on the master -

Send sh to Slack:

local.slack.post_message:

- tgt: {{ data['id'] }}

- kwarg:

channel: "[Slack Channel Name]"

api_key: "[Slack API Key]"

message: "{{ data['id'] }} : user {{ data['user'] }} ran command {{ data['cmd'] }} with arguments {{ data['args'] }}."

from_name: "Sh Bot"

Note: Every time a new reactor is deployed to the salt master, the master needs to be restarted.

After the beacons and reactors have been set up, you can also monitor the events using the command (in the salt master) - 'sudo salt-run state.event pretty=True'

Now, all commands that are run on the salt minions can be viewed in Slack (both on a the desktop and mobile apps). This way, the user logins, commands run etc. across the infrastructure can now be monitored and tracked using a Slack channel.

Slack Desktop

Slack Mobile

Summary

So far, we have seen how salt has been used to -

Ease the management of infrastructure (servers, desktops, IoT devices)
Maintain consistent state across the infrastructure environment
Proactively respond to events in the infrastructure for monitoring and compliance purposes

Tech - Spread the Salt evenly - Part 2

2020-03-29T21:21:00.002+05:30

Overview

In the previous post, we saw how to setup the salt master and minions. Now, let's see how we can use Salt to maintain a consistent state in the infrastructure. The state here could mean - software, configuration, permissions etc.

Salt Concepts Covered

In this post, the following concepts are covered -

Formulas (States) - A declarative or imperative representation of a system configuration.
Top File - Matches formulas and Salt pillar data to Salt minions.

State Management

One of the key concepts in Salt, is state management. State management in Salt ensures that the applications and infrastructure is managed centrally (as you would using other tools such as Chef, Ansible, Puppet). In addition, state management also ensures that if someone changes the configuration in a minion, state management will rest it back to what it should be. This way, Salt ensures that all changes to application, infrastructure, configuration, permission etc. are managed centrally through Salt master.

In the salt master server, create a folder called /srv/salt. In this folder, we are going to create the salt state management files. In this folder, create a file called top.sls -

# contents of the top.sls file
base:
'*':
- packages
- beacons

Add a file called packages.sls -

# contents of the packages.sls file
python-pip:
pkg.installed

pynotify:
pip.installed:
- require:
- pkg: python-pip

Add the last file in this folder, beacons.sls -

# contents of the beacons.sls file

/etc/salt/minion.d/beacons.conf:

file.managed:

- source:

- salt://files/minion.d/beacons.conf

- makedirs: True

Create a folder called /srv/salt/files/minion.d. In this folder, create a file called beacons.conf -

# contents of the beacons.conf file. beacons monitor minion nodes
beacons:
sh: {}

Finally, run the following command - 'sudo salt '*' state.apply'. This command will take the top.sls and apply the state defined in it.

This will now set the 'state' in all the machines running the salt minion agents. This means that the pynotify package is installed in all machines and also create the '/etc/salt/minion.d/beacons.conf' file. We will come to the beacon in the next post.

Summary

Salt can be used to ensure that infrastructure components - web/application servers, firewalls, other system configuration are maintained consistently. Any change to the infrastructure has to be made through salt and this way configuration "drift" is avoided.

In the next post, we will look at how the infrastructure can be proactively monitored and managed using beacons and reactors.

Tech - Sprinkle some Salt - Part 1

2019-09-22T12:27:00.001+05:30

Overview

With the advent of DevOps, many tools have come up on the topic of automation. These include tools such as Jenkins, TeamCity, Ansible, Chef, Puppet, Vagrant, Terraform etc. And, they are used to deploy infrastructure and applications.

However, once an infrastructure or application is deployed into an environment (Development, Testing, Production), there are changes made to it. The operations and/or site reliability teams may tweak operating system, application configuration to ensure that the environment is stable and functioning well. And, when these changes are made, the environment tends to 'drift' when compared to the source code or documentation.

This is where tools such as Salt Stack are useful. It is a tool from https://www.saltstack.com.

At a high level, Salt is a configuration management system.
Like most other automation tools, it can deploy infrastructure / application.
In addition, it can also monitor and react to various events. This greatly improves the life of the operations and site reliability teams.

In this post, we will go over the basics of Salt Stack (also known as Salt).

Salt Architecture

Salt is a versatile tool. As you can see from the architecture, Salt can be used to manage and monitor infrastructure in diverse scenarios -

on premise data centre physical servers
on premise data centre virtual servers
network switches, routers
public cloud servers
IoT devices

Image from Mirantis

In a series of posts, we will see how Salt can be setup for managing virtual server and IoT devices.

Salt Concepts Covered

Each blog post in these series will cover the salt concepts incrementally. In this post, the following concepts are covered -

Salt Master - This is the master is used to send commands and configurations to the Salt minion that is running on managed systems.
Salt Minions - This system runs the Salt minion which receives commands and configuration from the Salt master.
Execution Modules - Ad hoc commands can be executed against the managed systems. These are typically used for - one-off commands, deploy software updates etc.
Grains - Grains are the information underlying managed system and include operating system, memory, and many other system properties. One can also define custom grains for any system.

Pre-requisites

It may be best to have a few virtual or test machines for getting started with Salt. In this example, a bunch of virtual machines in Oracle Virtualbox are used.

xubuntu-jumpbox - Acts as a Salt master server
xubuntu-node1, xubuntu-node2, xubuntu-node3 - Act as a nodes in a server cluster
All the virtual machines run xubuntu 18.04. They have multiple network interfaces such as - NAT and Host-Only. xubuntu is chosen here as they have a very light memory footprint.
For fun, there is a Raspberry Pi 3 as well that acts as an IoT node.
An SSH client such as Putty or Bitvise. In this series, Bitvise and Windows Subsystem for Linux in Windows are used as clients.

Installation

Salt has two key components - master and minion. Masters act as controllers and the minions acts as agents.

Login to the master node using an SSH client. Before installation, ensure that the ports 4505 and 4506 are open in the master node, in this case xubuntu-jumpbox. The command to open ports in firewall is 'sudo ufw allow 4505:4506/tcp'. Check the status of the firewall using 'sudo ufw status'.

Add the salt repository using the command - 'sudo add-apt-repository ppa:saltstack/salt'.

Next, run the command - 'sudo apt-get update'.

Finally, run this command to install salt master - 'sudo apt-get install salt-master -y'.

Edit the salt master configuration file by this command - 'sudo nano /etc/salt/master'. Add the host-only ip address of virtual machine to this file.

Restart salt master using the command - 'sudo systemctl restart salt-master'.

Run 'sudo systemctl status salt-master' to view details of the service.

Next run this command - 'sudo salt-master --version'.

Now, that the master is setup, let's start installing the minions. Start the other virtual machines in virtualbox. Login to them using the SSH client.

Add the salt repository using the command - 'sudo add-apt-repository ppa:saltstack/salt'.

Next, run the command - 'sudo apt-get update'.

Finally, run this command to install salt master - 'sudo apt-get install salt-minion -y'.

Edit the salt master configuration file by this command - 'sudo nano /etc/salt/minion'. Add the host-only ip address of virtual machine to this file.

Restart salt master using the command - 'sudo systemctl restart salt-minion'.

Run 'sudo systemctl status salt-minion' to view details of the service.

Once the salt-minion service is restarted, it would have sent the public key of the minion to the salt master. Repeat this process for all the nodes including the Raspberry Pi 3.

Similarly, the version of the minion can also be checked by 'sudo salt-minion --version'.

The request from the minion can be seen in the salt server by running the command - 'sudo salt-key -L'.

Now, run the command to add the minion to the Salt master - 'sudo salt-key --accept='raspberrypi-node1''.

Once the keys have been accepted in the salt master, the master can start interacting with all the nodes via the minions.

Salt comes with several pre-built modules. One of them is cmd. In this example, this module is used to run commands on all nodes - 'sudo salt '*node*' cmd.run 'sudo apt-get install nmon --upgrade -y' --output=pprint '. This command will run nmon tool on all the minions that have the text 'node' in their machine name.

Let's verify if the nmon tool has indeed been installed in the nodes. Login to one of the nodes and verify, in this case xubuntu-node3.

Let's try another command - 'sudo salt '*node*' cmd.run 'sudo apt-get update | sudo apt-get upgrade -y' --output=pprint '. As seen in this example, multiple commands are separated by the pipe operator.

You can also run commands such as 'sudo salt '*' test.ping' and 'sudo salt '*' network.get_hostname' to view the status of the minions.

You can run the command 'sudo salt '*' grains.items' to view the system information for all the minions. In this example, we can see the system information specific for a Raspberry Pi. These grains can also be used for targeting specific states.

Summary

In this post, we have installed/updated software in 4 nodes (3 virtual machines and 1 Raspberry Pi) using 1 salt master. Imagine a scenario where, there are 100's or 1000's of machines that need to be maintained. It is in such scenarios that Salt proves it's usefulness.

It is possible to have masters in a high-availability mode, but that it out of scope of these posts. In addition to multi-master mode, it is also possible to federate the architecture using features such as Salt Syndic. To know more about Salt architecture and how it scales, you can read about it here.

If you are interested in a primer for the various salt modules and commands, you an refer the samples in my github gist. As and when I come across something useful, I will keep updating the gist.

More about other interesting features of Salt in the next post.

Azure Chronicles - VM Security

2019-08-22T00:10:00.003+05:30

Using the cloud has many benefits, some of them such as elasticity and scalability. On the other hand, some other aspects such as security and cost need to be paid attention to.

Let's look at the security aspect in this post.

Lynis is one of the most popular Linux auditing and hardening tools. It has community and enterprise editions. As an example, the community edition is being used here. Let's scan a Linux based VM in Azure and find out the results. This post assumes that you have an existing subscription in Microsoft Azure.

The below steps show how to scan a VM in Azure.

A new class of VM's called B1ls was launched. These VM's are quite small and cheap. As this is just an experiment, let's try this new VM.
Create a B1ls VM with the Ubunto 18.04 LTS image on it. Make sure that the VM has a public IP address and use a public key for access.
Once the VM is created, login to the VM using a tool such as Putty.
Follow the steps described here to install the Lynis tool.
Run the following command to analyze the VM - sudo lynis audit system.
The report generated by the lynis tool is available in /var/log/lynis-report.dat. This is a text file and can be read in any text editor.

A sample report is shown below for the Ubuntu 18.04 LTS VM from Azure. The vulnerabilities are marked in bold below.

[ Lynis 2.7.5 ]

################################################################################
Lynis comes with ABSOLUTELY NO WARRANTY. This is free software, and you are
welcome to redistribute it under the terms of the GNU General Public License.
See the LICENSE file for details about using this software.

2007-2019, CISOfy - https://cisofy.com/lynis/
Enterprise support available (compliance, plugins, interface and tools)
################################################################################

[+] Initializing program
------------------------------------
- Detecting OS... [ DONE ]
- Checking profiles... [ DONE ]

---------------------------------------------------
Program version: 2.7.5
Operating system: Linux
Operating system name: Ubuntu Linux
Operating system version: 18.04
Kernel version: 5.0.0
Hardware platform: x86_64
Hostname: ubuntu1804ltsvm
---------------------------------------------------
Profiles: /etc/lynis/default.prf
Log file: /var/log/lynis.log
Report file: /var/log/lynis-report.dat
Report version: 1.0
Plugin directory: /usr/share/lynis/plugins
---------------------------------------------------
Auditor: [Not Specified]
Language: en
Test category: all
Test group: all
---------------------------------------------------
- Program update status... [ NO UPDATE ]

[+] System Tools
------------------------------------
- Scanning available tools...
- Checking system binaries...

[+] Plugins (phase 1)
------------------------------------
Note: plugins have more extensive tests and may take several minutes to complete

- Plugins enabled [ NONE ]

[+] Boot and services
------------------------------------
- Service Manager [ systemd ]
- Checking UEFI boot [ DISABLED ]
- Checking presence GRUB [ OK ]
- Checking presence GRUB2 [ FOUND ]
- Checking for password protection [ NONE ]
- Check running services (systemctl) [ DONE ]
Result: found 23 running services
- Check enabled services at boot (systemctl) [ DONE ]
Result: found 49 enabled services
- Check startup files (permissions) [ OK ]

[+] Kernel
------------------------------------
- Checking default run level [ RUNLEVEL 5 ]
- Checking CPU support (NX/PAE)
CPU support: PAE and/or NoeXecute supported [ FOUND ]
- Checking kernel version and release [ DONE ]
- Checking kernel type [ DONE ]
- Checking loaded kernel modules [ DONE ]
Found 61 active modules
- Checking Linux kernel configuration file [ FOUND ]
- Checking default I/O kernel scheduler [ NOT FOUND ]
- Checking for available kernel update [ OK ]
- Checking core dumps configuration [ DISABLED ]
- Checking setuid core dumps configuration [ PROTECTED ]
- Check if reboot is needed [ NO ]

[+] Memory and Processes
------------------------------------
- Checking /proc/meminfo [ FOUND ]
- Searching for dead/zombie processes [ OK ]
- Searching for IO waiting processes [ OK ]

[+] Users, Groups and Authentication
------------------------------------
- Administrator accounts [ OK ]
- Unique UIDs [ OK ]
- Consistency of group files (grpck) [ OK ]
- Unique group IDs [ OK ]
- Unique group names [ OK ]
- Password file consistency [ OK ]
- Query system users (non daemons) [ DONE ]
- NIS+ authentication support [ NOT ENABLED ]
- NIS authentication support [ NOT ENABLED ]
- sudoers file [ FOUND ]
- Permissions for directory: /etc/sudoers.d [ OK ]
- Permissions for: /etc/sudoers [ OK ]
- Permissions for: /etc/sudoers.d/90-cloud-init-users [ OK ]
- Permissions for: /etc/sudoers.d/README [ OK ]
- PAM password strength tools [ SUGGESTION ]
- PAM configuration files (pam.conf) [ FOUND ]
- PAM configuration files (pam.d) [ FOUND ]
- PAM modules [ FOUND ]
- LDAP module in PAM [ NOT FOUND ]
- Accounts without expire date [ OK ]
- Accounts without password [ OK ]
- Checking user password aging (minimum) [ DISABLED ]
- User password aging (maximum) [ DISABLED ]
- Checking expired passwords [ OK ]
- Checking Linux single user mode authentication [ OK ]
- Determining default umask
- umask (/etc/profile) [ NOT FOUND ]
- umask (/etc/login.defs) [ SUGGESTION ]
- LDAP authentication support [ NOT ENABLED ]
- Logging failed login attempts [ ENABLED ]

[+] Shells
------------------------------------
- Checking shells from /etc/shells
Result: found 6 shells (valid shells: 6).
- Session timeout settings/tools [ NONE ]
- Checking default umask values
- Checking default umask in /etc/bash.bashrc [ NONE ]
- Checking default umask in /etc/profile [ NONE ]

[+] File systems
------------------------------------
- Checking mount points
- Checking /home mount point [ SUGGESTION ]
- Checking /tmp mount point [ SUGGESTION ]
- Checking /var mount point [ SUGGESTION ]
- Query swap partitions (fstab) [ NONE ]
- Testing swap partitions [ OK ]
- Checking for old files in /tmp [ OK ]
- Checking /tmp sticky bit [ OK ]
- Checking /var/tmp sticky bit [ OK ]
- ACL support root file system [ ENABLED ]
- Mount options of / [ NON DEFAULT ]
- Checking Locate database [ FOUND ]
- Disable kernel support of some filesystems

[+] USB Devices
------------------------------------
- Checking usb-storage driver (modprobe config) [ NOT DISABLED ]
- Checking USB devices authorization [ DISABLED ]
- Checking USBGuard [ NOT FOUND ]

[+] Storage
------------------------------------
- Checking firewire ohci driver (modprobe config) [ DISABLED ]

[+] NFS
------------------------------------
- Check running NFS daemon [ NOT FOUND ]

[+] Name services
------------------------------------
- Checking search domains [ FOUND ]
- Checking /etc/resolv.conf options [ FOUND ]
- Searching DNS domain name [ FOUND ]
Domain name: tddhuyalmt4urnpob1l1hchkbh.bx.internal.cloudapp.net
- Checking /etc/hosts
- Checking /etc/hosts (duplicates) [ OK ]
- Checking /etc/hosts (hostname) [ SUGGESTION ]
- Checking /etc/hosts (localhost) [ OK ]
- Checking /etc/hosts (localhost to IP) [ OK ]

[+] Ports and packages
------------------------------------
- Searching package managers
- Searching dpkg package manager [ FOUND ]
- Querying package manager
- Query unpurged packages [ FOUND ]
- Checking security repository in sources.list file [ OK ]
- Checking APT package database [ OK ]
- Checking vulnerable packages [ WARNING ]
- Checking upgradeable packages [ SKIPPED ]
- Checking package audit tool [ INSTALLED ]
Found: apt-get
- Toolkit for automatic upgrades (unattended-upgrade) [ FOUND ]

[+] Networking
------------------------------------
- Checking IPv6 configuration [ ENABLED ]
Configuration method [ AUTO ]
IPv6 only [ NO ]
- Checking configured nameservers
- Testing nameservers
Nameserver: 127.0.0.53 [ OK ]
- Checking default gateway [ DONE ]
- Getting listening ports (TCP/UDP) [ DONE ]
- Checking promiscuous interfaces [ OK ]
- Checking waiting connections [ OK ]
- Checking status DHCP client [ NOT ACTIVE ]
- Checking for ARP monitoring software [ NOT FOUND ]

[+] Printers and Spools
------------------------------------
- Checking cups daemon [ NOT FOUND ]
- Checking lp daemon [ NOT RUNNING ]

[+] Software: e-mail and messaging
------------------------------------

[+] Software: firewalls
------------------------------------
- Checking iptables kernel module [ FOUND ]
- Checking iptables policies of chains [ FOUND ]
- Checking for empty ruleset [ WARNING ]
- Checking for unused rules [ OK ]
- Checking host based firewall [ ACTIVE ]

[+] Software: webserver
------------------------------------
- Checking Apache [ NOT FOUND ]
- Checking nginx [ NOT FOUND ]

[+] SSH Support
------------------------------------
- Checking running SSH daemon [ FOUND ]
- Searching SSH configuration [ FOUND ]
- SSH option: AllowTcpForwarding [ SUGGESTION ]
- SSH option: ClientAliveCountMax [ SUGGESTION ]
- SSH option: ClientAliveInterval [ OK ]
- SSH option: Compression [ SUGGESTION ]
- SSH option: FingerprintHash [ OK ]
- SSH option: GatewayPorts [ OK ]
- SSH option: IgnoreRhosts [ OK ]
- SSH option: LoginGraceTime [ OK ]
- SSH option: LogLevel [ SUGGESTION ]
- SSH option: MaxAuthTries [ SUGGESTION ]
- SSH option: MaxSessions [ SUGGESTION ]
- SSH option: PermitRootLogin [ OK ]
- SSH option: PermitUserEnvironment [ OK ]
- SSH option: PermitTunnel [ OK ]
- SSH option: Port [ SUGGESTION ]
- SSH option: PrintLastLog [ OK ]
- SSH option: StrictModes [ OK ]
- SSH option: TCPKeepAlive [ SUGGESTION ]
- SSH option: UseDNS [ OK ]
- SSH option: VerifyReverseMapping [ NOT FOUND ]
- SSH option: X11Forwarding [ SUGGESTION ]
- SSH option: AllowAgentForwarding [ SUGGESTION ]
- SSH option: AllowUsers [ NOT FOUND ]
- SSH option: AllowGroups [ NOT FOUND ]

[+] SNMP Support
------------------------------------
- Checking running SNMP daemon [ NOT FOUND ]

[+] Databases
------------------------------------
No database engines found

[+] LDAP Services
------------------------------------
- Checking OpenLDAP instance [ NOT FOUND ]

[+] PHP
------------------------------------
- Checking PHP [ NOT FOUND ]

[+] Squid Support
------------------------------------
- Checking running Squid daemon [ NOT FOUND ]

[+] Logging and files
------------------------------------
- Checking for a running log daemon [ OK ]
- Checking Syslog-NG status [ NOT FOUND ]
- Checking systemd journal status [ FOUND ]
- Checking Metalog status [ NOT FOUND ]
- Checking RSyslog status [ FOUND ]
- Checking RFC 3195 daemon status [ NOT FOUND ]
- Checking minilogd instances [ NOT FOUND ]
- Checking logrotate presence [ OK ]
- Checking log directories (static list) [ DONE ]
- Checking open log files [ DONE ]
- Checking deleted files in use [ FILES FOUND ]

[+] Insecure services
------------------------------------
- Installed inetd package [ NOT FOUND ]
- Installed xinetd package [ OK ]
- xinetd status [ NOT ACTIVE ]
- Installed rsh client package [ OK ]
- Installed rsh server package [ OK ]
- Installed telnet client package [ OK ]
- Installed telnet server package [ NOT FOUND ]

[+] Banners and identification
------------------------------------
- /etc/issue [ FOUND ]
- /etc/issue contents [ WEAK ]
- /etc/issue.net [ FOUND ]
- /etc/issue.net contents [ WEAK ]

[+] Scheduled tasks
------------------------------------
- Checking crontab and cronjob files [ DONE ]
- Checking atd status [ RUNNING ]
- Checking at users [ DONE ]
- Checking at jobs [ NONE ]

[+] Accounting
------------------------------------
- Checking accounting information [ NOT FOUND ]
- Checking sysstat accounting data [ NOT FOUND ]
- Checking auditd [ NOT FOUND ]

[+] Time and Synchronization
------------------------------------
- NTP daemon found: systemd (timesyncd) [ FOUND ]
- Checking for a running NTP daemon or client [ OK ]

[+] Cryptography
------------------------------------
- Checking for expired SSL certificates [0/1] [ NONE ]

[+] Virtualization
------------------------------------

[+] Containers
------------------------------------

[+] Security frameworks
------------------------------------
- Checking presence AppArmor [ FOUND ]
- Checking AppArmor status [ ENABLED ]
- Checking presence SELinux [ NOT FOUND ]
- Checking presence TOMOYO Linux [ NOT FOUND ]
- Checking presence grsecurity [ NOT FOUND ]
- Checking for implemented MAC framework [ OK ]

[+] Software: file integrity
------------------------------------
- Checking file integrity tools
- Checking presence integrity tool [ NOT FOUND ]

[+] Software: System tooling
------------------------------------
- Checking automation tooling
- Automation tooling [ NOT FOUND ]
- Checking for IDS/IPS tooling [ NONE ]

[+] Software: Malware
------------------------------------

[+] File Permissions
------------------------------------
- Starting file permissions check
/root/.ssh [ OK ]

[+] Home directories
------------------------------------
- Checking shell history files [ OK ]

[+] Kernel Hardening
------------------------------------
- Comparing sysctl key pairs with scan profile
- fs.protected_hardlinks (exp: 1) [ OK ]
- fs.protected_symlinks (exp: 1) [ OK ]
- fs.suid_dumpable (exp: 0) [ DIFFERENT ]
- kernel.core_uses_pid (exp: 1) [ DIFFERENT ]
- kernel.ctrl-alt-del (exp: 0) [ OK ]
- kernel.dmesg_restrict (exp: 1) [ DIFFERENT ]
- kernel.kptr_restrict (exp: 2) [ DIFFERENT ]
- kernel.randomize_va_space (exp: 2) [ OK ]
- kernel.sysrq (exp: 0) [ DIFFERENT ]
- kernel.yama.ptrace_scope (exp: 1 2 3) [ OK ]
- net.ipv4.conf.all.accept_redirects (exp: 0) [ DIFFERENT ]
- net.ipv4.conf.all.accept_source_route (exp: 0) [ OK ]
- net.ipv4.conf.all.bootp_relay (exp: 0) [ OK ]
- net.ipv4.conf.all.forwarding (exp: 0) [ OK ]
- net.ipv4.conf.all.log_martians (exp: 1) [ DIFFERENT ]
- net.ipv4.conf.all.mc_forwarding (exp: 0) [ OK ]
- net.ipv4.conf.all.proxy_arp (exp: 0) [ OK ]
- net.ipv4.conf.all.rp_filter (exp: 1) [ OK ]
- net.ipv4.conf.all.send_redirects (exp: 0) [ DIFFERENT ]
- net.ipv4.conf.default.accept_redirects (exp: 0) [ DIFFERENT ]
- net.ipv4.conf.default.accept_source_route (exp: 0) [ DIFFERENT ]
- net.ipv4.conf.default.log_martians (exp: 1) [ DIFFERENT ]
- net.ipv4.icmp_echo_ignore_broadcasts (exp: 1) [ OK ]
- net.ipv4.icmp_ignore_bogus_error_responses (exp: 1) [ OK ]
- net.ipv4.tcp_syncookies (exp: 1) [ OK ]
- net.ipv4.tcp_timestamps (exp: 0 1) [ OK ]
- net.ipv6.conf.all.accept_redirects (exp: 0) [ DIFFERENT ]
- net.ipv6.conf.all.accept_source_route (exp: 0) [ OK ]
- net.ipv6.conf.default.accept_redirects (exp: 0) [ DIFFERENT ]
- net.ipv6.conf.default.accept_source_route (exp: 0) [ OK ]

[+] Hardening
------------------------------------
- Installed compiler(s) [ NOT FOUND ]
- Installed malware scanner [ NOT FOUND ]

[+] Custom Tests
------------------------------------
- Running custom tests... [ NONE ]

[+] Plugins (phase 2)
------------------------------------

================================================================================

-[ Lynis 2.7.5 Results ]-

Warnings (2):
----------------------------
! Found one or more vulnerable packages. [PKGS-7392]
https://cisofy.com/lynis/controls/PKGS-7392/

! iptables module(s) loaded, but no rules active [FIRE-4512]
https://cisofy.com/lynis/controls/FIRE-4512/

Suggestions (35):
----------------------------
* Set a password on GRUB bootloader to prevent altering boot configuration (e.g. boot in single user mode without password) [BOOT-5122]
https://cisofy.com/lynis/controls/BOOT-5122/

* Install a PAM module for password strength testing like pam_cracklib or pam_passwdqc [AUTH-9262]
https://cisofy.com/lynis/controls/AUTH-9262/

* Configure minimum password age in /etc/login.defs [AUTH-9286]
https://cisofy.com/lynis/controls/AUTH-9286/

* Configure maximum password age in /etc/login.defs [AUTH-9286]
https://cisofy.com/lynis/controls/AUTH-9286/

* Default umask in /etc/login.defs could be more strict like 027 [AUTH-9328]
https://cisofy.com/lynis/controls/AUTH-9328/

* To decrease the impact of a full /home file system, place /home on a separate partition [FILE-6310]
https://cisofy.com/lynis/controls/FILE-6310/

* To decrease the impact of a full /tmp file system, place /tmp on a separate partition [FILE-6310]
https://cisofy.com/lynis/controls/FILE-6310/

* To decrease the impact of a full /var file system, place /var on a separate partition [FILE-6310]
https://cisofy.com/lynis/controls/FILE-6310/

* Disable drivers like USB storage when not used, to prevent unauthorized storage or data theft [STRG-1840]
https://cisofy.com/lynis/controls/STRG-1840/

* Add the IP name and FQDN to /etc/hosts for proper name resolving [NAME-4404]
https://cisofy.com/lynis/controls/NAME-4404/

* Purge old/removed packages (1 found) with aptitude purge or dpkg --purge command. This will cleanup old configuration files, cron jobs and startup scripts. [PKGS-7346]
https://cisofy.com/lynis/controls/PKGS-7346/

* Install debsums utility for the verification of packages with known good database. [PKGS-7370]
https://cisofy.com/lynis/controls/PKGS-7370/

* Update your system with apt-get update, apt-get upgrade, apt-get dist-upgrade and/or unattended-upgrades [PKGS-7392]
https://cisofy.com/lynis/controls/PKGS-7392/

* Install package apt-show-versions for patch management purposes [PKGS-7394]
https://cisofy.com/lynis/controls/PKGS-7394/

* Consider running ARP monitoring software (arpwatch,arpon) [NETW-3032]
https://cisofy.com/lynis/controls/NETW-3032/

* Consider hardening SSH configuration [SSH-7408]
- Details : AllowTcpForwarding (YES --> NO)
https://cisofy.com/lynis/controls/SSH-7408/

* Consider hardening SSH configuration [SSH-7408]
- Details : ClientAliveCountMax (3 --> 2)
https://cisofy.com/lynis/controls/SSH-7408/

* Consider hardening SSH configuration [SSH-7408]
- Details : Compression (YES --> NO)
https://cisofy.com/lynis/controls/SSH-7408/

* Consider hardening SSH configuration [SSH-7408]
- Details : LogLevel (INFO --> VERBOSE)
https://cisofy.com/lynis/controls/SSH-7408/

* Consider hardening SSH configuration [SSH-7408]
- Details : MaxAuthTries (6 --> 3)
https://cisofy.com/lynis/controls/SSH-7408/

* Consider hardening SSH configuration [SSH-7408]
- Details : MaxSessions (10 --> 2)
https://cisofy.com/lynis/controls/SSH-7408/

* Consider hardening SSH configuration [SSH-7408]
- Details : Port (22 --> )
https://cisofy.com/lynis/controls/SSH-7408/

* Consider hardening SSH configuration [SSH-7408]
- Details : TCPKeepAlive (YES --> NO)
https://cisofy.com/lynis/controls/SSH-7408/

* Consider hardening SSH configuration [SSH-7408]
- Details : X11Forwarding (YES --> NO)
https://cisofy.com/lynis/controls/SSH-7408/

* Consider hardening SSH configuration [SSH-7408]
- Details : AllowAgentForwarding (YES --> NO)
https://cisofy.com/lynis/controls/SSH-7408/

* Check what deleted files are still in use and why. [LOGG-2190]
https://cisofy.com/lynis/controls/LOGG-2190/

* Add a legal banner to /etc/issue, to warn unauthorized users [BANN-7126]
https://cisofy.com/lynis/controls/BANN-7126/

* Add legal banner to /etc/issue.net, to warn unauthorized users [BANN-7130]
https://cisofy.com/lynis/controls/BANN-7130/

* Enable process accounting [ACCT-9622]
https://cisofy.com/lynis/controls/ACCT-9622/

* Enable sysstat to collect accounting (no results) [ACCT-9626]
https://cisofy.com/lynis/controls/ACCT-9626/

* Enable auditd to collect audit information [ACCT-9628]
https://cisofy.com/lynis/controls/ACCT-9628/

* Install a file integrity tool to monitor changes to critical and sensitive files [FINT-4350]
https://cisofy.com/lynis/controls/FINT-4350/

* Determine if automation tools are present for system management [TOOL-5002]
https://cisofy.com/lynis/controls/TOOL-5002/

* One or more sysctl values differ from the scan profile and could be tweaked [KRNL-6000]
- Solution : Change sysctl value or disable test (skip-test=KRNL-6000:)
https://cisofy.com/lynis/controls/KRNL-6000/

* Harden the system by installing at least one malware scanner, to perform periodic file system scans [HRDN-7230]
- Solution : Install a tool like rkhunter, chkrootkit, OSSEC
https://cisofy.com/lynis/controls/HRDN-7230/

Follow-up:
----------------------------
- Show details of a test (lynis show details TEST-ID)
- Check the logfile for all details (less /var/log/lynis.log)
- Read security controls texts (https://cisofy.com)
- Use --upload to upload data to central system (Lynis Enterprise users)

================================================================================

Lynis security scan details:

Hardening index : 63 [############ ]
Tests performed : 220
Plugins enabled : 0

Components:
- Firewall [V]
- Malware scanner [X]

Lynis modules:
- Compliance status [?]
- Security audit [V]
- Vulnerability scan [V]

Files:
- Test and debug information : /var/log/lynis.log
- Report data : /var/log/lynis-report.dat

================================================================================

Lynis 2.7.5

Auditing, system hardening, and compliance for UNIX-based systems
(Linux, macOS, BSD, and others)

2007-2019, CISOfy - https://cisofy.com/lynis/
Enterprise support available (compliance, plugins, interface and tools)

================================================================================

[TIP]: Enhance Lynis audits by adding your settings to custom.prf (see /etc/lynis/default.prf for all settings)

The hardening score of 63 is not that bad. However, depending on the scenario, it may be better to use a hardened image.

So, let's run the Lynis audit on a CIS hardened image. A CIS hardened Ubuntu 18.04 based VM is created in Azure. The above steps are run again on the new VM. As before, the vulnerabilities are marked below in bold are marked in the report below.

[ Lynis 2.7.5 ]

################################################################################
Lynis comes with ABSOLUTELY NO WARRANTY. This is free software, and you are
welcome to redistribute it under the terms of the GNU General Public License.
See the LICENSE file for details about using this software.

2007-2019, CISOfy - https://cisofy.com/lynis/
Enterprise support available (compliance, plugins, interface and tools)
################################################################################

[+] Initializing program
------------------------------------
- Detecting OS... [ DONE ]
- Checking profiles... [ DONE ]

---------------------------------------------------
Program version: 2.7.5
Operating system: Linux
Operating system name: Ubuntu Linux
Operating system version: 18.04
Kernel version: 4.18.0
Hardware platform: x86_64
Hostname: cisubuntuvm
---------------------------------------------------
Profiles: /etc/lynis/default.prf
Log file: /var/log/lynis.log
Report file: /var/log/lynis-report.dat
Report version: 1.0
Plugin directory: /usr/share/lynis/plugins
---------------------------------------------------
Auditor: [Not Specified]
Language: en
Test category: all
Test group: all
---------------------------------------------------
- Program update status... [ NO UPDATE ]

[+] System Tools
------------------------------------
- Scanning available tools...
- Checking system binaries...

[+] Plugins (phase 1)
------------------------------------
Note: plugins have more extensive tests and may take several minutes to complete

- Plugins enabled [ NONE ]

[+] Boot and services
------------------------------------
- Service Manager [ systemd ]
- Checking UEFI boot [ DISABLED ]
- Checking presence GRUB [ OK ]
- Checking presence GRUB2 [ FOUND ]
- Checking for password protection [ NONE ]
- Check running services (systemctl) [ DONE ]
Result: found 24 running services
- Check enabled services at boot (systemctl) [ DONE ]
Result: found 50 enabled services
- Check startup files (permissions) [ OK ]

[+] Kernel
------------------------------------
- Checking default run level [ RUNLEVEL 5 ]
- Checking CPU support (NX/PAE)
CPU support: PAE and/or NoeXecute supported [ FOUND ]
- Checking kernel version and release [ DONE ]
- Checking kernel type [ DONE ]
- Checking loaded kernel modules [ DONE ]
Found 60 active modules
- Checking Linux kernel configuration file [ FOUND ]
- Checking default I/O kernel scheduler [ FOUND ]
- Checking for available kernel update [ OK ]
- Checking core dumps configuration [ DISABLED ]
- Checking setuid core dumps configuration [ DEFAULT ]
- Check if reboot is needed [ NO ]

[+] Memory and Processes
------------------------------------
- Checking /proc/meminfo [ FOUND ]
- Searching for dead/zombie processes [ OK ]
- Searching for IO waiting processes [ OK ]

[+] Users, Groups and Authentication
------------------------------------
- Administrator accounts [ OK ]
- Unique UIDs [ OK ]
- Consistency of group files (grpck) [ OK ]
- Unique group IDs [ OK ]
- Unique group names [ OK ]
- Password file consistency [ OK ]
- Query system users (non daemons) [ DONE ]
- NIS+ authentication support [ NOT ENABLED ]
- NIS authentication support [ NOT ENABLED ]
- sudoers file [ FOUND ]
- Permissions for directory: /etc/sudoers.d [ OK ]
- Permissions for: /etc/sudoers [ OK ]
- Permissions for: /etc/sudoers.d/90-cloud-init-users [ OK ]
- Permissions for: /etc/sudoers.d/README [ OK ]
- PAM password strength tools [ OK ]
- PAM configuration files (pam.conf) [ FOUND ]
- PAM configuration files (pam.d) [ FOUND ]
- PAM modules [ FOUND ]
- LDAP module in PAM [ NOT FOUND ]
- Accounts without expire date [ OK ]
- Accounts without password [ OK ]
- User password aging (minimum) [ CONFIGURED ]
- User password aging (maximum) [ CONFIGURED ]
- Checking expired passwords [ OK ]
- Checking Linux single user mode authentication [ OK ]
- Determining default umask
- umask (/etc/profile and /etc/profile.d) [ OK ]
- umask (/etc/login.defs) [ SUGGESTION ]
- LDAP authentication support [ NOT ENABLED ]
- Logging failed login attempts [ ENABLED ]

[+] Shells
------------------------------------
- Checking shells from /etc/shells
Result: found 6 shells (valid shells: 6).
- Session timeout settings/tools [ NONE ]
- Checking default umask values
- Checking default umask in /etc/bashrc [ OK ]
- Checking default umask in /etc/bash.bashrc [ OK ]
- Checking default umask in /etc/profile [ OK ]

[+] File systems
------------------------------------
- Checking mount points
- Checking /home mount point [ SUGGESTION ]
- Checking /tmp mount point [ SUGGESTION ]
- Checking /var mount point [ SUGGESTION ]
- Query swap partitions (fstab) [ NONE ]
- Testing swap partitions [ OK ]
- Testing /proc mount (hidepid) [ SUGGESTION ]
- Checking for old files in /tmp [ OK ]
- Checking /tmp sticky bit [ OK ]
- Checking /var/tmp sticky bit [ OK ]
- ACL support root file system [ ENABLED ]
- Mount options of / [ NON DEFAULT ]
- Mount options of /dev/shm [ HARDENED ]
- Checking Locate database [ FOUND ]
- Disable kernel support of some filesystems
- Discovered kernel modules: udf

[+] USB Devices
------------------------------------
- Checking usb-storage driver (modprobe config) [ NOT DISABLED ]
- Checking USB devices authorization [ DISABLED ]
- Checking USBGuard [ NOT FOUND ]

[+] Storage
------------------------------------
- Checking firewire ohci driver (modprobe config) [ DISABLED ]

[+] NFS
------------------------------------
- Check running NFS daemon [ NOT FOUND ]

[+] Name services
------------------------------------
- Checking search domains [ FOUND ]
- Checking /etc/resolv.conf options [ FOUND ]
- Searching DNS domain name [ FOUND ]
Domain name: yev42i4zysyuzcn5celowc504d.bx.internal.cloudapp.net
- Checking /etc/hosts
- Checking /etc/hosts (duplicates) [ OK ]
- Checking /etc/hosts (hostname) [ SUGGESTION ]
- Checking /etc/hosts (localhost) [ OK ]
- Checking /etc/hosts (localhost to IP) [ OK ]

[+] Ports and packages
------------------------------------
- Searching package managers
- Searching dpkg package manager [ FOUND ]
- Querying package manager
- Query unpurged packages [ FOUND ]
- Checking security repository in sources.list file [ OK ]
- Checking APT package database [ OK ]
- Checking vulnerable packages [ WARNING ]
- Checking upgradeable packages [ SKIPPED ]
- Checking package audit tool [ INSTALLED ]
Found: apt-get
- Toolkit for automatic upgrades (unattended-upgrade) [ FOUND ]

[+] Networking
------------------------------------
- Checking IPv6 configuration [ ENABLED ]
Configuration method [ MANUAL ]
IPv6 only [ NO ]
- Checking configured nameservers
- Testing nameservers
Nameserver: 127.0.0.53 [ OK ]
- Checking default gateway [ DONE ]
- Getting listening ports (TCP/UDP) [ DONE ]
- Checking promiscuous interfaces [ OK ]
- Checking waiting connections [ OK ]
- Checking status DHCP client [ NOT ACTIVE ]
- Checking for ARP monitoring software [ NOT FOUND ]

[+] Printers and Spools
------------------------------------
- Checking cups daemon [ NOT FOUND ]
- Checking lp daemon [ NOT RUNNING ]

[+] Software: e-mail and messaging
------------------------------------
- Postfix status [ RUNNING ]
- Postfix configuration [ FOUND ]
- Postfix banner [ WARNING ]

[+] Software: firewalls
------------------------------------
- Checking iptables kernel module [ FOUND ]
- Checking iptables policies of chains [ FOUND ]
- Checking for empty ruleset [ OK ]
- Checking for unused rules [ FOUND ]
- Checking host based firewall [ ACTIVE ]

[+] Software: webserver
------------------------------------
- Checking Apache [ NOT FOUND ]
- Checking nginx [ NOT FOUND ]

[+] SSH Support
------------------------------------
- Checking running SSH daemon [ FOUND ]
- Searching SSH configuration [ FOUND ]
- SSH option: AllowTcpForwarding [ SUGGESTION ]
- SSH option: ClientAliveCountMax [ SUGGESTION ]
- SSH option: ClientAliveInterval [ OK ]
- SSH option: Compression [ SUGGESTION ]
- SSH option: FingerprintHash [ OK ]
- SSH option: GatewayPorts [ OK ]
- SSH option: IgnoreRhosts [ OK ]
- SSH option: LoginGraceTime [ OK ]
- SSH option: LogLevel [ SUGGESTION ]
- SSH option: MaxAuthTries [ SUGGESTION ]
- SSH option: MaxSessions [ SUGGESTION ]
- SSH option: PermitRootLogin [ OK ]
- SSH option: PermitUserEnvironment [ OK ]
- SSH option: PermitTunnel [ OK ]
- SSH option: Port [ SUGGESTION ]
- SSH option: PrintLastLog [ OK ]
- SSH option: StrictModes [ OK ]
- SSH option: TCPKeepAlive [ SUGGESTION ]
- SSH option: UseDNS [ OK ]
- SSH option: VerifyReverseMapping [ NOT FOUND ]
- SSH option: X11Forwarding [ OK ]
- SSH option: AllowAgentForwarding [ SUGGESTION ]
- SSH option: AllowUsers [ NOT FOUND ]
- SSH option: AllowGroups [ NOT FOUND ]

[+] SNMP Support
------------------------------------
- Checking running SNMP daemon [ NOT FOUND ]

[+] Databases
------------------------------------
No database engines found

[+] LDAP Services
------------------------------------
- Checking OpenLDAP instance [ NOT FOUND ]

[+] PHP
------------------------------------
- Checking PHP [ NOT FOUND ]

[+] Squid Support
------------------------------------
- Checking running Squid daemon [ NOT FOUND ]

[+] Logging and files
------------------------------------
- Checking for a running log daemon [ OK ]
- Checking Syslog-NG status [ NOT FOUND ]
- Checking systemd journal status [ FOUND ]
- Checking Metalog status [ NOT FOUND ]
- Checking RSyslog status [ FOUND ]
- Checking RFC 3195 daemon status [ NOT FOUND ]
- Checking minilogd instances [ NOT FOUND ]
- Checking logrotate presence [ OK ]
- Checking log directories (static list) [ DONE ]
- Checking open log files [ DONE ]
- Checking deleted files in use [ FILES FOUND ]

[+] Insecure services
------------------------------------
- Installed inetd package [ NOT FOUND ]
- Installed xinetd package [ OK ]
- xinetd status [ NOT ACTIVE ]
- Installed rsh client package [ OK ]
- Installed rsh server package [ OK ]
- Installed telnet client package [ OK ]
- Installed telnet server package [ NOT FOUND ]

[+] Banners and identification
------------------------------------
- /etc/issue [ FOUND ]
- /etc/issue contents [ WEAK ]
- /etc/issue.net [ FOUND ]
- /etc/issue.net contents [ WEAK ]

[+] Scheduled tasks
------------------------------------
- Checking crontab and cronjob files [ DONE ]
- Checking atd status [ RUNNING ]
- Checking at users [ DONE ]
- Checking at jobs [ NONE ]

[+] Accounting
------------------------------------
- Checking accounting information [ NOT FOUND ]
- Checking sysstat accounting data [ NOT FOUND ]
- Checking auditd [ NOT FOUND ]

[+] Time and Synchronization
------------------------------------
- NTP daemon found: systemd (timesyncd) [ FOUND ]
- Checking for a running NTP daemon or client [ OK ]

[+] Cryptography
------------------------------------
- Checking for expired SSL certificates [0/2] [ NONE ]

[+] Virtualization
------------------------------------

[+] Containers
------------------------------------

[+] Security frameworks
------------------------------------
- Checking presence AppArmor [ FOUND ]
- Checking AppArmor status [ ENABLED ]
- Checking presence SELinux [ NOT FOUND ]
- Checking presence TOMOYO Linux [ NOT FOUND ]
- Checking presence grsecurity [ NOT FOUND ]
- Checking for implemented MAC framework [ OK ]

[+] Software: file integrity
------------------------------------
- Checking file integrity tools
- AIDE [ FOUND ]
- AIDE config file [ FOUND ]
- AIDE config (Checksum) [ OK ]
- Checking presence integrity tool [ FOUND ]

[+] Software: System tooling
------------------------------------
- Checking automation tooling
- Automation tooling [ NOT FOUND ]
- Checking for IDS/IPS tooling [ NONE ]

[+] Software: Malware
------------------------------------

[+] File Permissions
------------------------------------
- Starting file permissions check
/root/.ssh [ OK ]

[+] Home directories
------------------------------------
- Checking shell history files [ OK ]

[+] Kernel Hardening
------------------------------------
- Comparing sysctl key pairs with scan profile
- fs.protected_hardlinks (exp: 1) [ OK ]
- fs.protected_symlinks (exp: 1) [ OK ]
- fs.suid_dumpable (exp: 0) [ OK ]
- kernel.core_uses_pid (exp: 1) [ DIFFERENT ]
- kernel.ctrl-alt-del (exp: 0) [ OK ]
- kernel.dmesg_restrict (exp: 1) [ DIFFERENT ]
- kernel.kptr_restrict (exp: 2) [ DIFFERENT ]
- kernel.randomize_va_space (exp: 2) [ OK ]
- kernel.sysrq (exp: 0) [ DIFFERENT ]
- kernel.yama.ptrace_scope (exp: 1 2 3) [ OK ]
- net.ipv4.conf.all.accept_redirects (exp: 0) [ OK ]
- net.ipv4.conf.all.accept_source_route (exp: 0) [ OK ]
- net.ipv4.conf.all.bootp_relay (exp: 0) [ OK ]
- net.ipv4.conf.all.forwarding (exp: 0) [ OK ]
- net.ipv4.conf.all.log_martians (exp: 1) [ OK ]
- net.ipv4.conf.all.mc_forwarding (exp: 0) [ OK ]
- net.ipv4.conf.all.proxy_arp (exp: 0) [ OK ]
- net.ipv4.conf.all.rp_filter (exp: 1) [ OK ]
- net.ipv4.conf.all.send_redirects (exp: 0) [ OK ]
- net.ipv4.conf.default.accept_redirects (exp: 0) [ OK ]
- net.ipv4.conf.default.accept_source_route (exp: 0) [ OK ]
- net.ipv4.conf.default.log_martians (exp: 1) [ OK ]
- net.ipv4.icmp_echo_ignore_broadcasts (exp: 1) [ OK ]
- net.ipv4.icmp_ignore_bogus_error_responses (exp: 1) [ OK ]
- net.ipv4.tcp_syncookies (exp: 1) [ OK ]
- net.ipv4.tcp_timestamps (exp: 0 1) [ OK ]
- net.ipv6.conf.all.accept_redirects (exp: 0) [ OK ]
- net.ipv6.conf.all.accept_source_route (exp: 0) [ OK ]
- net.ipv6.conf.default.accept_redirects (exp: 0) [ OK ]
- net.ipv6.conf.default.accept_source_route (exp: 0) [ OK ]

[+] Hardening
------------------------------------
- Installed compiler(s) [ FOUND ]
- Installed malware scanner [ NOT FOUND ]

[+] Custom Tests
------------------------------------
- Running custom tests... [ NONE ]

[+] Plugins (phase 2)
------------------------------------

================================================================================

-[ Lynis 2.7.5 Results ]-

Warnings (2):
----------------------------
! Found one or more vulnerable packages. [PKGS-7392]
https://cisofy.com/lynis/controls/PKGS-7392/

! Found some information disclosure in SMTP banner (OS or software name) [MAIL-8818]
https://cisofy.com/lynis/controls/MAIL-8818/

Suggestions (33):
----------------------------
* Set a password on GRUB bootloader to prevent altering boot configuration (e.g. boot in single user mode without password) [BOOT-5122]
https://cisofy.com/lynis/controls/BOOT-5122/

* Default umask in /etc/login.defs could be more strict like 027 [AUTH-9328]
https://cisofy.com/lynis/controls/AUTH-9328/

* To decrease the impact of a full /home file system, place /home on a separate partition [FILE-6310]
https://cisofy.com/lynis/controls/FILE-6310/

* To decrease the impact of a full /tmp file system, place /tmp on a separate partition [FILE-6310]
https://cisofy.com/lynis/controls/FILE-6310/

* To decrease the impact of a full /var file system, place /var on a separate partition [FILE-6310]
https://cisofy.com/lynis/controls/FILE-6310/

* Disable drivers like USB storage when not used, to prevent unauthorized storage or data theft [STRG-1840]
https://cisofy.com/lynis/controls/STRG-1840/

* Add the IP name and FQDN to /etc/hosts for proper name resolving [NAME-4404]
https://cisofy.com/lynis/controls/NAME-4404/

* Purge old/removed packages (2 found) with aptitude purge or dpkg --purge command. This will cleanup old configuration files, cron jobs and startup scripts. [PKGS-7346]
https://cisofy.com/lynis/controls/PKGS-7346/

* Install debsums utility for the verification of packages with known good database. [PKGS-7370]
https://cisofy.com/lynis/controls/PKGS-7370/

* Update your system with apt-get update, apt-get upgrade, apt-get dist-upgrade and/or unattended-upgrades [PKGS-7392]
https://cisofy.com/lynis/controls/PKGS-7392/

* Install package apt-show-versions for patch management purposes [PKGS-7394]
https://cisofy.com/lynis/controls/PKGS-7394/

* Consider running ARP monitoring software (arpwatch,arpon) [NETW-3032]
https://cisofy.com/lynis/controls/NETW-3032/

* You are advised to hide the mail_name (option: smtpd_banner) from your postfix configuration. Use postconf -e or change your main.cf file (/etc/postfix/main.cf) [MAIL-8818]
https://cisofy.com/lynis/controls/MAIL-8818/

* Check iptables rules to see which rules are currently not used [FIRE-4513]
https://cisofy.com/lynis/controls/FIRE-4513/

* Consider hardening SSH configuration [SSH-7408]
- Details : AllowTcpForwarding (YES --> NO)
https://cisofy.com/lynis/controls/SSH-7408/

* Consider hardening SSH configuration [SSH-7408]
- Details : ClientAliveCountMax (3 --> 2)
https://cisofy.com/lynis/controls/SSH-7408/

* Consider hardening SSH configuration [SSH-7408]
- Details : Compression (YES --> NO)
https://cisofy.com/lynis/controls/SSH-7408/

* Consider hardening SSH configuration [SSH-7408]
- Details : LogLevel (INFO --> VERBOSE)
https://cisofy.com/lynis/controls/SSH-7408/

* Consider hardening SSH configuration [SSH-7408]
- Details : MaxAuthTries (4 --> 3)
https://cisofy.com/lynis/controls/SSH-7408/

* Consider hardening SSH configuration [SSH-7408]
- Details : MaxSessions (10 --> 2)
https://cisofy.com/lynis/controls/SSH-7408/

* Consider hardening SSH configuration [SSH-7408]
- Details : Port (22 --> )
https://cisofy.com/lynis/controls/SSH-7408/

* Consider hardening SSH configuration [SSH-7408]
- Details : TCPKeepAlive (YES --> NO)
https://cisofy.com/lynis/controls/SSH-7408/

* Consider hardening SSH configuration [SSH-7408]
- Details : AllowAgentForwarding (YES --> NO)
https://cisofy.com/lynis/controls/SSH-7408/

* Check what deleted files are still in use and why. [LOGG-2190]
https://cisofy.com/lynis/controls/LOGG-2190/

* Add a legal banner to /etc/issue, to warn unauthorized users [BANN-7126]
https://cisofy.com/lynis/controls/BANN-7126/

* Add legal banner to /etc/issue.net, to warn unauthorized users [BANN-7130]
https://cisofy.com/lynis/controls/BANN-7130/

* Enable process accounting [ACCT-9622]
https://cisofy.com/lynis/controls/ACCT-9622/

* Enable sysstat to collect accounting (no results) [ACCT-9626]
https://cisofy.com/lynis/controls/ACCT-9626/

* Enable auditd to collect audit information [ACCT-9628]
https://cisofy.com/lynis/controls/ACCT-9628/

* Determine if automation tools are present for system management [TOOL-5002]
https://cisofy.com/lynis/controls/TOOL-5002/

* One or more sysctl values differ from the scan profile and could be tweaked [KRNL-6000]
- Solution : Change sysctl value or disable test (skip-test=KRNL-6000:)
https://cisofy.com/lynis/controls/KRNL-6000/

* Harden compilers like restricting access to root user only [HRDN-7222]
https://cisofy.com/lynis/controls/HRDN-7222/

* Harden the system by installing at least one malware scanner, to perform periodic file system scans [HRDN-7230]
- Solution : Install a tool like rkhunter, chkrootkit, OSSEC
https://cisofy.com/lynis/controls/HRDN-7230/

Follow-up:
----------------------------
- Show details of a test (lynis show details TEST-ID)
- Check the logfile for all details (less /var/log/lynis.log)
- Read security controls texts (https://cisofy.com)
- Use --upload to upload data to central system (Lynis Enterprise users)

================================================================================

Lynis security scan details:

Hardening index : 70 [############## ]
Tests performed : 226
Plugins enabled : 0

Components:
- Firewall [V]
- Malware scanner [X]

Lynis modules:
- Compliance status [?]
- Security audit [V]
- Vulnerability scan [V]

Files:
- Test and debug information : /var/log/lynis.log
- Report data : /var/log/lynis-report.dat

================================================================================

Lynis 2.7.5

Auditing, system hardening, and compliance for UNIX-based systems
(Linux, macOS, BSD, and others)

2007-2019, CISOfy - https://cisofy.com/lynis/
Enterprise support available (compliance, plugins, interface and tools)

================================================================================

[TIP]: Enhance Lynis audits by adding your settings to custom.prf (see /etc/lynis/default.prf for all settings)

One of the more popular Linux distributions being used is Alpine Linux. Let's run one last test on this distribution. At the moment Alpine Linux is not available in Azure marketplace. Install Alpine Linux v3.10.2 locally on Oracle Virtual Box and run Lynis tool on it. The report is for Alpine Linux is below with the vulnerabilities marked in bold.

# Lynis Report
report_version_major=1
report_version_minor=0
report_datetime_start=2019-08-22 12:00:49
auditor=[Not Specified]
lynis_version=2.7.5
os=Linux
os_name=Linux
os_fullname=Linux 4.19.67-0-virt
os_version=4.19.67-0-virt
linux_version=
os_kernel_version=4.19.67
os_kernel_version_full=4.19.67-0-virt
hostname=alpine-vm
test_category=all
test_group=all
plugin_directory=/usr/local/lynis/plugins
lynis_update_available=-1
binaries_count=450
binary_paths=/bin,/sbin,/usr/bin,/usr/sbin,/usr/local/bin
vm=2
container=0
systemd=0
plugins_enabled=0
hostid=1f160923357b2308cc429fae4f6f3afd66120941
hostid2=1bdf03c6999546b96583ecf72515aee796d4e9838eead4d149cb282aa1daff7e
uptime_in_seconds=169
uptime_in_days=0
boot_loader=unknown
boot_uefi_booted=0
boot_uefi_booted_secure=0
service_manager=busybox
cpu_pae=1
cpu_nx=1
linux_kernel_release=4.19.67-0-virt
linux_kernel_version=#1-Alpine SMP Mon Aug 19 10:05:34 UTC 2019
linux_kernel_type=modular
loaded_kernel_module[]=ac
loaded_kernel_module[]=aes_x86_64
loaded_kernel_module[]=aesni_intel
loaded_kernel_module[]=af_packet
loaded_kernel_module[]=ata_generic
loaded_kernel_module[]=battery
loaded_kernel_module[]=button
loaded_kernel_module[]=cdrom
loaded_kernel_module[]=cfbcopyarea
loaded_kernel_module[]=cfbfillrect
loaded_kernel_module[]=cfbimgblt
loaded_kernel_module[]=crc16
loaded_kernel_module[]=crc32_pclmul
loaded_kernel_module[]=crc32c_intel
loaded_kernel_module[]=crct10dif_pclmul
loaded_kernel_module[]=cryptd
loaded_kernel_module[]=crypto_simd
loaded_kernel_module[]=drm
loaded_kernel_module[]=drm_kms_helper
loaded_kernel_module[]=drm_panel_orientation_quirks
loaded_kernel_module[]=e1000
loaded_kernel_module[]=ehci_hcd
loaded_kernel_module[]=ehci_pci
loaded_kernel_module[]=evdev
loaded_kernel_module[]=ext4
loaded_kernel_module[]=fb
loaded_kernel_module[]=fb_sys_fops
loaded_kernel_module[]=fbdev
loaded_kernel_module[]=font
loaded_kernel_module[]=ghash_clmulni_intel
loaded_kernel_module[]=glue_helper
loaded_kernel_module[]=hid
loaded_kernel_module[]=hid_generic
loaded_kernel_module[]=i2c_core
loaded_kernel_module[]=i2c_piix4
loaded_kernel_module[]=ipv6
loaded_kernel_module[]=jbd2
loaded_kernel_module[]=loop
loaded_kernel_module[]=mbcache
loaded_kernel_module[]=mousedev
loaded_kernel_module[]=ohci_hcd
loaded_kernel_module[]=ohci_pci
loaded_kernel_module[]=pcbc
loaded_kernel_module[]=psmouse
loaded_kernel_module[]=sd_mod
loaded_kernel_module[]=sr_mod
loaded_kernel_module[]=syscopyarea
loaded_kernel_module[]=sysfillrect
loaded_kernel_module[]=sysimgblt
loaded_kernel_module[]=ttm
loaded_kernel_module[]=usb_common
loaded_kernel_module[]=usb_storage
loaded_kernel_module[]=usbcore
loaded_kernel_module[]=usbhid
loaded_kernel_module[]=vmwgfx
exception_event[]=KRNL-5830:3|Could not find our running kernel on disk, which is unexpected|
memory_size=1015980
memory_units=kB
auth_group_ids_unique=1
auth_group_names_unique=1
real_user[]=root,0
real_user[]=admin,1234
suggestion[]=AUTH-9262|Install a PAM module for password strength testing like pam_cracklib or pam_passwdqc|-|-|
warning[]=AUTH-9308|No password set for single mode|-|-|
suggestion[]=AUTH-9308|Set password for single user mode to minimize physical access attack surface|-|-|
suggestion[]=AUTH-9328|Default umask in /etc/profile or /etc/profile.d/custom.sh could be more strict (e.g. 027)|-|-|
manual_event[]=AUTH-9328:03
auth_failed_logins_logged=0
ldap_auth_enabled=0
ldap_pam_enabled=0
password_min_days=-1
password_max_days=-1
available_shell[]=/bin/sh
available_shell[]=/bin/ash
session_timeout_enabled=0
suggestion[]=FILE-6310|To decrease the impact of a full /home file system, place /home on a separate partition|-|-|
suggestion[]=FILE-6310|To decrease the impact of a full /tmp file system, place /tmp on a separate partition|-|-|
suggestion[]=FILE-6310|To decrease the impact of a full /var file system, place /var on a separate partition|-|-|
swap_partition[]=UUID=0d548975-79df-430d-b61c-648f5460e997,UUID=0d548975-79df-430d-b61c-648f5460e997,
suggestion[]=STRG-1840|Disable drivers like USB storage when not used, to prevent unauthorized storage or data theft|-|-|
usb_authorized_default_device[]=/sys/bus/usb/devices/usb1
domainname=my.domain
localhost-mapped-to=::1
name_cache_used=0
suggestion[]=PKGS-7398|Install a package audit tool to determine vulnerable packages|-|-|
installed_kernel_packages=0
unattended_upgrade_option_available=0
ipv6_mode=auto
ipv6_only=0
nameserver[]=192.168.1.1
default_gateway[]=10.0.2.2
network_interface[]=lo
network_interface[]=eth0
network_mac_address[]=08:00:27:60:44:C5
network_ipv4_address[]=10.0.2.15
network_ipv4_address[]=127.0.0.1
network_ipv6_address[]=fe80::a00:27ff:fe60:44c5/64
network_ipv6_address[]=::1/128
network_listen_port[]=127.0.0.1:323|udp|chronyd|
network_listen_port[]=::1:323|udp|chronyd|
network_listen_port[]=0.0.0.0:22|tcp|sshd|
network_listen_port[]=:::22|tcp|sshd|
suggestion[]=NETW-3032|Consider running ARP monitoring software (arpwatch,arpon)|-|-|
imap_daemon=
pop3_daemon=
smtp_daemon=
suggestion[]=FIRE-4590|Configure a firewall/packet filter to filter incoming and outgoing traffic|-|-|
ssh_daemon_running=0
warning[]=LOGG-2138|klogd is not running, which could lead to missing kernel messages in log files|-|-|
log_directory[]=/var/log
log_rotation_config_found=1
log_rotation_tool=logrotate
exception_event[]=PackageIsInstalled:01||
exception_event[]=PackageIsInstalled:01||
exception_event[]=PackageIsInstalled:01||
exception_event[]=PackageIsInstalled:01||
exception_event[]=PackageIsInstalled:01||
exception_event[]=PackageIsInstalled:01||
exception_event[]=PackageIsInstalled:01||
exception_event[]=PackageIsInstalled:01||
exception_event[]=PackageIsInstalled:01||
exception_event[]=PackageIsInstalled:01||
suggestion[]=BANN-7126|Add a legal banner to /etc/issue, to warn unauthorized users|-|-|
weak_banner_file[]=/etc/issue
crond_running=1
scheduler[]=crond
suggestion[]=ACCT-9622|Enable process accounting|-|-|
suggestion[]=ACCT-9626|Enable sysstat to collect accounting (no results)|-|-|
suggestion[]=ACCT-9628|Enable auditd to collect audit information|-|-|
linux_auditd_running=0
audit_daemon_running=0
suggestion[]=TIME-3104|Use NTP daemon or NTP client to prevent time issues.|-|-|
tz_variable_empty=1
ntp_config_found=0
ntp_config_type_daemon=0
ntp_config_type_eventbased=0
ntp_config_type_scheduled=0
ntp_config_type_startup=0
ntp_daemon=
ntp_daemon_running=0
certificate[]=/etc/ssl/cert.pem|0|cn:subject=CN = ACCVRAIZ1, OU = PKIACCV, O = ACCV, C = ES;notafter:Dec 31 09:37:37 2030 GMT;|
certificate[]=/etc/ssl/certs/ca-certificates.crt|0|cn:subject=CN = ACCVRAIZ1, OU = PKIACCV, O = ACCV, C = ES;notafter:Dec 31 09:37:37 2030 GMT;|
certificates=2
framework_grsecurity=0
framework_selinux=0
suggestion[]=FINT-4350|Install a file integrity tool to monitor changes to critical and sensitive files|-|-|
suggestion[]=TOOL-5002|Determine if automation tools are present for system management|-|-|
automation_tool_present=0
malware_scanner_installed=0
home_directory[]=/
home_directory[]=/bin
home_directory[]=/home/admin
home_directory[]=/root
home_directory[]=/sbin
home_directory[]=/var/empty
home_directory[]=/var/log/chrony
home_directory[]=/var/spool/cron
details[]=KRNL-6000|sysctl|desc:No description;field:kernel.core_uses_pid;prefval:1;value:0;|
details[]=KRNL-6000|sysctl|desc:Restrict use of dmesg;field:kernel.dmesg_restrict;prefval:1;value:0;|
details[]=KRNL-6000|sysctl|desc:Restrict access to kernel symbols;field:kernel.kptr_restrict;prefval:2;value:0;|
details[]=KRNL-6000|sysctl|desc:Disable magic SysRQ;field:kernel.sysrq;prefval:0;value:1;|
details[]=KRNL-6000|sysctl|desc:Log all packages for which the host does not have a path back to the source;field:net.ipv4.conf.all.log_martians;prefval:1;value:0;|
details[]=KRNL-6000|sysctl|desc:Disable/Ignore ICMP routing redirects;field:net.ipv4.conf.all.send_redirects;prefval:0;value:1;|
details[]=KRNL-6000|sysctl|desc:Disable/Ignore ICMP routing redirects;field:net.ipv4.conf.default.accept_redirects;prefval:0;value:1;|
details[]=KRNL-6000|sysctl|desc:Disable IP source routing;field:net.ipv4.conf.default.accept_source_route;prefval:0;value:1;|
details[]=KRNL-6000|sysctl|desc:Log all packages for which the host does not have a path back to the source;field:net.ipv4.conf.default.log_martians;prefval:1;value:0;|
details[]=KRNL-6000|sysctl|desc:Disable/Ignore ICMP routing redirects;field:net.ipv6.conf.default.accept_redirects;prefval:0;value:1;|
suggestion[]=KRNL-6000|One or more sysctl values differ from the scan profile and could be tweaked||Change sysctl value or disable test (skip-test=KRNL-6000:)|
suggestion[]=HRDN-7230|Harden the system by installing at least one malware scanner, to perform periodic file system scans|-|Install a tool like rkhunter, chkrootkit, OSSEC|
compiler_installed=0
lynis_tests_done=189
report_datetime_end=2019-08-22 12:00:52
dhcp_client_running=1
arpwatch_running=0
firewall_active=0
firewall_empty_ruleset=0
firewall_installed=0
package_audit_tool=
package_audit_tool_found=0
vulnerable_packages_found=0
hardening_index=61
tests_executed=HRDN-7230|HRDN-7222|HRDN-7220|KRNL-6000|HOME-9350|HOME-9310|HOME-9302|FILE-7524|MALW-3284|MALW-3282|MALW-3280|MALW-3278|MALW-3276|MALW-3275|TOOL-5190|TOOL-5126|TOOL-5122|TOOL-5120|TOOL-5102|TOOL-5002|FINT-4350|FINT-4338|FINT-4330|FINT-4328|FINT-4326|FINT-4322|FINT-4318|FINT-4314|FINT-4310|MACF-6290|RBAC-6272|MACF-6240|MACF-6232|MACF-6204|CONT-8102|CRYP-7902|TIME-3170|TIME-3148|TIME-3104|ACCT-9636|ACCT-9628|ACCT-9626|ACCT-9622|SCHD-7718|SCHD-7704|SCHD-7702|BANN-7128|BANN-7126|BANN-7124|INSE-8322|INSE-8310|INSE-8342|INSE-8300|INSE-8102|INSE-8100|INSE-8000|LOGG-2192|LOGG-2190|LOGG-2180|LOGG-2170|LOGG-2154|LOGG-2146|LOGG-2142|LOGG-2138|LOGG-2240|LOGG-2230|LOGG-2210|LOGG-2136|LOGG-2132|LOGG-2130|SQD-3602|PHP-2211|LDAP-2219|DBS-1880|DBS-1860|DBS-1840|DBS-1826|DBS-1820|DBS-1818|DBS-1804|SNMP-3302|SSH-7402|HTTP-6702|HTTP-6622|FIRE-4594|FIRE-4590|FIRE-4524|FIRE-4502|MAIL-8880|MAIL-8860|MAIL-8838|MAIL-8820|MAIL-8814|MAIL-8802|PRNT-2314|PRNT-2304|NETW-3032|NETW-3030|NETW-3028|NETW-3015|NETW-3012|NETW-3008|NETW-3006|NETW-3004|NETW-3001|NETW-2705|NETW-2704|NETW-2600|PKGS-7420|PKGS-7410|PKGS-7398|NAME-4408|NAME-4406|NAME-4404|NAME-4402|NAME-4304|NAME-4230|NAME-4202|NAME-4034|NAME-4032|NAME-4028|NAME-4020|NAME-4018|NAME-4016|STRG-1920|STRG-1846|USB-3000|STRG-1842|STRG-1840|FILE-6430|FILE-6376|FILE-6374|FILE-6372|FILE-6368|FILE-6363|FILE-6362|FILE-6354|FILE-6344|FILE-6336|FILE-6332|FILE-6329|FILE-6324|FILE-6323|FILE-6310|SHLL-6230|SHLL-6220|SHLL-6211|AUTH-9402|AUTH-9328|AUTH-9308|AUTH-9288|AUTH-9283|AUTH-9282|AUTH-9278|AUTH-9268|AUTH-9266|AUTH-9264|AUTH-9262|AUTH-9250|AUTH-9242|AUTH-9240|AUTH-9234|AUTH-9226|AUTH-9222|AUTH-9208|AUTH-9204|PROC-3614|PROC-3612|PROC-3602|KRNL-5830|KRNL-5820|KRNL-5728|KRNL-5726|KRNL-5723|KRNL-5695|KRNL-5677|KRNL-5622|BOOT-5260|BOOT-5202|BOOT-5184|BOOT-5177|BOOT-5155|BOOT-5142|BOOT-5139|BOOT-5121|BOOT-5116|BOOT-5108|BOOT-5104|CORE-1000|
tests_skipped=MALW-3288|MALW-3286|TOOL-5104|FINT-4402|FINT-4336|FINT-4334|FINT-4315|MACF-6242|MACF-6234|MACF-6208|CONT-8108|CONT-8107|CONT-8106|CONT-8104|CONT-8004|TIME-3160|TIME-3136|TIME-3132|TIME-3128|TIME-3124|TIME-3120|TIME-3116|TIME-3112|TIME-3106|ACCT-9662|ACCT-9660|ACCT-9656|ACCT-9654|ACCT-9652|ACCT-9650|ACCT-9634|ACCT-9632|ACCT-9630|ACCT-2760|ACCT-2754|SCHD-7724|SCHD-7720|BANN-7130|BANN-7113|INSE-8050|INSE-8200|INSE-8116|INSE-8106|INSE-8104|INSE-8016|INSE-8006|INSE-8004|INSE-8002|LOGG-2164|LOGG-2162|LOGG-2160|LOGG-2152|LOGG-2150|LOGG-2148|LOGG-2134|SQD-3680|SQD-3630|SQD-3624|SQD-3620|SQD-3616|SQD-3614|SQD-3613|SQD-3610|SQD-3606|SQD-3604|PHP-2378|PHP-2376|PHP-2374|PHP-2372|PHP-2368|PHP-2320|LDAP-2224|DBS-1888|DBS-1886|DBS-1884|DBS-1882|DBS-1816|SNMP-3306|SNMP-3304|SSH-7440|SSH-7408|SSH-7406|SSH-7404|HTTP-6720|HTTP-6716|HTTP-6714|HTTP-6712|HTTP-6710|HTTP-6708|HTTP-6706|HTTP-6704|HTTP-6643|HTTP-6641|HTTP-6640|HTTP-6632|HTTP-6626|HTTP-6624|FIRE-4586|FIRE-4540|FIRE-4538|FIRE-4536|FIRE-4534|FIRE-4532|FIRE-4530|FIRE-4526|FIRE-4520|FIRE-4518|FIRE-4513|FIRE-4512|FIRE-4508|MAIL-8920|MAIL-8818|MAIL-8817|MAIL-8816|MAIL-8803|PRNT-2420|PRNT-2418|PRNT-2316|PRNT-2308|PRNT-2307|PRNT-2306|PRNT-2302|NETW-3014|PKGS-7394|PKGS-7393|PKGS-7392|PKGS-7390|PKGS-7388|PKGS-7387|PKGS-7386|PKGS-7384|PKGS-7383|PKGS-7382|PKGS-7381|PKGS-7380|PKGS-7378|PKGS-7370|PKGS-7366|PKGS-7354|PKGS-7352|PKGS-7350|PKGS-7348|PKGS-7346|PKGS-7345|PKGS-7334|PKGS-7332|PKGS-7330|PKGS-7328|PKGS-7322|PKGS-7320|PKGS-7314|PKGS-7312|PKGS-7310|PKGS-7308|PKGS-7306|PKGS-7304|PKGS-7303|PKGS-7302|PKGS-7301|NAME-4306|NAME-4238|NAME-4236|NAME-4232|NAME-4210|NAME-4206|NAME-4204|NAME-4036|NAME-4026|NAME-4024|STRG-1930|STRG-1928|STRG-1926|STRG-1906|STRG-1904|STRG-1902|FILE-6410|FILE-6439|FILE-6330|FILE-6312|FILE-6311|SHLL-6202|AUTH-9410|AUTH-9409|AUTH-9408|AUTH-9406|AUTH-9340|AUTH-9306|AUTH-9304|AUTH-9286|AUTH-9254|AUTH-9252|AUTH-9228|AUTH-9489|AUTH-9218|AUTH-9216|AUTH-9212|PROC-3604|KRNL-5788|KRNL-5770|KRNL-5831|KRNL-5745|KRNL-5730|BOOT-5263|BOOT-5262|BOOT-5180|BOOT-5165|BOOT-5159|BOOT-5126|BOOT-5261|BOOT-5124|BOOT-5122|BOOT-5117|BOOT-5106|BOOT-5102|
finish=true

As is evident from the two Lynis reports above, the CIS hardened Ubuntu 18.04 image has a higher score (70) than the Canonical Ubuntu 18.04 score (63) and Alpine Linux score (61).

Depending on the scenario, a normal or a hardened Linux image could be used in a VM. You could perform the hardening of Linux by yourself or use a hardened image. However, the cost angle comes into play. The hardened images usually come at an additional cost.

Some of the scenarios where such audit tools could be used are -

As part of software development process when new infrastructure is setup
Help site reliability engineers ensure that any patches/upgrades made to an existing application/VM do not add new vulnerabilities
Use them as a quality gate in CI/CD process of infrastructure as code
Verify new VM's in blue-green deployments when new VM's are swapped-in for the older ones

Of course, Lynis audit results are not the only criteria for choosing a Linux distribution. Other factors such as risk vs effort/cost trade-off, compatibility with the application, performance etc. also need to be taken into account before taking a decision.

Either way, it's a good idea to run a tool like Lynis (or something similar) to audit the VM's in the cloud (or on-premise). Based on the output from the tool and the non-functional requirements, one can tweak the configuration of the VM and/or Azure.

Cloudera Quick Start VM in Hyper-V

2019-03-26T15:26:00.002+05:30

Overview

Marc Andreessen penned his now famous essay, “Why Software Is Eating the World” in Wall Street Journal in 2011. However, if we dig under the covers, it's really virtualization that is powering this phenomenon. Today, literally every hardware component such as - processor, memory, network, storage, network load-balancer, network router and switch, firewall are virtualized. In fact, even a user is now "virtualized" especially in scenarios such as load testing and synthetic monitoring. Finally, as a concept, money is also virtualized in the form of crypto-currencies.

In this post, let's go through how we can implement virtualization in a local machine and get better understanding of the concepts of virtualization.

Virtualization

One key aspect, driver and benefit of virtualization is sharing of the physical resources. Before we dive in, let's define some concepts.

Host Machine - This can be a server rack, single server or a desktop/laptop. In our case, it is a laptop. Typically, virtualization has to be configured in the hardware BIOS.
Host Resources - These are the "physical" resources in the host machine such as cpu, memory, network cards, disk, display. peripheral devices.
Host OS - The operating system of the host machine. In addition, virtualization software/drivers need to be installed int he Host OS too.
Guest VM - This is the "virtual machine" that will be created. So, one could create multiple VM's in a single physical machine.
Guest Resources - Each guest VM can have it's own set of resources. For example, it is possible to create a VM without a display attached to it.
Guest OS - In each VM, one can have different operating systems and applications.
Hypervisor - Type 1 or Type 2

Hypervisor

The key construct of virtualization is a hypervisor. Hypervisor is the component that allocates and manages the various physical resources among the virtual machines. There are broadly two types of hypervisors - Type 1 and Type 2.

Image Source: IBM

As seen in the figure above, Type 1 hypervisors (Hyper-V, VMWare ESX etc.) are more closely integrated with the host OS. Type 2 hypervisors (Oracle Virtual Box, VMWare Player etc.) run as an "application" on to of the Host OS. For running applications in "live" environment, Type 1 is typically used. Type 2 is more used in the "development" environment. However, now with Hyper-V from Microsoft, it is now possible to have a Type 1 hypervisor that can be used in a Windows desktop/laptop.

Exercise

In this example, the following are going to be used -

Host Machine - Dell Inspiron 15 7000 Gaming laptop.
Host Resources - The laptop has 8 logical cores (i7 Processor). 16 GB RAM, 1 TB Hard Disk and 1 Network Card.
Host OS - Windows 10 Professional.
Guest VM - 1 Guest VM.
Guest Resources - Guest VM has 4 cores, 10 GB RAM and virtual hard disk assigned to it.
Guest OS - Cent OS 6.
Hypervisor - Type 1 (Windows Hyper V)

Prerequisites

Ensure that the virtualization is enabled in the desktop/laptop BIOS.
Open the "Windows Features" option under "Control Panel". Enable Hyper-V option as shown below. Restart the desktop/laptop.

Setup

In this example, let's deliberately take a complex use case. We need a big data framework to be setup. Let's look at Cloudera distribution. Cloudera internally consists of various software components such as HDFS, HBase, Impala, Solr, Spark etc. It is possible to setup these components separately, but it is time consuming and error prone. Here comes virtualization to the rescue!

The steps are -

Download the VMWare image of Cloudera Quick VM. Copy it to a folder say "D:\Temp".
Install Oracle Virtual Box.
Open the Virtual Box folder in Command Prompt. Run this command to convert the image from VMWare to Hyper-V format - vboxmanage clonehd “D:\Temp\cloudera-quickstart-vm-5.13.0-0-vmware\cloudera-quickstart-vm-5.13.0-0-vmware.vmdk” “D:\Temp\cloudera-quickstart-vm-5.13.0-0-vmware.vhd" --format vhd
Open the Hyper-V Manager and create a new VM. Create a new external virtual switch and call it 'External Switch'.
Create a new VM with following settings -

Generation 1 VM
4 Cores
Min 8 GB, Max - 10 GB RAM (with dynamic allocation)
Instead of creating a new virtual hard disk, assign the virtual hard disk created above
Assign the 'External Switch'

Start the VM.
The user id and password for the VM and all services is cloudera/cloudera.
Depending on the amount of CPU and memory assigned to the VM, allow the various Cloudera services to start. It may take anywhere from 3-5 minutes.
Once all the services are up, the Cloudera instance can be used by connecting to the VM.

Tests

You can also get the IP address of the VM and open http://[VM IP Address]:7180 in the host machine browser to access Cloudera Manager.

This is a big data framework after all. Let's now see if we can ingest some sample data.

Open Hue from http://[VM IP Address]:8888
User ID/password is cloudera/cloudera.
Using Hue, one can create table in Impala by running this command - CREATE TABLE default.t1 (x INT, y STRING);
Next step is to insert multiple rows using - INSERT INTO default.t1 VALUES (1, 'one'), (2, 'two'), (3, 'three');
Finally, query the table using - SELECT * FROM default.t1;

Benefits

From a development perspective, here are the advantages of virtualization -

It saves the time of having to setup multiple components for a complex software.
A VM can be setup in minutes to run any kind of application
If, for some reason, the VM is corrupted, it can always be re-installed.
Each developer in the team can have their own full fledged Cloudera instance for developing, testing, troubleshooting etc.

Azure Chronicles - Scaling Mule

2019-02-17T15:05:00.000+05:30

Overview

In the previous post, we saw how to install and test Mule run time on a standalone VM in Azure. In this post, we will take it a step further to scale out the Mule run time.

There are several ways one could scale a Mule runtime. They are -

Anypoint Cloudhub
Anypoint Runtime Fabric (internally uses Docker and Kubernetes at the moment)
Container as a Service provided by multiple cloud vendors - Google Container Engine, Amazon Container Service, Azure Container Service etc.
Build your own cluster with containers

As we go down the above list, the level of details we need to get into is more. So, this post will get into explain how to implement option 4 above.

We will make use of various technologies and Azure components such as - Docker, Containers and Azure VM Scale Set to achieve it. For those of you are new to these concepts, we will go through each of the steps in detail.

Application

Let's use the same Mule API that was created in the previous post. The code is available here. Import this application in Anypoint Studio 6.5 and test it locally.

Container

Next, we need to 'package' this application in a container. As Docker is the prevalent tool in this area, let's use it. The container used in this example is built using this Dockerfile. The Docker image created is available in Docker Hub. The image can be pulled from Docker Hub to a Docker host machine and tested locally.

Scheduler

In this example, we just have one container. But, if we have a more complex example where multiple containers make up an application, then we need to use a Scheduler. Like Docker, the most popular scheduler is Kubernetes. Just like the containers are deployed to an auto scaling VM cluster, Kubernetes can also be deployed in the same way. Perhaps, in another post, we will cover this too.

Cloud

Now, for the main part of this post. In this example, we are going to take it a step further and create an auto scaling cluster that can add/remove VM's based on the load on the application.

Now, that we have the container ready, we need to deploy this container to the VM in an automated manner. Using Azure Custom Extensions, the container can be deployed. The custom extension comprise of two files - a shell script that is executed for each VM and a configuration file. Both can be found here. The container is deployed to a VM whenever it is created for the first time.

In addition, the VM Scale Set also has scale-out and scale in rules defined. If the CPU goes above 50%, an additional VM is added (and the container deployed). This is the scale-out rule. If the CPU goes below 30%, the VM is deleted. This is the scale-in rule.

1. We initial start with a VM Scaled Set which has the minimum VM's set to 2 and maximum to 4. Initially, with little or no load, the 2 VM's are running (as our minimum is set to 2 for resilience). the minimum and maximum are configurable.

2. The CPU usage in both the VM's is negligible.

3. The load test used in the last post duplicated and modified to point to the public IP of the load balancer of the VM set. The load test results this time are much better. We will see in a moment, why.

4. While the load test is running, the CPU in the VM's jumps to above 50%. In the portal, there is also an 'Updating' message which shows that the VM Scale Set is being updated.

5. When the instances tab is viewed, the additional VM's have been created. The Docker container have also been deployed automatically to these VM's.

6. The additional VM's have also been added to the load balancer automatically. The incoming requests to the Mule API are now load balanced across 4 VM's.

7. Once the load test ends, the CPU usage across all VM's comes down to the original state.

8. One by one, the newer VM's are also deleted completely.

9. The status of the deletions can also bee seen in the Azure Portal.

10. Finally, we are back to the original cluster size of 2 VM's.

11. Once the test is over, we can also see the amount of CPU credits utilized. The below image shows that more CPU credits were consumed only when the 2 additional VM's were created.

Summary

Scaling out an application horizontally in the cloud helps when an application faces unexpected load. Generally speaking, this approach is more scalable than vertical scaling. Having said that, depending on the type of application, horizontally scalable applications come with their own challenges such as maintaining consistency and state across a cluster of VM's.
As this example shows, the benefit of the elastic scalability is that money is spent only when needed. This ends up saving money, if we had a fixed VM capacity.
The combination of Docker and VM Custom Extension is quite powerful as almost any kind of application (like web applications, databases, 3rd party software) can be deployed and scaled automatically.
One could say that the auto scaling and orchestration using Docker Swarm or Kubernetes is faster and more powerful.
However, if there are use cases such as a Python based web application, this approach might be simpler and equally beneficial because languages such as Python are available by default in most Linux distributions. Also, it saves the effort for a team of learning and supporting additional technologies.
In this example here, the assumption is that this API will have an unpredictable or varying load on it. For this kind of use case, the 'burstable' B-series VM types are ideal. We get charged for the actual CPU utilization and not the vCPU's allocated in the VM. More details can be found here. So, the above VM Scale Set is using the B1ms VM type.

Finally, it is quicker and cheaper to try out various application architectures and deployment models in the cloud. This example took a a day to build and test. And, it cost around 80 INR (roughly equivalent to 1.1 USD). And, that is roughly the same price as a McDonalds Chicken Kebab Burger in India!

Azure Chronicles - Mule

2019-02-03T13:09:00.002+05:30

Overview

Continuing from the previous post, I now move to the other end of the spectrum, from Raspberry Pi to the cloud. Microsoft Azure cloud has been making impressive strides in the cloud space, especially among the enterprises.

This is an experiment of running the same Mule run time in an Azure cloud server. The server is created in the East US Azure location.

I created an A series server (Extra Small) in Azure using the ARM templates. An A series Extra Small server has 1 CPU core and 07.GB RAM. On this server, the Mule run time is on a JVM with 256 MB heap size.

Then, I then ran some scripts to perform the following -

Install Oracle JDK 8
Configure the Mule run time (version 3.9)
Deploy a simple REST based API in Mule

Objective

The purpose of this experiment is to test how much load can a very simple REST API in Mule handle, while running on a very low specification server in Azure. I wanted to see how the Mule run time would perform from a performance perspective.

A simple REST API is created using Anypoint Studio. The response of the REST API is dynamic based on the parameters passed to it. This ensures that the response doesn't get cached in any layer - server/network/client.

Once the REST API is deployed in Mule run time in Azure, a sample request is tested from a browser. The JSON response is as below. This sample API is based on the sample RAML available at Mulesoft documentation site.

As the REST API is publicly exposed, the load tests are setup in Blazemeter.

Before starting the load test from Blazemeter, the resource statistics on the server are normal. Press F5 in htop utility to view the processes in a tree structure.

The Blazemeter tests are now kicked off and the test results are impressive. The 90 percentile response time is under 200 ms. For a "read" operation on a REST API, this response time is pretty good. The average throughput is also looking good at 170 hits/second. This test simulates 20 concurrent virtual users hitting the REST API endpoint. For an A0 server, this performance is not bad at all.

During the above test, the load on the CPU has spiked to 100%. The server stats can be viewed from the Azure Portal.

Now, I setup the New Relic agent on the server, restarted the Mule run time and re-ran the same test. The results are now starkly different. The 90 percentile response time has gone up by 5 times, to above 1 second. The average throughput has dropped by more than 50% to 71 hits/second.

While we can see the application statistics in New Relic, there is a warning as well. This is about a circuit breaker being triggered in the New Relic agent.

The circuit breaker gives more details that the JVM heap is under pressure.

Summary -

When running API's or websites in a production environment, there is usually a server cluster / farm that handles all requests. While a Mule run time can technically run on an A0 server, one should probably use at least a Dv3 or a Ev3 VM type in Azure. For details on more Linux VM types, please refer here.
New Relic agent has the "circuit breaker" pattern implemented, which is quite thoughtful and helpful.
When sizing the memory for a JVM based application, keep in mind the additional CPU/Memory requirements for an agent such as New Relic.
If the API's are publicly exposed, evaluate cloud based tools such as Blazemeter. Also, ensure that the cloud based tool and the server are in different clouds, to simulate traffic across the internet. For instance, here the server is in Azure while Blazemeter test runner is in AWS.
Potentially, Blazemeter could be used not just for load testing of API's but also as a synthetic monitoring tool.
Mule Community edition is a great way to explore the tool. From an enterprise perspective, as of now, multiple deployment options are available. They are - On Premise, Anypoint Cloudhub (SaaS) and Anypoint Runtime Fabric (PaaS).

Adventures with Raspberry Pi - Part 4

2018-11-03T17:49:00.001+05:30

As we saw in the previous post, we now have Raspberry Pi server setup and integrated with multiple mobile devices successfully, let's try a more complex integration.

API's have become the de-facto standard for communicating between devices. But, if we also wanted to implement any logic, data transformation, loose coupling between devices, then we could use an ESB. One can think of the Mule ESB as another form of IFTTT. We can use the multiple connectors to link up different kinds of devices, services and applications.

Let's two kinds of integrations - event driven and batch.

Event Driven

In this example, a service pushes data to the Raspberry Pi.

Event Driven Flow

I am using Anypoint Studio 7 and Mule Runtime 4. A mockup product API is created in the Mule project. This API is exposed as a REST API with a GET operation. When someone calls the API, the Mule flow is executed. In the Mule flow, the data is read from the API and sent via SFTP to a target device.

In this case, the target device is a Raspberry Pi with SFTP server running on it. As part of the Mule flow, additionally logging and error handling are also implemented.

Mule Flow in Anypoint Studio

Once the Mule project is built and run, the API and the flow is hosted in a Mule runtime (which in turn is running on a JVM). An API console is available to test or call the product API.

API Console

On invoking the product API, the data is transferred in the form of a JSON file to the Raspberry Pi SFTP folder.

Raspberry Pi SFTP

Batch

Let's now look at a batch example. Of course, one could also have a "reverse" flow. Raspberry Pi could monitor it's own temperature and push it to the Mule ESB as well.

Batch Mule Flow

Setup a cron job in the Raspberry Pi that runs every minute to collect the CPU/board temperature.

Crontab entry for a "every minute" job

The cron job writes the temperature details to a text file locally on the Raspberry Pi. The same file is over written every minute.

Temperature details

There is a Mule flow that reads the text file from the Raspberry Pi using an SFTP connector. The temperature details are then emailed using an Email connector to Outlook 365. Unlike the earlier flow, this one runs as a batch driven by the Scheduler component in the first step.

Scheduled job in a Mule Flow

In order to test this example properly, one would need two email addresses - one to send and one to receive. For security reasons, the email address is masked in the screenshot below.

Status emails in Outlook 365

Of course, now one could setup many other kinds of flows like - collect the weather from an web based API and email, monitor the stock prices and email etc.

Mule ESB on IoT

Lastly, the Mule ESB can also be run on the Raspberry PI. Both version 3.x and 4.x can be installed side-by-side in the same Pi. Below is an example of Mule 3.9 Community Runtime running in a Raspberry Pi model 3 B.

Mule CE Runtime v3.9 on a Raspberry Pi 3 B

In addition, the New Relic APM agent is also running in the Pi and is monitoring the Mule API's, flows and JVM statistics.

New Relic Dashboard showing Mule statistics

Adventures with Raspberry Pi - Part 3

2018-08-29T19:10:00.001+05:30

Continuing from the previous post, lets now build a Chatbot on the Raspberry Pi. If we have some digital media such as music, movies etc. stored in a media server on the Pi, these can be accessed using a Chatbot too.

I have used Telegram to build the bot. The Telegram app is available for all platforms - iOS, Android, Web etc. Using the funnily named BotFather from Telegram, anyone can create a Bot account.

Once the account is created, the "brain" of the Chatbot needs to be implemented using a programming language and framework. I have use the python based chatterbot framework to create a bot called - Frasier (named after the TV Series). The bot in turn interacts with the Plex Media Server API's to search and download the requested media such as a song, TV show or a movie. Telegram currently allows files up to 1.5 GB to be shared via the app.

Telegram App in an iPhone requesting a song from a Chatbot running on a Raspberry Pi

In the screenshot above, the chats in Blue are my commands and in Gray are the responses from the Chatbot.

Requesting for digital content is just one of the examples. One could build additional commands such as "/weather" to know the weather, "/stocks" to know about shares, "/news" to download news articles etc. Of course, for each such command, some API from underlying services such as Weather Underground, Yahoo Finance, News Sites would have to be integrated into the Chatbot.

Have fun building your own Chatbots in the Raspberry Pi!

Adventures with Raspberry Pi - Part 2

2018-07-22T19:39:00.004+05:30

Continuing from the previous post, let us now put the Pi to some practical uses.

File Server

The Pi can serve as a file server for home use. One could use it to store files for use across multiple computers, backup files, share media (songs, photos and videos) etc. As most of us use Windows at home, you can setup a Samba file server. You can read/write files easily from Windows computers. In addition, there are Apps for iOS and Android, so you can access these files from mobile/tablets too.

You can SSH into the Pi and run the following commands to install Samba

sudo apt-get update
sudo apt-get upgrade
sudo apt-get install samba samba-common-bin

You could give all access to the Public folder under /home/pi/ folder using the command - chmod +777.
Create a Samba user and password using the command - sudo smbpasswd -a pi
Configure Samba as shown below -

Edit the Samba configuration - sudo vi /etc/samba/smb.conf
[share]
Comment = Pi shared folder
Path = /home/pi/Public
Browseable = yes
Writeable = Yes
only guest = no
create mask = 0777
directory mask = 0777
Public = yes
Guest ok = yes

Finally, restart and reload Samba using the command - sudo /etc/init.d/samba restart.

The above steps are just a one-off. Whenever the Pi is restarted, the Samba server will be up. From a a Windows computer, you can use the following address in Explorer to access the file share - \\[Pi Name]\Public. The files from the computer can now be copied to the above shared folder that you have just created.

Media Server

Digital Living Network Alliance (DLNA) was initially created by a group of consumer electronic companies. It's aim was to promote inter-operability among gadgets produced by various electronics manufacturers.

Suppose, you have copied some media files (audio and video) to the above shared folder on the Pi. You can setup a DLNA software server in the Pi. Using the DLNA server, this media can be streamed to other devices within your network.

Install Mini DLNA server using the command - sudo apt-get install minidlna.
Edit the DLNA configuration - sudo nano /etc/minidlna.conf.
media_dir=A,/home/pi/Public/Music
media_dir=V,/home/pi/Public/Movies
friendly_name=Pi-Media-Server
sudo service minidlna restart
sudo service minidlna force-reload

These steps are again a one-off. The DLNA server will show up as "Pi-Media-Server" in all the clients such as a Smart TV's, Mobile/Tablet Apps etc. You can verify the DLNA server by opening the URL in browser - http://[Pi Name]:8200/, you should see the status page below.

For a more advanced media server, one can also install the Plex Media Server on the Raspberry Pi. In the Plex Server, one can also enable it to be accessed from the internet. The official Plex app is available for iOS, Android and Smart TV devices. This means that all the movies, TV shows and songs in the Plex App can be accessed on the go wherever you are. The Plex server and apps are all free to download and use. Plex server can be downloaded and installed from here. The Plex mobile apps will be available int he respective app stores.

Plex App in iPhone connected to the Raspberry Pi

Plex App in iPad connected to the Raspberry Pi

You can buy wireless external hard drives that come with the same features (Samba, Plex etc.) as above. These drives normally come bundled with the Plex media/DLNA server and cost anywhere from INR Rs 16000 upwards in the online shopping sites such as Amazon (India). It cost me about INR Rs 6000 to create the same using the Pi.

So, not only do you get to save some money but also have fun in the process! More coming up in the next post.

Adventures with Raspberry Pi - Part 1

2018-05-29T18:26:00.000+05:30

Raspberry Pi has been around for a few years. It was initially launched in 2012 by the UK based Raspberry PI foundation. Over the years, many flavors of the Raspberry PI have been created.

While the Pi computer has been created as a learning tool, it can be used for some practical use cases too. These can be for both home and business use.

Below is my Raspberry Pi setup at home.

Imperial storm troopers protecting their precious cargo of Raspberry Pi, Mi power bank and Western Digital hard disk. All the secrets of the empire are safe! #starwars #raspberrypi #stormtrooper #maker #diy

A post shared by Rajan (@rajanmanickavasagam) on May 9, 2018 at 8:33pm PDT

The bill of parts for the above setup consists of (left to right) -

Mi 20000 maH Power Bank
USB to Micro USB cable with a power on/off switch
Raspberry Pi case with heat sink
Raspberry PI Model 3
Class 10 SD Card 16 GB capacity
Western Digital 1 TB external hard drive
The storm troopers just showed up! 😊

There are many tutorials available online which show how to put all of these together. Here are some useful links -

My initial observations are -

The default OS in the Pi is Debian based. This means one can use the "apt-get" package manager to install required software like Java, Python, NodeJS etc.
Containers like Docker and databases like MySQL can be setup on the Pi.
Given the limited resources (CPU/memory/disk) on the Pi, some of these software may not perform at "full throttle" but at least functionally they will work.
Keep an eye on the "heat" from the Pi. Without the heat sink, the temperature Raspberry Pi was around 10-15 celsius higher.
Normally Pi's are powered from a main power source. The power bank is a quirky addition. One can now literally run a portable and fully functioning Pi computer for several days. Imagine, you could now run a Hadoop cluster on a handful of Pi computers and carry it around in a bag.

My experience with Pi started more as a hobby. But, the ease and stability of using the Pi encouraged me to explore more use cases. Some of the home/personal use cases I have tried are -

Setup a file server in the Pi
Setup a streaming media server in the Pi
Build a personal assistant
IoT style integration

More about this in the next blog post.

Let Us Say Grace

2018-01-13T11:45:00.002+05:30

As we head into another new year, the calendar has changed to 2018. But the differences in the world continue to remain. As long as the human condition prevails, these differences will always exist. These differences periodically surface as flash points in politics, economics, societies, arts and personal freedoms.

Saying Grace by Norman Rockwell (Source - Wikipedia)

There is a painting "Saying Grace" by the iconic American illustrator Norman Rockwell. It was created for the cover of the Saturday Evening Post in 1951. As is characteristic of Norman Rockwell paintings, it captures an ordinary moment with extraordinary detail. A boy and his mother say their prayers in a restaurant before a meal in a restaurant. We can also see the reactions of the other diners, some fully, some partially and some not at all.

What is truth? What is justice? What is equality? What is morality? Before we even debate these topics, we must search for the meaning of them individually. We must accept that on these and many other topics, we sometimes understand it fully, partial or not at all. Only life can reveal the understanding to us in many different ways. And, that is a journey we all have to undertake.

So, in the new year and beyond, like the diners in the painting let us observe and not judge harshly what we do not understand. Even if we disagree or debate on our differences, let us do so with grace. Perhaps the grace will help us come together.

Happy New Year to everyone!

Rajan's View

Clawing Now to Speed Up Later

OpenClaw

Taking it for a spin

OpenClaw The Builder

Investment Council

Key Features

Key Benefits

Stock Analysis

📊 Comprehensive Multi-Dimensional Analysis

🔍 Deep Research Engine

🎨 Advanced Visualization Suite

🛠️ Production-Grade Reliability

📄 Retail-Friendly Report Generation

⚙️ Modular & Scalable Architecture

Tips for Builders

OpenClaw The Worker

OpenClaw The Assistant

What works well

How to apply & scale it in Enterprises

Gemini CLI - Your Personal AI Investment Advisor

Unlocking Financial Insights with Gemini CLI and MCP Servers

The Brains Behind the Brawn: Gemini CLI Meets MCP

My Setup: A Glimpse Under the Hood

What Can Your AI Investment Advisor Do?

Getting started

Provide real-time stock quotes and historical data.Let's use Google as a test case for all our analysis. You can type in a prompt - "get basic fundamentals of GOOG".

Analyze key performance indicators.Next, type in the prompt - "analyze the technical setup of GOOG". Again, a different tool asks for permission.Once, the tool runs, the technical analysis results are shown. Not bad.

Summarize news and sentiment around specific stocks or sectors.

Now, let's ask for "get the latest market news on tech stocks".

Once the tool confirmation is provided, the news is retrieved from the MCP Servers/API's.

Generate charts and visualizations of financial data.We can now try to visualize the data a bit better by asking - "what time series data do we have available for GOOG and MSFT". Based on the data already retrieved, charts can also now be created.

Create reports and presentations on demand

Now that we have done all the analysis, let's create a presentation - "create a presentation with all the analysis done so far and add the charts as well". The relevant tool now proceeds to create the presentation with all the details and charts.

Roadmap

The Future of Personal Finance is Agentic

Vibe coding your way to Agentic AI

Tools of the Trade

Beyond suggestions, I also used Roo/Gemini 2.0 to write the code and update the README.md. You can see the generated agent code at GitHub. The steps to install, test and deploy the app are in the same GitHub repository.

Agentic AI using Google ADK

Design Pattern

What’s Next

Meta Prompting for Success

Meta Prompt

How to Create a Meta Prompt

A Sample Meta Prompt

Gemini Draft Response 1

Executing Prompts

Prompt 1

Prompt 2

Prompt 3

Prompt 4

Prompt 5

Gemini Draft Response 2

Executing Prompts

Prompt 1

Prompt 2

Prompt 3

Conclusion

The Many Faces of Val Kilmer

Simon Magus

John Rossi

Martin de Porres

Bruno Hautenfaust

Thomas More

Charles Borroneo

Peter Damian

Luigi Guanella

Nicholas Owen

Vincent Ferrer

August Christopher

Notes

Update

Now you can see me

Overview

Bridging the Satisfaction Gap

Art of the Possible, Today

Capturing Intent Signals

Understanding the Intent

Creating Experience from the Intent

Provide real-time stock quotes and historical data.
Let's use Google as a test case for all our analysis. You can type in a prompt - "
get basic fundamentals of GOOG".

Analyze key performance indicators.
Next, type in the prompt - "analyze the technical setup of GOOG". Again, a different tool asks for permission.

Once, the tool runs, the technical analysis results are shown. Not bad.

Generate charts and visualizations of financial data.
We can now try to visualize the data a bit better by asking - "what time series data do we have available for GOOG and MSFT".

Based on the data already retrieved, charts can also now be created.